The Demo Retrieval Works. The 10,000th Query Doesn't.
Most RAG prototypes are built the same way: chunk the documents, embed them, pull the top-5 nearest vectors, stuff them into the prompt. It works beautifully on the five questions the team tested. It starts failing the moment real users ask real questions — because real questions are messy, partial, multi-constraint, and sometimes the answer isn't even in the corpus.
Retrieval-augmented generation is not an embedding lookup. It is a pipeline with measurable stages, and every stage can be budgeted, tested, and improved. Here is the version that survives production.
Step 1: Chunking Is a Precision Decision
Chunk size determines what a single retrieval hit can contain. Too small, and context is fragmentary — the model sees the fact but not the caveat. Too large, and each hit wastes token budget on noise while relevant details get diluted. Practical starting points:
- A few hundred tokens per chunk with 10-20% overlap usually beats both extremes.
- Respect structure: never split a table, a code block, or a list mid-item.
- Store chunk metadata (source, section, date, document type). You will need it for filtering and for citations.
There is no universal number. The chunking parameters are a starting hypothesis, and the retrieval eval from step 4 is what confirms or kills it.
Step 2: Hybrid Retrieval Beats Any Single Method
Dense vector search catches paraphrases ("how do I cancel my plan" finds the refund policy). But it is reliably weak at exact identifiers: product codes, error strings, person names, part numbers. Keyword methods like BM25 are the opposite — superb on rare exact terms, blind to synonyms. A retrieval stack that uses only one of them ships a blind spot by design.
The standard fix is to run both retrievers and fuse their rankings. Reciprocal Rank Fusion (a simple, tunable-free formula over the two ranked lists) is a strong default. Then apply a reranker: a cross-encoder model that reads the query together with each candidate chunk and re-scores true relevance. Retrieve generously (top 50-100 from each retriever), fuse, rerank, keep the best 5-10 for the prompt.
Step 3: Not Everything Belongs in the Vector Store
A counterintuitive lesson from systems that answer high-volume factual questions: some questions can never be answered correctly by retrieving raw documents. Questions about distributions, boundaries, and aggregates — "in what share of cases does X happen?" — lure the model into interpolating from a handful of retrieved anecdotes. The failure is systematic, not random.
The fix is offline preparation: precompute the answers to predictable aggregate questions as distilled summary cards, store them alongside raw documents, and route statistical questions to the cards, not to raw retrieval. Treat this as part of knowledge-base writing governance: raw documents are one representation of knowledge; curated summaries are another, and both have their place.
Step 4: Acceptance Is a Recall Number, Not a Vibe
A RAG pipeline is a retrieval system first and a generation system second. Give retrieval its own acceptance test, independent of the LLM:
- Build a golden set: 50-200 real questions with the chunks (not just answers) that must be found.
- Measure recall@k — the share of golden chunks present in the top-k fused results. This is your pipeline's contract.
- Only then measure end-to-end answer quality on the same set.
When answer quality drops, the recall number tells you instantly whether to blame retrieval or generation. Without it, teams spend weeks tuning prompts for what is actually a retrieval regression.
Step 5: Agentic Retrieval for the Hard Tail
Fixed pipelines retrieve once and answer. Harder queries — comparisons across documents, multi-hop questions, "find and verify" tasks — need the agent to decide whether to retrieve, what to search for next based on what came back, and when it has enough. That is agentic RAG: retrieval as a tool in the agent's loop rather than a fixed pre-step.
The trade-off is latency and cost: every extra retrieval round is another LLM call. The pragmatic pattern is escalation — try the cheap fixed pipeline first, detect low-confidence answers, and only then let the agent run a multi-step search.
Key Takeaways
- Chunking is a hypothesis; the retrieval eval is the experiment.
- Run dense + keyword retrieval in parallel, fuse with RRF, rerank before the prompt.
- Precompute summary cards for statistical and boundary questions — raw chunks will answer them wrong.
- Gate the pipeline on recall@k against a golden set of real questions.
- Escalate to agentic multi-step retrieval only when the cheap path is not confident.
