Building RAG systems that hold up in production

Retrieval-augmented generation (RAG) sounds simple: split your documents into chunks, embed them, find the chunks closest to the question, and give them to a language model with the question. A first version takes a day. Getting it to answer correctly, cite its sources and say "I don't know" when the documents don't contain the answer takes much longer. The failures are almost always in retrieval, not in the model.

Where RAG systems go wrong

Three problems account for most bad answers.

Chunks that lose their context. Splitting every document into fixed-size blocks cuts tables in half, separates a heading from the paragraph it explains, and produces chunks such as "This does not apply to contractors" with no record of what "this" is.

Retrieval that misses the right passage. Vector similarity finds passages that mean something close to the question. It is weak on exact terms: product codes, error messages, names, clause numbers. If the right chunk is not retrieved, the model cannot use it.

Too much, or the wrong mix, in the prompt. Passing every retrieved chunk makes the model work harder to find the answer, raises cost and latency, and gives it more conflicting text to blend.

Chunk by structure

Split along the structure the document already has: sections, headings, list items, table rows. Keep the heading path on every chunk, so a chunk from "Leave policy > Contractors > Exceptions" carries that path into the prompt. Add a small overlap between neighboring chunks only where a section runs long enough to be split.

chunks = chunk_by_structure(
    document,
    max_tokens=512,          # upper bound; most chunks follow section boundaries and are smaller
    overlap_tokens=50,       # only applied when a long section has to be split
    keep_heading_path=True,  # "Leave policy > Contractors > Exceptions" travels with the text
    keep_tables_whole=True,
)

Configure the pipeline as data

Indexing has four steps: load, chunk, embed and write to the index. Describe each step as configuration, so a change to the chunking or the embedding model is a reviewed change with a version number, and so you can rebuild the index from scratch when it changes.

interface PipelineConfig {
  chunking: { strategy: 'structure'; maxTokens: number; overlapTokens: number };
  embedding: { model: string; dimensions: number };
  index: { store: string; namespace: string };
}

const config: PipelineConfig = {
  chunking: { strategy: 'structure', maxTokens: 512, overlapTokens: 50 },
  // text-embedding-3-large returns 3072 dimensions by default; the API can shorten it.
  embedding: { model: 'text-embedding-3-large', dimensions: 1536 },
  index: { store: 'postgres-pgvector', namespace: 'kb-v2' },
};

await runIndexing(documents, config);

A new namespace for each configuration version lets you build the new index next to the old one, evaluate it, and switch over only when it scores better.

Retrieve in stages

Retrieval works best as a funnel: a wide first pass, then a precise second pass.

async function retrieve(query: string) {
  // Stage 1: keyword (BM25) and vector search together, for recall.
  const candidates = await hybridSearch(query, { vectorWeight: 0.7, keywordWeight: 0.3, limit: 50 });

  // Stage 2: a cross-encoder scores each candidate against the query, for precision.
  const reranked = await rerank(query, candidates, { model: 'cross-encoder/ms-marco-MiniLM-L-6-v2', limit: 10 });

  // Stage 3: choose what goes into the prompt (next section).
  return selectForContext(reranked, { maxTokens: 4000 });
}

The weights and limits above are starting points. Tune them against an evaluation set, not by reading a few answers.

When questions are short or use different words from the documents, rewrite the query before searching: expand abbreviations, add the terms your documents use, or have the model write two or three alternative phrasings and merge the results.

Decide what goes into the context

Suppose retrieval returns ten good chunks and you want to pass five. Which five?

1. Rank by the reranker's score, not the first-pass score.
2. Drop near-duplicates. Two versions of the same paragraph waste space and can conflict.
3. Keep a token budget. Stop adding chunks when the budget is reached, even if the context window has room. Current context windows hold hundreds of pages, so the limit is no longer the window. It is answer quality, cost and latency. Models use information at the start and end of a long context more reliably than information in the middle (Liu et al., "Lost in the Middle", 2023), so more text is not free.
4. Put the strongest chunk first, and keep chunks from the same section together in document order, with their heading paths, so the model reads them as one passage.
5. Pass nothing when nothing is relevant. If the best reranker score is below a threshold, let the model answer that the documents don't cover the question.

Generate with citations, then check them

Number each chunk in the prompt and ask the model to cite the numbers it used. Then check the citations in code before showing the answer.

const response = await llm.generate({
  system: 'Answer only from the numbered sources. Cite them as [1], [2]. If the sources do not answer the question, say so.',
  context: chunks.map((c, i) => `[${i + 1}] ${c.headingPath}\n${c.text}`),
  query: userQuery,
});

const cited = extractCitations(response.text);           // e.g. [1, 3]
const unknown = cited.filter((n) => n < 1 || n > chunks.length);
const uncited = cited.length === 0 && !saysNotFound(response.text);
if (unknown.length || uncited) return retryOrRefuse(); // never show an answer with broken sources

For higher-stakes answers, also check that each quoted figure or phrase appears in the chunk it cites.

Measure it

Build an evaluation set of real questions, each with the passage that answers it, plus questions the documents cannot answer. Then track:

• Retrieval recall: how often the answering passage is in the top results.
• Faithfulness: how often every claim in the answer is supported by a cited chunk.
• Correct refusals: how often the system says it does not know when the answer is not in the documents.
• Latency and cost per query, by stage.

Rerun the set before every change to chunking, embedding, retrieval settings, prompt or model.

When not to use RAG

• The documents fit in the context window. If the whole corpus is a few hundred pages or less, put all of it in the prompt and skip retrieval. Prompt caching keeps the repeated cost down. Fine-tuning is not the alternative here: it teaches a model a format or a style, not facts it can quote reliably.
• The answer is in structured data. Questions about counts, totals and dates belong in a database. Give the model a tool that runs a checked query, rather than retrieving rows as text.
• The data changes faster than the index. RAG works on current data as long as the index keeps up. Re-index documents when they change, or have retrieval call the live system through an API. The question is how fresh the index is, not whether the data is new.

How this fits with agents

In most systems we build, retrieval is one tool an agent can call, next to database queries and API calls. The same rules apply as for any tool: a clear definition, a schema the API accepts, an evaluation set, and tracing on every call. That is the work of our Production AI agent platform.