Retrieval-Augmented Generation in Production - Lessons from 2026

Every RAG tutorial makes it look like three lines of code: chunk the docs, embed them, ask a question. Every RAG system that has actually survived a year in production looks nothing like that - it's chunking heuristics tuned per document type, hybrid retrieval, a reranker, a staleness problem nobody budgeted for, and an evaluation harness that runs on every prompt change. Here's what we learned scaling ours.

Chunking Strategy Is Not a Solved Problem

Fixed-size chunking (e.g. 512 tokens with 50 token overlap) is the default everyone starts with, and it is wrong for almost every document type except plain prose. What actually worked for us:

• Markdown/structured docs - chunk on heading boundaries first, then split oversized sections by token count, so a chunk never straddles two unrelated topics
• Tables - never split a table across chunks; serialize the whole table (or a logical sub-table) as one chunk with a text summary prepended
• Code - chunk by function/class boundary using a language-aware parser, not a token count
• Legal/contract text - chunk by numbered clause, since a clause is the smallest unit that's still meaningful in isolation

def chunk_markdown(document: str, max_tokens: int = 400) -> list[str]:
    sections = split_on_headings(document)
    chunks = []
    for section in sections:
        if count_tokens(section) <= max_tokens:
            chunks.append(section)
        else:
            chunks.extend(split_by_token_budget(section, max_tokens, overlap=40))
    return chunks

The overlap number matters less than people think once you have decent boundary-aware chunking - we run 10-15% overlap, mostly as insurance against a sentence getting cut mid-thought.

Embedding Model Choice

We benchmarked text-embedding-3-large, Cohere's embed-v4, and a self-hosted bge-large-en against our own retrieval eval set (query/relevant-chunk pairs curated from real support tickets) rather than trusting public MTEB leaderboard rankings, which don't reflect domain-specific vocabulary.

results = {}
for model in ["text-embedding-3-large", "embed-v4", "bge-large-en"]:
    embeddings = embed_corpus(corpus, model=model)
    results[model] = evaluate_recall_at_k(embeddings, eval_queries, k=10)

# recall@10 on our eval set:
# text-embedding-3-large: 0.87
# embed-v4:               0.91
# bge-large-en:           0.83

embed-v4 won on our data specifically because of heavier representation of technical jargon in its training mix - a reminder that you should always re-run this benchmark on your own corpus before committing.

Hybrid Search: Vector Alone Isn't Enough

Pure vector search misses exact-match queries badly - a product SKU, an error code, a customer's account number. We run BM25 keyword search and vector search in parallel and merge with reciprocal rank fusion.

def hybrid_search(query: str, k: int = 20) -> list[Chunk]:
    vector_hits = vector_index.search(embed(query), top_k=k)
    keyword_hits = bm25_index.search(query, top_k=k)

    scores: dict[str, float] = {}
    for rank, hit in enumerate(vector_hits):
        scores[hit.id] = scores.get(hit.id, 0) + 1 / (60 + rank)
    for rank, hit in enumerate(keyword_hits):
        scores[hit.id] = scores.get(hit.id, 0) + 1 / (60 + rank)

    ranked_ids = sorted(scores, key=scores.get, reverse=True)
    return [chunk_store[cid] for cid in ranked_ids[:k]]

This single change - adding BM25 alongside vector search - fixed more retrieval failures in our eval set than any embedding model swap did.

Reranking Before It Reaches the LLM

Retrieval recall at k=20 is cheap to get high; precision at the top 3-5 chunks you actually feed the LLM is where quality lives. A cross-encoder reranker sits between retrieval and generation and re-scores the candidate set with much more context than a bi-encoder embedding comparison can capture.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

pairs = [(query, chunk.text) for chunk in candidates]
scores = reranker.predict(pairs)

reranked = [c for c, _ in sorted(zip(candidates, scores), key=lambda x: -x[1])]
top_chunks = reranked[:5]

Reranking added roughly 80-120ms of latency for us, which was a fine trade for the drop in irrelevant-chunk-induced hallucinations.

Handling Stale and Updated Documents

The problem nobody's tutorial covers: what happens when the source document changes? Re-embedding everything on every edit doesn't scale, and stale chunks silently answering with outdated policy is a real production incident, not a theoretical one.

• Track a content hash per source document; only re-chunk and re-embed documents whose hash changed
• Version chunks with an effective_date and superseded_by pointer so old answers can be traced back to the version they were grounded in
• Run a nightly job that flags any chunk older than a configurable TTL for manual review rather than silently trusting it forever

async function syncDocument(doc: SourceDocument) {
  const newHash = sha256(doc.content);
  const existing = await chunkStore.getByDocId(doc.id);

  if (existing?.contentHash === newHash) return; // unchanged, skip

  await chunkStore.markSuperseded(doc.id, existing?.version);
  const chunks = chunkDocument(doc.content);
  await chunkStore.upsertChunks(doc.id, chunks, {
    contentHash: newHash,
    version: (existing?.version ?? 0) + 1,
  });
  await vectorIndex.reindex(doc.id, chunks);
}

Evaluating Groundedness and Hallucination Rate

We run an LLM-as-judge groundedness check on a sample of production traffic daily, not just at release time - retrieval quality silently degrades as the corpus grows and nobody notices until a customer does.

judge_prompt = """
Given the CONTEXT and the ANSWER, rate from 1-5 how well the answer
is supported by the context alone. An answer that introduces facts
not present in the context should score 1-2, regardless of whether
those facts are true in general.

CONTEXT: {context}
ANSWER: {answer}
"""

groundedness_scores = [judge_llm.score(judge_prompt.format(**sample)) for sample in daily_sample]
hallucination_rate = sum(1 for s in groundedness_scores if s <= 2) / len(groundedness_scores)

We alert if hallucination_rate crosses 5% on the rolling daily sample - it's the earliest signal we get that either the corpus has drifted or a retrieval regression shipped.

What We'd Tell a Team Starting Today

• Build the eval set before you build the pipeline, not after
• Hybrid search is not optional past a few thousand documents
• Budget for a reranker from day one; retrofitting it later means re-tuning your whole latency budget
• Staleness handling is a first-class feature, not a cron job you bolt on when someone complains

RAG in production is a data engineering problem wearing an LLM costume. The model call is the least interesting five percent of the system - the chunking, retrieval, and staleness handling around it are what actually determine whether users trust the answers.