Beyond Naive RAG: Building Production-Grade Retrieval Pipelines

The standard RAG tutorial flow — split documents into fixed-size chunks, embed them, retrieve top-k by cosine similarity, pass to LLM — collapses under real enterprise conditions. Fixed chunking breaks semantic units across boundaries. Cosine similarity retrieves lexically similar but semantically irrelevant passages. Top-k retrieval with no reranking delivers inconsistent precision. When your knowledge base has 500,000 documents and users ask complex multi-part questions, these failures compound into an unreliable system that erodes user trust within weeks of deployment.

Replace fixed-size chunking with semantic chunking. Parse documents into their natural structural units — paragraphs, sections, tables, code blocks — and use a sentence boundary detector to avoid mid-sentence splits. For documents with rich structure (legal contracts, technical manuals, financial reports), build a hierarchical index: embed both the full section and its subsections, and use parent-child retrieval to return the most relevant level of granularity. This alone typically improves retrieval precision by 25–40% on enterprise document sets.

Dense retrieval (embedding-based) excels at semantic similarity but misses exact-match queries — product codes, regulatory article numbers, specific named entities. Sparse retrieval (BM25, TF-IDF) handles exact matches but misses paraphrased queries. Production systems need both. Implement a reciprocal rank fusion (RRF) ensemble that combines dense and sparse scores, weighted by query type. A query classifier trained on 1,000 labelled examples from your domain can route queries to the optimal retrieval strategy before the ensemble step.

After retrieval, run a cross-encoder reranker — models like BGE-reranker-large or Cohere Rerank — over the top 20–50 candidates to produce a final top-5. Cross-encoders attend to both the query and the passage simultaneously, producing much higher-precision rankings than embedding similarity. The latency cost is 80–150ms, which is acceptable for most enterprise applications. For latency-sensitive use cases, use a lighter bi-encoder reranker or cache reranked results for frequent queries.

Short, ambiguous user queries are the enemy of good retrieval. Query expansion generates multiple reframings of the original query and retrieves against all of them, then deduplicates and reranks the combined result set. Hypothetical Document Embeddings (HyDE) takes a different approach: ask the LLM to generate a hypothetical answer to the query, then embed and retrieve against that hypothetical. HyDE works well for open-domain questions but can hallucinate in constrained enterprise settings — use it selectively, with a fallback to direct retrieval when the hypothetical is low-confidence.

You cannot improve what you cannot measure. Build an offline evaluation pipeline using a golden dataset of 500–1,000 query-answer pairs from your domain. Track retrieval recall@5 (was the right passage in the top 5?), answer faithfulness (did the LLM use the retrieved context?), and answer correctness (is the answer right?). Deploy online monitoring that logs queries, retrieved contexts, and generated answers, with a sample going to human review. Set up automated alerts when retrieval recall drops below threshold — this usually signals data drift or a knowledge base update that broke existing queries.

Found this useful?

Share the key finding on X.