RAG in Production 2026: Chunking, Retrieval, and Evaluation That Works

Architecture That Survives Week Two

Production RAG is 20% prompting and 80% data pipeline. Ingest with structure-aware chunking (headers, tables, slides), embed with a model matched to your domain language, store in a vector DB with metadata filters (tenant, doc type, date), and retrieve with hybrid search — vector plus BM25 — before the LLM ever sees a token.

Re-embed on schedule, not on hope. Contracts and policies change; stale retrieval is the top cause of 'the bot lied' escalations.

Evaluation Before and After Launch

Build a golden set of 50-200 question-answer pairs from real user tickets. Measure retrieval recall@k, answer faithfulness, and citation accuracy weekly. LLM-as-judge helps at scale; human review on failures weekly.

• Chunk size 400-800 tokens for prose; row-level for tables
• Metadata filters beat bigger embeddings for multi-tenant SaaS
• Require citations in UI — users trust and verify
• Shadow mode two weeks minimum before customer-facing launch

Cost Controls

Cache embeddings, compress context, route simple queries to small models, and cap max chunks per request. A poorly tuned RAG system can cost more than human review.