Production RAG

How I Build Production-Ready RAG Systems

A practical production RAG blueprint covering source governance, ingestion, hybrid retrieval, grounded answers, evaluation, and safe rollout.

March 28, 20264 min readRAGRetrieval-Augmented Generation

Most RAG demos work because the corpus is small, clean, and manually curated. Enterprise documents are different: multiple versions coexist, layouts break parsers, ownership is unclear, access varies by user, and critical answers may depend on a table or attachment that indexing silently discarded. The objective is not to make the model answer one example. It is to produce consistent, explainable behavior under changing data, latency, cost, and governance constraints. That requires treating retrieval-augmented generation as a complete information system rather than a prompt connected to a vector database.

Establish source scope before building retrieval

I begin with a source registry. For each source, it records the owner, authority, audience, access policy, refresh cadence, retention rule, and current status. This prevents archived or unofficial documents from competing with the approved answer. Scope resolution should happen before retrieval. User identity, business context, document status, language, and effective date can determine which corpus is valid. Applying those constraints only after semantic search risks exposing data or grounding an answer in the wrong version. The ingestion pipeline then needs observable stages:

collect the approved source and preserve its identity;
parse text, tables, structure, and attachments;
normalize metadata and access attributes;
split content according to document structure;
create searchable representations;
publish a versioned index and ingestion report.

Failures should be visible at document and section level. A successful pipeline run is not proof that a table was parsed correctly or that every expected document reached the index.

Design retrieval as a measurable ranking system

I evaluate retrieval before generation. If the evidence is missing, a stronger model usually produces a more convincing unsupported answer. A practical retrieval stack can combine:

metadata and permission filters;
lexical search for exact identifiers and domain terminology;
vector search for semantic similarity;
query rewriting for ambiguous or conversational requests;
reranking for better ordering of the candidate set;
diversity or parent-document expansion when fragments lose context.

The right configuration depends on the corpus. Legal policies, maintenance procedures, product catalogs, and financial reports have different structures and failure costs. Chunk size alone is not a sufficient retrieval strategy. For evaluation, I store real questions, expected answer points, required sources, and unacceptable claims. Metrics such as recall at k and ranking quality help diagnose search, while human review verifies whether the retrieved passages are complete and usable. The RAG evaluation framework explains how I separate retrieval, generation, citation, and operational signals.

Production RAG is a connected pipeline. Quality can fail before the model receives any context.

Constrain answer generation with evidence

The answer layer should receive traceable evidence and explicit response rules. I define how the system cites sources, handles conflicting documents, expresses uncertainty, and refuses when evidence is insufficient. A refusal is a valid outcome when the alternative is an unsupported claim. I distinguish between answer quality and evidence quality. A fluent response with an invalid citation is a failure. A correct statement supported by an obsolete document is also a failure. Citation checks should therefore verify that the referenced passage exists, is accessible to the user, and actually supports the claim. The system should also expose safe fallbacks: refine the query, ask a clarifying question, suggest an approved source, or route the request to a human owner. It must not silently answer from model memory when the product promises source-grounded behavior.

Observe the pipeline and classify failures

End-to-end traces connect the user request to scope resolution, query transformation, retrieved items, reranking, model calls, guardrails, citations, latency, and cost. This makes failures attributable:

source failure: the required content is absent, obsolete, or unauthorized;
parsing failure: structure, tables, or text were corrupted;
retrieval failure: valid evidence exists but was not selected;
synthesis failure: the model misread sufficient evidence;
policy failure: the system answered, refused, or escalated incorrectly;
runtime failure: latency, timeout, or integration behavior broke the journey.

This taxonomy prevents prompt changes from becoming the default response to every quality issue. It also clarifies ownership across content, data, retrieval, application, and platform teams. My public DAISI case study presents source-grounded answers, evaluation, security controls, and operational testing as distinct production concerns. The exact implementation remains specific to its environment, but the separation of responsibilities is broadly useful.

Roll out progressively and plan for corpus change

I start with a curated evaluation set and a read-only pilot. Ingestion and retrieval are validated before the answer layer. A small user group then supplies trace-linked feedback. Expansion depends on understood failure modes, stable access controls, acceptable latency and cost, and a tested fallback. Index releases need the same care as application releases. A parser, embedding model, chunking rule, or source change can alter behavior even when the prompt is unchanged. I version the corpus and retrieval configuration, compare candidate indexes, and keep a path to the last validated version. Tradeoffs remain. Hybrid retrieval improves coverage but adds tuning and runtime cost. Reranking can improve precision while increasing latency. More context can improve completeness while diluting relevant evidence. Aggressive freshness can publish poorly validated content. These choices should be made per workflow and measured against real questions. A production RAG system earns trust through evidence: known sources, observable ingestion, measurable retrieval, bounded generation, reproducible evaluation, and a rollout that can stop or reverse when the corpus changes unexpectedly.

Sources and references

Google Cloud: RAG-capable generative AI applicationReference architecture for ingestion, retrieval, generation, and application serving
LangChain semantic search and RAG tutorialOfficial guide to loaders, embeddings, vector stores, retrievers, and minimal RAG workflows

From principles to shipped systems

These articles document the methods behind my work. The project case studies show how I apply them across enterprise agents, RAG, and MLOps.

Explore project case studies View my CV

Continue exploring

Related field notes on the architecture, evaluation, and operating decisions behind production AI systems.

April 9, 20264 min readRAG EvaluationRAG