Production RAG

RAG Evaluation Framework: Metrics That Explain Production Quality

A layered RAG evaluation framework for datasets, retrieval, grounded answers, citations, operations, release decisions, and production feedback.

April 9, 20264 min readRAG EvaluationRAG

Testing a RAG system with a few happy-path questions and a subjective “looks good” review cannot support a production release. A final answer may be wrong because the source is absent, parsing destroyed a table, retrieval selected the wrong passage, the model ignored good evidence, or the response policy allowed an unsupported claim. A useful evaluation framework isolates these layers. Its purpose is not to produce one impressive score. It should reveal where quality changed, make releases comparable, and turn real failures into reproducible cases.

Build the dataset around real information needs

I begin with representative user questions rather than generated variants alone. For every example, the dataset should capture:

the raw request and relevant user context;
the intended source scope and access constraints;
required documents or evidence passages;
the main answer points;
claims that must not appear;
acceptable refusal or clarification behavior;
intent, difficulty, language, and risk labels.

The source snapshot matters. If documents change while the same evaluation is rerun, the comparison is ambiguous. Dataset version, corpus version, parsing configuration, and retrieval configuration should therefore be linked to every experiment. Synthetic examples can improve coverage for rare conditions, but they should not replace real terminology, ambiguity, and document defects. I use them to extend a grounded core dataset, then review whether they represent plausible requests and expected behavior.

Evaluate retrieval before generation

Retrieval evaluation asks whether the system found useful evidence and ranked it high enough for the answer layer. Common metrics include:

Recall at k: whether at least one required source appears in the candidate set;
Precision at k: how much of the selected context is relevant;
MRR: how early the first relevant result appears;
nDCG: how well the complete ranking reflects graded relevance;
filter correctness: whether access, status, language, and date constraints were applied.

The metric choice depends on the workflow. A question requiring one exact policy paragraph differs from a comparison requiring evidence across several documents. Document-level relevance may also be too coarse when the required fact lives in one section. Metrics need qualitative trace review. A chunk can be labeled relevant while lacking the heading or table context needed for interpretation. Conversely, a different passage may support the same answer. Evaluation should allow multiple valid evidence sets where the domain permits them.

Evaluate grounded answers and citations separately

Once retrieval is understood, I assess:

Dimension	Evaluation question
Groundedness	Is every material claim supported by the supplied evidence?
Completeness	Does the answer cover the required points without hiding uncertainty?
Citation validity	Does each citation point to an accessible passage that supports the claim?
Policy behavior	Did the system clarify, refuse, or escalate when evidence was insufficient?
Presentation	Is the answer usable for the intended workflow and audience?

Deterministic checks should cover citation existence, source access, required fields, and prohibited outputs. Rubric-based evaluators can help with groundedness and completeness, but they need calibrated examples, evaluator versioning, and periodic human review. A model judge is another component that can drift or disagree.

Evidence attached to a release candidate

Turn metrics into risk-based release decisions

There is no universal groundedness or recall threshold that makes every RAG system safe. A release gate should be established from a validated baseline, the importance of each intent, the cost of a wrong answer, and the noise of the evaluator. I use three categories:

Hard blockers for authorization failures, inaccessible citations, prohibited claims, and critical unsupported answers.
Segmented comparisons for important intents, languages, source families, and difficult questions.
Trend review for quality, latency, cost, fallback, and coverage movements that require context.

Every failed gate should link to examples and traces. The team then needs to determine whether the root cause belongs to source scope, ingestion, parsing, retrieval, synthesis, policy, or the evaluator itself. My RAG Equity Research Agent case study illustrates how hybrid retrieval, reranking, citations, and multi-step synthesis create several independently testable surfaces. The evaluation policy for any production system must still reflect its own users and risk.

Close the loop with production evidence

Offline evaluation protects known behavior. Production reveals new formulations, changing sources, access conditions, and integration failures. Useful signals include trace-linked feedback, repeated questions, fallbacks, source misses, invalid citations, expert corrections, latency, and cost per successful outcome. Samples should be curated before they enter the regression set. Sensitive information must be protected, duplicates consolidated, and expected behavior reviewed. The production RAG guide covers the wider ingestion, retrieval, observability, and rollout architecture that produces these signals.

Failure modes, tradeoffs, and rollout

Evaluation can create false confidence. A dataset may overrepresent easy questions. Relevance labels may be incomplete. Aggregate scores can hide one critical segment. Model judges may prefer verbosity or share the same blind spots as the system under test. Production feedback can be biased toward highly engaged users. I start with a small, carefully reviewed dataset for the highest-value workflows. Retrieval metrics and deterministic policy checks come first. Groundedness and completeness rubrics are calibrated with human examples. The suite initially informs release review before stable checks become blocking. Production failures are then added with root-cause labels. The strongest framework does not claim that a single score represents quality. It preserves enough context to answer four questions: what failed, in which segment, in which layer, and what evidence proves that the correction works?

Sources and references

RAGAS documentationEvaluation concepts for retrieval context, grounded generation, and experiment workflows
LangSmith evaluation documentationDatasets, experiments, evaluators, comparative analysis, and trace review

From principles to shipped systems

These articles document the methods behind my work. The project case studies show how I apply them across enterprise agents, RAG, and MLOps.

Explore project case studies View my CV

Continue exploring

Related field notes on the architecture, evaluation, and operating decisions behind production AI systems.

March 28, 20264 min readRAGRetrieval-Augmented Generation