RAG Evaluation Metrics: Retrieval, Grounding, Completeness

A metric map with a repair owner

Measure retrieval before judging prose

When relevance judgments exist, compare the retrieved documents or chunks with known good evidence. Inspect whether required items appear, how they are ordered, and whether irrelevant or duplicate passages crowd the useful region. Segment by question type, corpus, document format, language, tenant policy, and freshness requirement.

The Microsoft Foundry RAG evaluator documentation explicitly separates document retrieval and textual retrieval evaluation from final-response evaluation. It describes ground-truth document judgments for search metrics and a context-based retrieval evaluator when labels are unavailable. Treat model-judged retrieval as diagnostic evidence, not a substitute for reviewed relevance on critical flows.

Context sufficiency is not the same as relevance

A passage can be relevant but insufficient. A policy summary may answer the topic while omitting the exception; one table row may lack its unit or column header. Label the minimum evidence bundle needed for each case, then verify that context assembly preserves it after deduplication and truncation.

Separate groundedness, completeness, and usefulness

Groundedness asks whether claims stay within the supplied context. Completeness asks whether the answer omits critical expected content. Relevance asks whether it addresses the question. A terse answer can be grounded yet incomplete; a comprehensive answer can be relevant yet introduce an unsupported claim. These are independent failure modes.

The RAG evaluation survey by Yu and colleagues reviews retrieval and generation metrics and emphasizes the hybrid nature of RAG evaluation. Use deterministic checks for citations, required fields, forbidden claims, and reference IDs where possible. Use rubric-based human or model judgment only for qualities that truly require interpretation, and calibrate that judgment against reviewed anchors.

Design cases around failure coverage

A useful evaluation corpus includes normal lookup, paraphrase, multi-hop synthesis, conflicting revisions, missing evidence, table interpretation, authorization denial, stale content, ambiguous questions, and requests that should trigger abstention. Keep provenance for every question, expected evidence, reference requirement, and reviewer decision.

Hold out cases from tuning and preserve incidents as regression fixtures. When a production failure appears, classify the earliest broken boundary rather than merely saving the bad final answer. The companion AI agent evaluation checklist applies the same end-state and trajectory discipline when retrieval is embedded in a tool-using workflow.

Evaluation failure cases

• One blended score: improvement hides a retrieval regression behind better writing. Keep stage metrics visible.
• Answer-only fixtures: no expected evidence is recorded, so a plausible unsupported answer passes. Label documents and claims.
• Judge without anchors: rubric scores drift across model or prompt versions. Compare against human-reviewed examples and record versions.
• No abstention cases: the suite rewards answering even when evidence is absent. Include unanswerable and permission-denied questions.
• Freshness blindness: the correct old policy passes despite a newer controlling revision. Record effective dates and deletion behavior.
• Averages erase critical slices: common easy questions outweigh rare high-risk failures. Use explicit release gates per critical slice.

A practical evaluation sequence

1. Define the task contract. Name the reader goal, required evidence, allowed sources, freshness, authorization, citation, and abstention behavior.
2. Capture pipeline evidence. Record query, filters, candidates, ranks, selected context, response, citations, versions, timing, and errors.
3. Write deterministic assertions. Check eligibility, known document presence, schema, required facts, forbidden claims, and citation identifiers.
4. Add reviewed rubrics. Score relevance, groundedness, completeness, and usefulness independently with clear anchors.
5. Segment the results. Report by question shape, document type, corpus, risk tier, and release version rather than one global average.
6. Assign repair ownership. Route failures to ingestion, authorization, retrieval, ranking, context, generation, or product policy.
7. Gate and monitor releases. Re-run held-out and incident cases, then connect online traces to the same failure taxonomy.

Decision summary

Separate retrieval recall and ranking from groundedness, completeness, and answer usefulness so each failure has an owner. A single end-to-end score hides the repair path; sliced metrics plus representative cases make the next engineering action visible.