How to Test a RAG System: A Small Dataset and Scoring Walkthrough

Your document assistant gives the wrong answer. Should you fix retrieval or change how the model uses its sources? If you cannot tell which step failed, another prompt revision may leave the actual problem untouched.

Test retrieval, answer quality and access rules separately. Then inspect whether the complete response meets the user's task. That gives you a debugging decision: find the missing source, correct an unsupported answer or stop information reaching someone who cannot access it. A single average can hide those differences.

Work through the six-case fixture pack to see the checks in practice. It includes fictional documents, expected behavior and deliberately imperfect example outcomes you can inspect by hand. This is a teaching dataset, not a benchmark of any model.

Define the answer you would accept

Retrieval-augmented generation, or RAG, gives a model selected external material to use when answering. That material still needs to be relevant, current and appropriate for the requesting user.

Write acceptance rules before testing. For a simple internal policy assistant, an acceptable response might need to answer the question, support material claims with eligible sources, respect access restrictions and acknowledge missing information.

Do not confuse correctness with grounding. An answer can faithfully repeat an outdated document and still be wrong for today’s policy. It can also contain a correct fact that the retrieved material does not support.

LangChain’s RAG evaluation tutorial distinguishes answer correctness, answer relevance, grounding in retrieved documents and retrieval relevance. Those checks compare different parts of the system; none is a substitute for the others. Evaluate a RAG application.

Create a tiny corpus with known boundaries

For this example, imagine a fictional company with these records:

The amounts and policies are invented for this walkthrough. The private document exists to test a boundary, not to demonstrate that a real customer’s information was accessed.

Give documents stable identifiers and explicit metadata. In your own system, retain version, status and access information with the text. A highly similar archived paragraph should not become eligible merely because it matches the question well.

Apply authorization before restricted text enters the model context. Filtering the final answer after exposing the source to the model does not test the same boundary. For implementation considerations, see RAG metadata filtering and access control.

Write six questions with expected behavior

Keep the first set small enough that a subject-matter reviewer can explain every label.

R4 deliberately asks for information the corpus does not contain. R5 is an access test. Neither should be scored as successful just because the system produces a fluent answer.

For each case, save the requesting identity, required source IDs, forbidden sources, expected facts, expected abstention behavior and the reason for those expectations. Do not rely on a gold answer string alone: valid wording can vary.

Capture enough evidence to diagnose a failure

Retain the query, authorization context, retrieved document IDs and versions, final model context, answer and citations. Also retain the retriever configuration, model identifier, prompt version and relevant settings.

Inspect the context actually sent to the model. A search response can look correct while a later context-building step drops the most useful paragraph or adds a restricted one.

You can begin with human grading. Separate observable checks, such as whether a forbidden document appeared, from judgment calls about completeness. If you later add an automated judge, compare its decisions with reviewed examples and inspect disagreements. A model-generated score is another component to evaluate.

Score a deliberately imperfect run

Suppose the example outcomes are:

Three of six cases pass: 50% end-to-end success on this constructed example. This is arithmetic over invented outcomes, not an observed production accuracy rate.

Now separate the layers. Among the four answerable cases, R1, R2, R3 and R6, the complete required source set is present in three. That is 3/4, or 75%, complete-source coverage under this specific definition. It is not a general-purpose retrieval recall metric.

R6 shows why retrieval coverage does not establish answer correctness. The model had both sources but still produced an incorrect fact. R3 needs a retrieval/eligibility investigation. R5 needs an authorization investigation and cannot be made acceptable by improving the average score.

The included scoring script reproduces these counts from the fixture’s explicit review labels and retrieved IDs. It does not independently determine the truth of arbitrary natural-language answers.

Choose metrics that change what you do

For a retrieval experiment, measure whether eligible evidence for each answerable question appears within the selected top results. State the cutoff and how you define a relevant document or passage.

For the answer, record supported facts, incorrect or unsupported claims, missing parts and citation quality. A citation should support its attached claim; simply attaching a link does not establish grounding.

For the complete workflow, track accepted responses, appropriate abstentions and access failures separately. Keep latency and operating cost alongside quality so a larger context window cannot hide an unacceptable cost or response-time tradeoff.

Choose release rules for the risk of your use case. In this example, any forbidden-document exposure blocks release even if aggregate answer quality improves. That is the proposed policy for the exercise, not a threshold established for every RAG application.

Fix one layer and rerun the same cases

If an archived policy wins retrieval, inspect eligibility metadata and filtering before rewriting the answer prompt. If the correct source is present but the answer invents a fact, inspect the instructions, context presentation and generation behavior. If the source itself is ambiguous, correct the source or define an escalation path.

Keep the original failing case after the repair. Add related variations, and retain a separate holdout set you do not continually tune against. A six-case pack helps explain a mechanism; it cannot represent all users, document types, languages or future failures.

Run repeated trials where variation matters. Record the denominator, failure examples and versions each time. A single successful rerun establishes that trial’s outcome, not reliable behavior across all similar requests.

Start with questions your users already ask

Take a small set of redacted real questions and have a knowledgeable person define the acceptable answer and eligible evidence. Include at least one unanswerable question, one outdated-source trap and one permission boundary.

Run them through the complete application and inspect the failures together. That conversation is more useful than debating a dashboard score without knowing what it measures.

If you want help building the evaluation set or tracing retrieval and authorization failures, review Private AI & Knowledge Systems. A small document sample and a few known failure cases are enough to begin scoping the work.