Acceptance Criteria for a Local Retrieval-augmented Knowledge System: What Must Be Proven

The direct answer

Write a test for “sources and access classes are inventoried” before execution and keep “restricted documents placed in a shared index” as a release-blocking counterexample.

For a local retrieval-augmented knowledge system, the relevant audience is teams that need answers grounded in owned documents while keeping the retrieval and model path inside their infrastructure. The decision should cover source inventory, access classification, parsing, chunking, indexing, retrieval, answer generation, citation checks, evaluation, and refresh. The supplied boundary starts with approved documents or data exports, access rules, answer use cases, and evaluation examples and ends with a local-model RAG system checked by a QA agent, presented in reviewable form.

Local deployment reduces some egress paths but does not make the data correct, the retrieval complete, or the answer safe. Access control, backups, logs, and operators remain part of the threat model.

Turn each requirement into a proof obligation

The expected deliverable is a local-model RAG system checked by a QA agent.

Use approved documents or data exports, access rules, answer use cases, and evaluation examples as the controlled starting material.

The evaluation owner adjudicates every pass, hold, or fail verdict against these registered statements.

Cover more than the happy path

The normal flow should establish whether “sources and access classes are inventoried” holds. An alternate flow should vary a permitted input while testing whether “retrieval permissions match source permissions” holds. The failure flow should use a fixture demonstrating “stale chunks surviving source deletion” and verify containment.

Add a recovery flow for “citations pointing to a relevant page but not the claim”.

Judge evidence quality and freshness

For a local retrieval-augmented knowledge system, a result from another environment cannot prove that “answers cite claim-level evidence” holds in the buyer's environment.

Define pass, hold, and fail before execution

Keep sign-off independent

The implementer may produce artifacts, but the evaluation owner should judge whether “adversarial documents are included in tests” holds against criteria written before the result was seen.

Record the business decision of the data owner, the technical evidence reviewed by the retrieval engineer, the acceptance verdict recorded by the evaluation owner, and residual risk accepted by the system operator.

A screenshot or self-score cannot prove that “adversarial documents are included in tests” holds under the failure condition “prompt injection entering through indexed documents”.

Reopen criteria when the system changes

Changes to source inventory, access classification, parsing, chunking, indexing, retrieval, answer generation, citation checks, evaluation, and refresh can invalidate a test even when the requirement text stays the same.

How the sources bound the acceptance criteria decision

For a local retrieval-augmented knowledge system, the live catalog limits the offer to two elements. The supplied boundary is approved documents or data exports, access rules, answer use cases, and evaluation examples. The catalog names the deliverable as a local-model RAG system checked by a QA agent. It cannot establish whether “sources and access classes are inventoried” holds in the buyer's environment.

Connect those narrow roles to a local fixture for “retrieval evaluated only by answer fluency” rather than treating citation status as a pass.

For a local retrieval-augmented knowledge system, limit the conclusion to the documented workflow and let the privacy owner retain the current source-to-claim map. Reopen the source judgment if the failure case “restricted documents placed in a shared index” changes the tested conditions.

Product-specific acceptance criteria review drills

These drills connect a local retrieval-augmented knowledge system to concrete inputs, failures, acceptance statements, and owners. For a local retrieval-augmented knowledge system, the drills map each criterion to a reviewable verdict.

Acceptance for a local retrieval-augmented knowledge system is judged against a boundary record covering approved documents or data exports, access rules, answer use cases, and evaluation examples, never live protected material. The data owner requires synthetic, non-secret cases; messages, writes, state changes, and all other external effects stay inside the fixture throughout and after each case.

Requirement trace

Use the requirement trace review to examine what follows from the failure case “restricted documents placed in a shared index”. Before intervention, the data owner retains the observable handoff.

Use an authorized test case within the boundary covering approved documents or data exports, access rules, answer use cases, and evaluation examples to establish whether “retrieval permissions match source permissions” holds. Record configuration and reviewer identity beside the result.

The evaluation owner resolves the drill with one finding about “retrieval permissions match source permissions”. For a local retrieval-augmented knowledge system, the deliverable decision in the requirement trace review advances only when that finding is supported. In the requirement trace review, evidence for “retrieval permissions match source permissions” maps support to pass, contradiction to fail, and unresolved to hold.

Reopen this drill after a change to “restricted documents placed in a shared index”, the input class, or the authority held by the data owner.

Normal-flow result

Ask how the normal-flow result review handles the failure case “retrieval evaluated only by answer fluency”. The privacy owner freezes the local portion of source inventory, access classification, parsing, chunking, indexing, retrieval, answer generation, citation checks, evaluation, and refresh before drawing a conclusion.

Bind the fixture to a scope record covering approved documents or data exports, access rules, answer use cases, and evaluation examples; its expected condition is that “deletion and refresh propagate to the index” holds. The fixture version is part of the receipt.

The evaluation owner records pass only for “deletion and refresh propagate to the index”. Any wider claim about a local-model RAG system checked by a QA agent stays outside the drill. In the normal-flow result review, evidence for “deletion and refresh propagate to the index” maps support to pass, contradiction to fail, and unresolved to hold.

The result expires when the workflow boundary for source inventory, access classification, parsing, chunking, indexing, retrieval, answer generation, citation checks, evaluation, and refresh no longer follows the tested path or when evidence for “deletion and refresh propagate to the index” cannot be replayed.

Alternate-flow result

Reproduce a safe case involving “stale chunks surviving source deletion” as the entry condition for the alternate-flow result review. The retrieval engineer preserves the last state that the workflow can prove.

Let the system operator inspect a scope record covering approved documents or data exports, access rules, answer use cases, and evaluation examples and the evidence for “sources and access classes are inventoried”. For a local retrieval-augmented knowledge system, the alternate-flow result review cannot rely on a demonstration selected after execution.

The evaluation owner links the finding “sources and access classes are inventoried” to go, revise, or stop in the decision record. It does not treat completion of a local-model RAG system checked by a QA agent as proof of every outcome. In the alternate-flow result review, evidence for “sources and access classes are inventoried” maps support to pass, contradiction to fail, and unresolved to hold.

Expire the disposition if the retrieval engineer cannot reproduce the case for “stale chunks surviving source deletion” under the recorded authority.

Failure-flow result

Create the failure-flow result review scenario from a safe case involving “citations pointing to a relevant page but not the claim”. The system operator records the affected portion of source inventory, access classification, parsing, chunking, indexing, retrieval, answer generation, citation checks, evaluation, and refresh before intervention.

Give the system operator an authorized, read-only boundary record covering approved documents or data exports, access rules, answer use cases, and evaluation examples plus the criterion “answers cite claim-level evidence”. Their receipt identifies any missing proof.

When evidence supports “answers cite claim-level evidence”, the evaluation owner can close the failure-flow result review. Contradictory evidence fails the drill; stale evidence keeps it open. In the failure-flow result review, evidence for “answers cite claim-level evidence” maps support to pass, contradiction to fail, and unresolved to hold.

Reopen this result after a change to the input, the authority of the system operator, or the workflow condition represented by “citations pointing to a relevant page but not the claim”.

Independent verdict

During the independent verdict review, reproduce a safe case involving “prompt injection entering through indexed documents”. The system operator records what remains observable before the next role acts.

The data owner receives a boundary record covering approved documents or data exports, access rules, answer use cases, and evaluation examples with an explicit request to verify whether “adversarial documents are included in tests” holds. Input identity and judgment stay in the same receipt.

The evaluation owner advances only when the receipt establishes “adversarial documents are included in tests”. Missing proof keeps a local-model RAG system checked by a QA agent on hold; contradictory proof makes the evaluation owner record fail. In the independent verdict review, evidence for “adversarial documents are included in tests” maps support to pass, contradiction to fail, and unresolved to hold.

Keep a reopen event for new authority, stale evidence, or a changed consequence associated with “prompt injection entering through indexed documents”.

Evidence expiry

Stage a safe instance of “restricted documents placed in a shared index” inside an authorized fixture for the evidence expiry review. The data owner notes the last trusted state in source inventory, access classification, parsing, chunking, indexing, retrieval, answer generation, citation checks, evaluation, and refresh.

Link the evidence expiry review to a scope record covering approved documents or data exports, access rules, answer use cases, and evaluation examples and the proof target “retrieval permissions match source permissions”. The retained record identifies both versions.

The evaluation owner limits acceptance to “retrieval permissions match source permissions” and nothing beyond it, leaving a named hold for any unsupported part of a local-model RAG system checked by a QA agent. In the evidence expiry review, evidence for “retrieval permissions match source permissions” maps support to pass, contradiction to fail, and unresolved to hold.

Do not carry this verdict into a changed workflow, input class, or response to “restricted documents placed in a shared index”; create a new bounded record.

Frequently asked question

What acceptance criteria should I use for Private AI Brain?

Require observable evidence that sources and access classes are inventoried and include “restricted documents placed in a shared index” as a negative case. The evaluation owner should record pass, hold, or fail before expansion.

A product bridge, with a boundary

The Private AI Brain is the relevant sincLLM offer for this narrow problem. The frozen live catalog describes its required boundary as approved documents or data exports, access rules, answer use cases, and evaluation examples and its deliverable as a local-model RAG system checked by a QA agent. The buyer must judge fit and results in its own environment; the catalog does not certify compliance, safety, or technical sufficiency.