LLM Evaluation Dataset Design: Coverage, Slices, and Provenance

Design coverage before collecting rows

Turn coverage gaps into review work

A coverage review should answer three questions: which product decisions lack a case, which slices contain only easy examples, and which expected behaviors no longer match the product? Give each gap a next step and a reviewer. When requirements change, update them first, revise the affected cases, and preserve the earlier run for comparison. This prevents a quiet change in what the evaluation means. A dataset is representative only relative to an explicit deployment claim, so state that claim beside every release report.

Give every case a testable contract

Store a stable case ID, task slice, input or artifact reference, expected behavior, allowed alternatives, forbidden behavior, severity, grader, and reviewer. Also record origin, collection date, usage basis, edits, and review status. Provenance makes a disputed label repairable; it does not make the label automatically correct.

Use deterministic expected outputs for schemas, classifications, calculations, policy decisions, and tool invariants. Use rubrics for open-ended qualities, with human-reviewed anchor cases for calibration. The LLM-as-judge guide explains why a model grader should remain distinct from the case definition.

OpenAI's eval guidance treats data schema and testing criteria as separate ingredients. Preserve that separation in any framework so the same cases can be regraded when a rubric improves without rewriting their inputs.

Keep release cases separate

Use one group of cases during iteration and a different group for release checks. If a release case influences a change, reclassify it as an iteration case and add a new example that tests the same requirement.

Deduplicate by meaning and origin, not only exact text. Paraphrases of one record can carry the same answer across splits. Keep related conversations, documents, or events together. Version the split manifest so each result maps to an exact case set.

NIST's Generative AI Profile recommends representative in-context testing, documented data origin, and caution against extrapolating from narrow assessments. Use those as review prompts rather than a universal sampling formula.

Review coverage as a matrix of requirement, slice, examples, gaps, and review date. Quantity alone is weak evidence: many near-duplicates can add less confidence than one carefully judged boundary case. When a slice is too small for a stable summary, show the individual outcomes and uncertainty instead of hiding them in an overall score. This makes an empty or shallow slice visible before a release decision.

Dataset design failure cases

• Coverage by accident: many examples repeat an easy task. Name the required slices before choosing cases.
• Only happy paths: the suite never tests ambiguity, refusal, recovery, or policy boundaries. Add alternative and failure flows.
• Unexplained expectations: reviewers cannot resolve disagreements. Record the criterion and support behind each expected result.
• Iteration cases reported as proof: repeated tuning makes the score optimistic. Keep separate release cases.
• The same scenario in both groups: results look stronger than they are. Cluster related examples before assignment.
• One overall score: broad gains conceal a critical regression. Report each risk slice and use blocking cases.
• A frozen suite: new task patterns and incidents fall outside coverage. Add reviewed cases through a versioned process.

A dataset implementation sequence

1. Define the decision. Name the product behavior, deployment context, risk, and release question the dataset will measure.
2. Create the slice map. Cover tasks, input shapes, difficulty, users, failure modes, and action risk. Assign an owner to each slice.
3. Write case contracts. Add stable IDs, expected behavior, allowed alternatives, forbidden outcomes, severity, grader, and rationale.
4. Review origin and use. Confirm where each case came from, why it may be used, what changed, and who approved it.
5. Create iteration and release groups. Cluster related examples, assign groups together, and record the manifest version.
6. Validate graders. Run deterministic checks and calibrate rubric-based grades against human-reviewed anchors.
7. Maintain coverage. Promote confirmed failures through review, rerun prompt regression tests, and track gaps with production eval coverage.

Decision summary

Build evaluation cases from product decisions and observed failure slices, not from convenient examples. Record each case origin, keep release gates separate from iteration feedback, and add incidents only after they become stable, reviewable fixtures.