Prompt Regression Testing: Cases, Graders, and Release Gates
Treat every prompt change as a behavior change. Run the candidate and baseline on the same versioned cases, grade hard requirements deterministically, use calibrated rubrics only for judgment, and block release on critical failures even when the average improves.
A prompt regression case contract
| Field | What to record | Why it matters | Failure signal |
|---|---|---|---|
| Case identity | Stable ID, task slice, severity, and owner. | Supports history and slice reporting. | A case changes meaning without a new version. |
| Runtime | Model, revision, settings, tool contracts, and retrieval version. | Separates prompt effects from other changes. | Baseline and candidate run under different conditions. |
| Expected behavior | Required outcome, allowed alternatives, and forbidden behavior. | Defines what pass means before output exists. | A grader invents criteria after seeing a result. |
| Graders | Schema checks, exact assertions, rubric version, and human review rule. | Matches evidence type to measurement. | One subjective score replaces hard checks. |
| Comparison | Baseline grade, candidate grade, changed invariants, and concise reason. | Shows the behavioral delta. | Only the candidate score is retained. |
| Decision | Threshold, critical-case rule, reviewer, and disposition. | Turns evaluation into a release gate. | A failing case is waived without ownership. |
OpenAI's evaluation guide frames evals as a way to test outputs against specified style and content criteria. The engineering step is to put that run inside change control: compare with a pinned baseline, preserve case-level results, and make the gate explicit.
Layer graders from hard facts to judgment
Run parsers, schemas, exact labels, required fields, citations, policy assertions, and tool invariants before any model grader. These failures have objective evidence. For qualities such as completeness or clarity, use a task-specific rubric calibrated against human-reviewed examples. Keep the grader version separate from the prompt version so a grading change does not masquerade as a product improvement.
Review changed cases, not only summary scores. Capture baseline and candidate outputs, criterion results, tool trajectory, and end-state evidence. A prompt can improve the final prose while choosing the wrong tool or crossing a permission boundary. The AI agent evaluation checklist explains why trajectory and outcome need separate assertions.
Use thresholds and critical cases together
A release policy can allow bounded movement in an aggregate metric while declaring specific cases non-negotiable. Examples include unauthorized actions, unsupported material claims, wrong tenant access, invalid structured output, or failure to stop. Define these categories before running the candidate.
For noisy rubric scores, repeat cases where necessary and report uncertainty. Do not tune a threshold until a desired candidate passes. If a change intentionally alters behavior, update the requirement, case, and baseline through review rather than deleting the failure.
Attach run and trace identifiers so an operator can inspect the changed path. OpenTelemetry's semantic conventions help standardize operation metadata, while production observability connects offline regressions to runtime signals.
Separate prompt effects from system effects
A prompt comparison can attribute a change only when the surrounding runtime is controlled. Pin the model revision, decoding settings, retrieval snapshot, tool contracts, context assembly, and grader versions. If one of those must change in the same release, run additional comparisons that isolate it. Otherwise, a better score may be credited to prompt wording when the real cause was a different model or evidence path.
Produce a case-level change report, not just a dashboard number. For every changed outcome, show the slice, criterion, severity, baseline result, candidate result, and review decision. Group similar failures so one root cause is not counted as many unrelated defects. This report gives the release owner a concrete choice: accept the intended behavior, repair the candidate, narrow the rollout, or restore the prior prompt while the cause is investigated.
Prompt regression failure cases
- Changing prompt and model together: the result cannot identify the cause. Isolate changes or run a controlled comparison matrix.
- Golden-string testing: a valid paraphrase fails while a factual error passes. Assert meaning, structure, and invariants at the right layer.
- Candidate-only scoring: a pass has no reference point. Run the baseline on the same case and grader versions.
- Average-only gates: a critical case disappears inside a mean. Add severity and release-blocking assertions.
- Grader drift: a new rubric changes scores for both systems. Regrade the baseline before comparing.
- Flaky failure ignored: intermittent loops or unsafe calls are dismissed as noise. Repeat, preserve evidence, and investigate hard violations.
- Deleting inconvenient cases: the suite becomes easier after a change. Update requirements through review and retain history.
A CI-ready implementation sequence
- Freeze the requirement. Define expected behavior, risk tiers, accepted alternatives, and critical failures before editing the prompt.
- Version the test inputs. Record case manifest, baseline prompt, candidate prompt, model configuration, graders, and owned dependencies.
- Run the baseline. Preserve outputs, traces, criterion results, and failures under the same runtime used for the candidate.
- Run the candidate. Use stable settings where possible and repeat cases whose stochastic variation matters.
- Apply layered graders. Run deterministic assertions first, then calibrated rubrics and required human review.
- Enforce the gate. Fail on any critical regression and on declared threshold breaches. Produce a case-level difference report.
- Review and archive. Approve intentional changes, add incidents as new cases, and retain rollback information. Use the dataset design guide to preserve coverage quality.
Decision summary
Treat a prompt change like a behavior release: run versioned representative cases, deterministic checks first, calibrated graders second, and critical-case gates before aggregates. Record the model and surrounding system so a prompt is not blamed for unrelated drift.
Primary and official sources
- OpenAI: Working with evals — official task, test-data, grader, run, and analysis concepts.
- OpenTelemetry semantic conventions — official operation metadata used to correlate regression evidence with traced runs.