AI Agent Evaluation Checklist: What to Test Before Production
An agent is ready only when it reaches the right end state through permitted actions, preserves state boundaries, recovers from failure, and stops when it should. Score the outcome and the trajectory; either one can fail while the other looks convincing.
The production AI agent evaluation checklist
Write each row as an assertion with a fixture, observable evidence, and a clear failure. The expected evidence should come from the end state, the tool trace, or both—not from the agent's own explanation of what it did.
| Evaluation area | Pass assertion | Evidence to capture | Failure probe |
|---|---|---|---|
| Goal completion | The requested end state exists and matches the acceptance criteria. | Artifact, record, response, and independent state readback. | Return a fluent final message before the required artifact exists. |
| Tool selection | The agent chooses an allowed tool that can perform the action and avoids redundant calls. | Tool name, decision reason, call order, and call count. | Offer a plausible but irrelevant tool beside the correct one. |
| Tool arguments | Arguments satisfy the tool contract and come from grounded task data. | Validated argument object with sensitive values redacted. | Remove a required field or introduce an ungrounded identifier. |
| Permission boundary | Forbidden actions are blocked before execution and risky actions pause at the required approval point. | Policy decision, denied call, approval event, and unchanged state. | Ask the agent to cross an account, tenant, role, or production boundary. |
| State and memory | Task state is scoped, necessary, and correct across retries; unrelated state stays unchanged. | Before/after state diff, memory reads, memory writes, and provenance. | Repeat the same event or inject stale state from another task. |
| Recovery and retry | Transient failures use bounded retries; terminal failures preserve a safe, explainable state. | Error class, attempt history, backoff decision, rollback, and final status. | Return timeouts, partial writes, duplicate events, and malformed tool responses. |
| Stop and escalation | The agent stops on success, budget exhaustion, policy denial, or unresolved uncertainty. | Stop reason, remaining work, and human handoff context. | Make the goal impossible or keep a dependency unavailable. |
| Output quality | The final response is correct, grounded, appropriately uncertain, and consistent with the observed state. | Reference answer, rubric score, citations, and contradiction checks. | Provide conflicting evidence or an incomplete result with confident wording. |
| Operating envelope | Latency, model usage, tool calls, and spend remain within the product's declared budget. | End-to-end timing, token usage, tool latency, retries, and estimated cost. | Create a looping path or a slow dependency that should trigger fallback. |
| Repeatability | Repeated runs preserve hard invariants even when the reasoning path varies. | Pass distribution, invariant violations, and trajectory clusters across repeated runs. | Run paraphrases, reordered inputs, and controlled model variation. |
Evaluate four layers, not one score
1. Component and contract checks
Test tool schemas, argument validation, policy decisions, state transitions, and deterministic business rules without asking a model to judge them. These checks should fail on a concrete contract violation: wrong type, missing field, denied action, duplicate write, or broken invariant.
2. Trajectory checks
Inspect the sequence of model decisions and tool calls. A correct outcome reached through an unauthorized call, accidental data exposure, or unbounded retry is still a failed run. Conversely, an alternate valid trajectory should not fail merely because it differs from one golden trace.
3. End-state and output checks
Read the system after the run. Confirm that the required artifact or record exists, unrelated state did not change, and the final response accurately describes the result. Use deterministic assertions where possible; use a rubric-based model judge only for qualities that cannot be expressed as invariants, and calibrate that judge against human-reviewed examples.
4. Adversarial and operational checks
Inject denied requests, prompt injection, unavailable tools, malformed responses, partial writes, conflicting instructions, stale memory, and budget exhaustion. The OWASP AI Agent Security Cheat Sheet provides a primary security reference for threats such as excessive agency, tool abuse, memory poisoning, and unsafe inter-agent communication.
Common evaluation failure cases
- Outcome-only scoring: the agent reaches the right answer through a prohibited or fragile path. Keep trajectory and boundary assertions beside task success.
- Exact-trace scoring: a valid alternate path fails because it does not match one recorded sequence. Assert necessary invariants and forbidden actions instead.
- Mocks that hide contract drift: a tool double accepts arguments or responses the real interface rejects. Keep contract fixtures aligned with the owned tool schema.
- One stochastic run: a single pass hides intermittent loops and rare boundary violations. Repeat representative cases and report the distribution of hard failures.
- The model judges itself: the same failure can shape both the output and its grade. Prefer state readback, validators, and independently calibrated rubrics.
- No negative permission tests: the suite proves allowed actions work but never proves forbidden actions stay blocked. The agent safety guide outlines the guardrails that need explicit denial cases.
- Evaluation touches live systems: an adversarial probe becomes a real side effect. Use synthetic fixtures and isolated environments for all staged checks.
A practical implementation sequence
- Write the end-state contract. Define the required artifact, allowed side effects, unchanged resources, approval points, and stop conditions before writing cases.
- Define a trace contract. Capture run ID, model and prompt version, decision events, tool names, redacted arguments, tool results, policy decisions, retries, state mutations, timing, usage, and stop reason.
- Build normal, alternative, and failure fixtures. Include successful tasks, ambiguous goals, denied actions, missing dependencies, malformed responses, partial completion, duplicate events, and explicit human escalation.
- Assert hard invariants first. Validate schemas, permissions, state changes, idempotency, and stop behavior with code. These are release gates, not model opinions.
- Add rubric grading where judgment is unavoidable. Define dimensions such as correctness, evidence use, and calibrated uncertainty. Review disagreements and retain human-approved examples for calibration.
- Repeat and segment runs. Group failures by task type, tool, model version, prompt version, and failure category. Averages alone can hide a rare but severe permission breach.
- Promote incidents into regression cases. Every observed failure should add a fixture, assertion, or coverage category. Use the production eval coverage guide to track what the suite still does not exercise.
Primary and official sources
- OpenAI: A Practical Guide to Building AI Agents — official guidance on agent components, orchestration, guardrails, and human intervention.
- OWASP AI Agent Security Cheat Sheet — primary security guidance for agent threats and controls.
- NIST AI Resource Center and the NIST Generative AI Profile — official risk-management resources for measurement and governance.