LLM Red-Team Checklist: From Threat Model to Regression

The minimum red-team coverage matrix

Write scope and rules before testing

Name the system version, included interfaces, excluded live systems, test window, owners, stop conditions, and allowed techniques. Use synthetic fixtures and isolated accounts. Define how a tester reports an unexpected side effect and who can halt the exercise. A security test should not become an uncontrolled production incident.

Start from assets and authority: user information, business records, tools, retrieval stores, model context, downstream actions, and availability. Then map attacker-controlled entry points and the control expected to contain each one. Include ordinary users, malicious users, compromised content, faulty tools, and accidental misuse.

NIST's Generative AI Profile describes AI red teaming as an evolving practice often conducted in controlled settings. It also notes that team expertise and deployment context affect findings and that results need further analysis before governance decisions. Build a mixed team of security, domain, product, safety, and operations reviewers accordingly.

Capture evidence and triage by consequence

For each case, record an ID, attack class, channel, preconditions, safe test fixture, expected defense, actual outcome, trace reference, side effects, reproducibility, and impact. Preserve concise evidence that another tester can replay. Do not keep unnecessary user content in the red-team record.

Triage false positives separately from successful attacks. A fluent unsafe-looking answer may have no consequence, while a quiet tool call may alter state. Confirm the end state and authority boundary. Rank remediation by impact, reachability, repeatability, and control weakness rather than by how dramatic the prompt appears.

Fix the control layer that failed: authorization, tool design, retrieval policy, output handling, loop budget, or review gate. A phrase-level refusal patch is acceptable only when the failure is genuinely response policy. Retest the original case, nearby variants, and normal tasks to detect safety-utility regressions. Link confirmed injection paths to the indirect prompt injection guide.

Close findings with owned retests

Give every finding a status, impact statement, responsible owner, planned control change, and retest date. A risk accepted for one release needs a stated reason, compensating controls, and an expiry for reconsideration; “known issue” is not a terminal state. Closure requires the original case to stop reproducing, nearby variants to remain contained, and normal tasks to retain acceptable utility. Keep the finding open when evidence is ambiguous or the repair only changes the demonstrated wording.

Red-team program failure cases

• Prompt collection without a threat model: many variants test one surface while tools, retrieval, and output handling remain untouched.
• Testing only the base model: the deployed application adds authority and failure paths. Exercise the complete system.
• Real side effects in testing: an adversarial case harms users or production. Isolate fixtures and define a stop procedure.
• Success means one refusal: an attack can route through tools or retrieved content. Assert the trajectory and end state.
• Severity by novelty: a clever prompt outranks a reproducible high-impact flaw. Triage consequence and reachability.
• Patch the wording only: the same weakness returns through another channel. Repair the underlying control.
• No regression case: a finding is forgotten after remediation. Add it to a versioned suite and rerun it at release.

A red-team implementation sequence

1. Define the target. Pin application, model, prompt, retrieval, tool, and policy versions and name the owner.
2. Threat-model the system. Map assets, entry channels, trust boundaries, authority, consequences, and expected controls.
3. Set engagement rules. Agree on scope, safe examples, limits, stop criteria, and reporting route.
4. Build the case matrix. Cover direct and indirect injection, information boundaries, tool abuse, unsafe outputs, retrieval tampering, and resource exhaustion.
5. Execute and observe. Record actual behavior, policy decisions, tool trajectory, end state, and bounded usage.
6. Triage and repair. Reproduce findings, rank impact, assign owners, fix the failed control, and document residual risk.
7. Retest and retain. Run original and neighboring cases, check normal utility, and add confirmed findings to prompt regression testing and the agent evaluation suite.

Decision summary

A red-team exercise earns release value when its scope maps to consequences, every finding has reproducible evidence, and repaired cases become regression tests. Unowned findings or one-time demonstrations are signals to pause, not proof that the system was tested.