AI Rollback Playbook: What Happens When You Have to Revert a Model Update at 2 AM

The 2 AM Scenario (Why Most Teams Are Not Ready)

A vendor ships a model update during a maintenance window. No advance notice arrives. Within the hour, structured outputs start failing schema validation, P95 latency climbs, and a customer-facing workflow begins returning errors. The on-call engineer is paged. When they call vendor support, the answer is: "Rollback to the prior model version is not available as a self-service action."

This is a pattern that appears in production AI systems that lack a rollback runbook. The team recovers through improvisation, then faces the same structural gap the next time. The three discovery paths for model degradation are: a customer complaint, a monitoring alert, and an end-of-day review. Two of the three are too slow. A customer complaint means the degradation has already been customer-visible for minutes or hours. An end-of-day review means hours of SLO breach before anyone acts. Only a monitoring alert gives the team a window to intervene before customer impact accumulates. This article is the playbook for that intervention.

A rollback playbook is not a full incident response plan. It covers the rollback decision and procedure: when to revert, who decides, what preconditions must exist, and what to do when the vendor cannot revert. For the full AI incident response plan covering detection, escalation, and post-mortem, see the adjacent article in this series.

Part 1: Pre-Conditions for Rollback (Before the Incident Happens)

In electrical engineering, a fault-protection relay is tested before the fault occurs, not during it. If the relay test is deferred to the moment the fault arrives, the fault-protection system is not a system: it is a hope. The same principle governs AI rollback capability. Rollback is a pre-planned capability, not an improvised response. The five pre-conditions below cannot be assembled under incident conditions. Confirm each one before any model update window opens.

The 12-Control AI Incident Readiness Audit covers rollback as control 9. The checklist below maps directly to that control's requirements.

Baseline established: you cannot detect degradation without a pre-update benchmark

Before any update, record the current output distribution: schema validation pass rate, P95 latency, and task-success rate on a fixed eval set. Without a pre-update benchmark, the on-call engineer has no objective basis for deciding whether observed degradation is outside normal variance. Action: run your eval set 24 hours before any scheduled update window and store the results with a timestamp and the current model version identifier.

Vendor rollback capability confirmed: the five questions to ask before any update window

The five questions in Part 4 of this article determine whether rollback is available as a self-service action, what the maximum time from request to restore is, and how long the prior version remains available. Confirm the answers in writing before the update lands. A vendor who cannot answer these questions in writing before an update is a vendor who cannot be relied on for rollback after one. This maps to 10-Point AI Vendor Audit criterion 7: model-update cadence and rollback.

Fallback path operational: if rollback is unavailable, the fallback must be tested and ready

The fallback path is a pre-built degraded-mode route: typically a prior-generation model, a cached-output layer, or a rules-based fallback for the highest-criticality outputs. "Degraded mode" is an engineering design state, not a failure state. It keeps customer-facing systems running while the vendor resolves the model issue. Test the fallback path against your SLOs before the update window, not after the page arrives. For the full treatment of fallback path design, see the article on AI fallback path production system design.

Rollback decision authority assigned: who has the authority to trigger rollback at 2 AM without a committee

Incident response degrades under time pressure. Assign a single named role (not a named individual: on-call rotates) with explicit authority to trigger rollback or fallback activation without requiring committee approval. Document the authority assignment in the runbook alongside the trigger criteria. The 35 minutes spent confirming what the team already suspects is almost always 35 minutes spent waiting for authorization that was never pre-assigned.

Runbook location documented and accessible: not in a Slack thread from three months ago

The rollback runbook must be in the on-call engineer's primary incident-response tool, not in an email thread or a Slack message that requires a search to find. The runbook location must be in the paging alert itself, or in the first document the on-call engineer opens when paged. If the runbook requires more than two clicks to reach from the alert, it will not be used under incident conditions.

The pre-conditions above form a gate. All five must pass before any model update proceeds. The checklist below is formatted for screenshot or copy-paste into your pre-update procedure.

Part 2: The Decision Tree for the 2 AM Moment

The decision tree below is designed to be executable at 2 AM with degraded cognition. Three binary gates determine the output path. Work through them in order. Do not skip gates. The time limits are engineering starting points, not contractual SLAs: they reflect the window within which orderly action is possible before the incident escalates beyond the on-call team's authority.

For AI production systems, AI system stability requires that the recovery procedure be as well-specified as the nominal operating procedure. The decision tree below is that specification for the rollback case.

Gate 1: Is the degradation confirmed by instrumentation, or only by a customer complaint?

If the only signal is a customer complaint and no monitoring alert has fired, do not treat the degradation as confirmed. Investigate: compare current output metrics against the pre-update baseline. If instrumentation confirms the degradation (schema validation failure rate elevated above baseline, latency at P95 above SLO threshold, or eval task-success rate below the pre-update benchmark), proceed to Gate 2. If instrumentation does not confirm it, move to Hold and Monitor: escalate the monitoring interval and set a re-evaluation checkpoint within 15 minutes. The free Stability Auditor assesses whether your current pipeline has the monitoring layer this gate requires.

Gate 2: Is a vendor rollback available as a self-service action, or does it require vendor support?

If vendor rollback is available as a self-service action and the vendor confirmed the capability in writing before this update window (pre-condition 2), proceed to Roll Back. If vendor rollback requires opening a support ticket or is unavailable, move to Gate 3.

Gate 3: Is the degradation causing active customer impact, or is it a quality regression below the SLO threshold?

If the degradation is causing active customer impact (errors visible to customers, SLO threshold breached), activate the fallback path immediately. If the degradation is a quality regression below the SLO threshold (outputs are degraded but within SLO bounds), move to Hold and Monitor with a shortened re-evaluation window.

Part 3: What to Do When the Vendor Says Rollback Is Not Available

When vendor rollback is unavailable, the playbook does not end. It shifts from the Roll Back path to the Fall Back path, and from incident response to contract leverage. The three actions in this section assume the fallback path was built before the incident (pre-condition 3).

Activating the fallback path (the pre-built degraded-mode route)

Activate the fallback path as designed: route traffic to the degraded-mode system, notify customers per your SLO communication protocol, and open a vendor support escalation with a clear statement of customer impact. Do not improvise a fallback under incident conditions. If the fallback path was not built before the incident, the only option is to hold and monitor while the vendor resolves the issue on their timeline, which places SLO breach duration under the vendor's control.

For AI drift detection in production, the instrumentation that triggers the decision tree is the same instrumentation that allows you to verify when the degradation resolves, whether via vendor rollback or via a subsequent update.

Documenting the vendor failure for contract leverage

Record the following during the incident, not after: the timestamp of the vendor update, the timestamp of first degradation signal, the timestamp of vendor contact, the vendor's stated reason for rollback unavailability, and the customer impact duration. This documentation is the evidence base for the post-incident contract conversation. A vendor who cannot revert on demand has failed the commitment implied by criterion 7 of the 10-Point AI Vendor Audit: model-update cadence and rollback.

The post-incident requirement: add rollback availability to the vendor contract before the next update

The post-incident contract conversation has two objectives: a written commitment to advance notice before model updates, and a written commitment to rollback availability with a stated maximum restore time. A vendor who cannot provide either commitment in writing is a vendor whose model-update risk cannot be managed through a rollback playbook. The fallback path becomes the primary risk control, not a secondary one.

ISO/IEC 42001:2023, the AI Management System standard, requires organizations to establish corrective action processes for nonconformities, including those arising from supplier actions. NIST AI RMF 1.0, in the MANAGE function, addresses incident recovery and the need for documented procedures for deployed AI systems. Neither standard mandates specific rollback SLAs; they establish the process requirement for having a documented and tested recovery path. Cite these frameworks in the post-incident vendor conversation as the governance basis for requiring written rollback commitments.

Audit criterion 7 mapping: model-update cadence and rollback

The 10-Point AI Vendor Audit criterion 7 covers model-update cadence and rollback. After the incident, use criterion 7 as the vocabulary for the vendor conversation: what is the update cadence, how much advance notice is provided, and what is the vendor's rollback capability and SLA. A vendor who passes criterion 7 provides written answers to all three before any update window. A vendor who cannot provide these answers after an incident is a vendor who will not provide them before the next one either.

Five Questions to Ask Your Vendor About Rollback Capability

These questions are designed to expose a vendor who cannot revert on demand. A vague positive answer ("rollback is supported") to any of them is a red flag. A specific, written answer to all five is the minimum bar for vendor rollback capability. Send these before any model update window, not after the incident.

OWASP LLM Top 10 (2025) identifies LLM09 (Overreliance) as the risk that materializes when a team has no rollback path and cannot revert a degraded model update. The five questions above are the operational controls that close LLM09 at the vendor-dependency layer, not just the prompt or output layer.

How the 12-Control AI Incident Readiness Audit Covers Rollback

Control 9 (rollback) is one of 12 controls in the 12-Control AI Incident Readiness Audit. It is necessary but not sufficient in isolation. A rollback playbook that executes cleanly requires at least three adjacent controls to be in place: kill-switch (control 1), eval coverage (control 8), and failure-mode visibility (control 12).

Consider the dependency chain. The decision tree at Gate 1 asks whether degradation is confirmed by instrumentation. If failure-mode visibility (control 12) is absent, Gate 1 cannot be answered from instrumentation: the team reverts to customer complaints as the primary signal, which is two of the three discovery paths that are too slow. If eval coverage (control 8) is absent, the team cannot verify that a rollback succeeded: the revert may restore the prior model version while the degradation persists in a component the revert did not reach. If kill-switch (control 1) is absent, there is no fast-path to halt all AI-assisted actions while the rollback proceeds, which means the degraded model continues processing production traffic during the rollback window.

A reader who has built a rollback runbook covering control 9 has addressed one of twelve controls. The 12-Control AI Incident Readiness Audit covers the full control stack, including tool boundary docs (control 2), audit-trail completeness (control 3), sandbox separation (control 4), prompt-injection defenses (control 6), and production data isolation (control 10). Each control has a dependency on the others. Rollback without kill-switch means the degraded model runs during the revert. Kill-switch without eval coverage means the team cannot confirm the revert resolved the degradation. The audit covers all twelve, with the dependency structure made explicit.

Conclusion

Rollback is not a recovery option you discover in the moment. It is a pre-condition you confirm before the update lands. The five pre-conditions in Part 1 are cheap to build before an incident and very expensive to assemble during one. The decision tree in Part 2 is executable in under 15 minutes when the pre-conditions are in place and degrades to improvisation when they are not. The vendor questions in Part 4 are the written commitments that determine whether rollback is in the playbook at all, or whether the fallback path is the only viable option.

The engineering background here is 7 years electrical engineering experience and BSEE from the University of South Florida, applied to AI systems that require the same pre-planned fault-protection discipline as any critical production system. Fault relays are not optional in power systems. Rollback pre-conditions are not optional in AI production systems.