AI Vendor Audit: What to Do After the Vendor Fails One of Your Criteria

A Criterion Failure Is Not the Same as a Deal Breaker

You ran the 10-Point AI Vendor Audit and the vendor cannot satisfy one of the criteria. What you do in the next 48 hours determines whether that gap becomes a negotiation lever or a production liability.

Not all criterion failures carry the same weight. A missing SLO definition (criterion 2) is a different class of problem from missing source-code ownership documentation (criterion 3). The first can be cured in 30 days with a written commitment. The second, if not resolved before signing, eliminates your ability to exit or audit the system without the vendor's cooperation. Your job has shifted from evaluation to structured response: classify the gap, document it, require a written commitment, and apply a go/no-go gate before you move forward.

The four-step protocol below maps to the real 10 criteria from the AI Vendor Audit. It gives you a specific action for each class of failure and the decision matrix you need to apply the gate without guessing.

Step 1: Document the Gap in Writing Before Anything Else

Before you send a follow-up email, before you schedule a call with the vendor, and before you discuss the gap with your own team, write it down. Specifically: the criterion by name and number, the evidence of failure (what the vendor said or failed to produce), the vendor's response if any, and the date you discovered it.

Verbal acknowledgment from the vendor has no contractual standing. A vendor that says "we are working on that" in a sales call is not committing to anything. Once the conversation continues without a written record, you have no basis to reference the gap in negotiation and no audit trail for legal if the issue surfaces post-launch. Criterion 3 from the 10-Point AI Vendor Audit, source-code ownership and audit trail, exists precisely because audit-trail discipline separates engineering accountability from sales narrative.

Use the checklist below as the template for your written gap record. Complete it before any further vendor interaction.

Step 2: Classify the Failure Severity Against the 10 Criteria

The 10 criteria from the 10-Point AI Vendor Audit divide into two classes: hard stops and remediable gaps. A hard stop is a criterion where a gap creates an irreversible position or eliminates your ability to respond to an incident. A remediable gap is one where a written commitment and a 30 to 90 day deadline can close the gap without restructuring the contract.

Classifying the failure correctly is the highest-value step in the protocol. NIST AI RMF 1.0 (GOVERN function, airc.nist.gov/RMF/1) describes supplier risk monitoring as an ongoing governance requirement, not a one-time sign-off. The severity classification below maps directly to that governance requirement: hard stops require contractual cure before signing; remediable gaps require a time-boxed written commitment.

Hard-Stop Criteria (Require Contractual Cure or Exit)

These four criteria are hard stops because a gap in any of them either locks you into the vendor's infrastructure without exit rights or eliminates your ability to respond when something goes wrong in production. OWASP LLM Top 10 (2025, LLM06: Excessive Agency and LLM07: System Prompt Leakage) name control gaps of this type as the highest-severity production risks.

• Criterion 3: Source-Code Ownership and Audit Trail. If you do not own the source code and cannot produce an audit trail independently of the vendor, you have no exit path. See the source-code ownership clause guide for the contract language this requires.
• Criterion 7: Model-Update Cadence and Rollback. A vendor without a documented rollback procedure means you cannot revert a model update that degrades your pipeline. There is no engineering basis to proceed without this.
• Criterion 8: On-Call and Incident Response Structure. If the vendor cannot show you a documented incident response structure with named owners and SLAs, the 12-Control AI Incident Readiness Audit maps exactly what is missing. A vendor without this capability is not production-ready.
• Criterion 10: Documented Handover and No Lock-In. A vendor that cannot provide a documented handover procedure is keeping you in the relationship by eliminating the exit path, not by delivering value. The Build vs Buy Framework covers switching cost and lock-in tolerance as a formal decision criterion.

Remediable Criteria (30 to 90 Day Window)

These six criteria can be addressed with a written remediation commitment and a specific deadline, provided the vendor agrees in writing and the deadline carries a consequence clause. ISO/IEC 42001:2023 supplier relationship sections (iso.org/standard/81230.html) provide the governance framework for treating these as corrective action items with documented closure requirements.

• Criterion 1: Monitoring on Every Critical Path. Remediable with a 30-day deadline. Require the vendor to provide monitoring dashboard access or documented alerting coverage before go-live.
• Criterion 2: Error Budgets and SLOs. Remediable with a 30-day deadline. Require written SLO definitions with measurement methodology and reporting cadence.
• Criterion 4: Drift Detection. Remediable with a 60-day deadline. Require documented detection method, alert threshold, and owner.
• Criterion 5: Fallback Paths. Remediable with a 60-day deadline. Require a documented fallback procedure with a tested activation sequence.
• Criterion 6: Cost-Anomaly Alarms. Remediable with a 30-day deadline. Require alert configuration and threshold documentation.
• Criterion 9: Data-Handling and Privacy Boundaries. Remediable with a 90-day deadline, depending on regulatory context. EU AI Act deployer obligations (Regulation 2024/1689) and functional safety procurement standards both require documented data boundaries for high-risk AI systems. A gap here is remediable only if the vendor can produce the documentation within the deadline.

Step 3: Set a Written Remediation Deadline in the Contract

Once you have documented the gap and classified it as remediable, the next step is a written remediation clause in the contract. "Written" here means a contract amendment, an addendum, or at minimum a formal written notice that the vendor acknowledges in writing. An email thread is weak; a signed addendum is enforceable.

A remediation clause must contain four elements to be usable: the specific criterion by name and number, a measurable outcome that defines what "remediated" looks like, a deadline date, and a consequence clause that specifies what happens if the deadline is missed (typically the right to exit the contract without penalty or a fee reduction). Without all four elements, the clause is unenforceable in practice because there is nothing measurable to verify at the re-audit date.

What happens when a vendor refuses to put the remediation commitment in writing? That refusal is itself a classification signal. A vendor that can satisfy the criterion has no reason to refuse a written commitment. Refusal signals either that the capability does not currently exist or that the vendor is unwilling to be held to a measurable standard. Both outcomes put the gap in the hard-stop class, regardless of the original classification. Cross-reference the source-code ownership and exit clause guide for the contract language patterns that apply when the vendor is unwilling to commit in writing.

For high-risk AI systems, EU AI Act deployer obligations (Regulation 2024/1689) require documented monitoring of vendor performance. A remediation clause with a measurable outcome and a deadline is the minimum structure for satisfying that requirement. ISO/IEC 42001:2023 (iso.org/standard/81230.html) supplier relationship sections require documented nonconformity correction, which maps directly to this step.

Step 4: Apply the Go / No-Go Gate

The go/no-go gate is a three-row decision matrix. You apply it once you have the written gap record (step 1), the severity classification (step 2), and the vendor's written response to the remediation request (step 3). The matrix removes the judgment call from the decision and replaces it with a documented process.

When you document the decision for legal and finance, the memo needs three things: the criterion that failed, the severity classification and how you reached it, and the action taken and why. This documentation protects you if the vendor later disputes the exit or if the gap surfaces in a due-diligence review. The pre-signing questions checklist describes the questions that should have surfaced these gaps earlier in the evaluation; if those questions were not asked, the gap record is also evidence for improving the next vendor evaluation process.

Re-Running the Audit After Remediation

The remediation deadline is not the finish line. It is the re-audit trigger. When the deadline arrives, you run the 10-Point AI Vendor Audit again against the specific criteria that failed, using documented evidence the vendor provides rather than verbal updates. The re-audit is the same instrument applied to the same criteria; it is not a new evaluation and it does not replace the original audit record.

The recommended re-audit cadence for remediable gaps is: 30 days after the remediation clause is signed (interim check: is the vendor making documented progress?), at the deadline date (full re-audit of the failed criterion), and at go-live (full re-audit of all 10 criteria, not just the ones that failed). Passing a re-audit on one criterion at one point in time does not extend to all criteria or to future states of the system. A vendor that passes re-audit at go-live but then pushes a model update without a documented procedure has reopened criterion 7.

Engineering discipline here mirrors the same audit-trail requirement from criterion 3: every re-audit produces a dated record with the evidence reviewed, the outcome (pass or fail per criterion), and the next review date. A vendor that resists re-audit after a written remediation commitment is signaling the same thing as a vendor that refused to commit in writing: the capability does not exist.

When to Bring in an Independent Engineer

Three situations warrant independent verification beyond what an internal team can provide.

First: your internal team does not have production AI engineering experience. Verifying that a rollback procedure is operational (not just documented) requires someone who can read the procedure and assess whether it maps to a real system state, not just a PDF. The same applies to audit-trail completeness (criterion 3) and incident response structure (criterion 8). Marketing language and engineering documentation look similar to a non-technical reader.

Second: the vendor relationship is high-stakes. A multi-year, multi-system contract creates significant switching cost if the relationship fails. At that scale, the cost of an independent production audit is a rounding error compared to the cost of discovering a hard-stop gap 18 months into the contract. The Build vs Buy Framework treats vendor lock-in tolerance as a formal criterion precisely because the switching cost compounds over the contract term.

Third: the gap is in a technical criterion you cannot independently verify. If the vendor claims to have remediated criterion 7 (rollback) by updating a runbook, verifying that claim requires reading the runbook against the system architecture and confirming that the rollback procedure would actually work in a production incident. Without production AI engineering experience, you cannot distinguish a real rollback from a description of one. The production AI audit service covers exactly this verification: you bring the vendor's documentation, and the engineer applies the same production-engineering standard used in building sincllm-mcp v2.0.0 (12 tools) and the 99% pipeline reliability benchmark on sr-demo-ai.com (sincllm's own production benchmark, not a vendor SLA or client guarantee).

Conclusion

A criterion failure found before signing is leverage. You have a documented gap, a severity classification, and a contractual instrument to require a cure or exit cleanly. The same failure discovered six months after go-live is a liability: no audit trail, no remediation clause, no exit right, and a vendor whose incentives are now to minimize the problem rather than fix it.

The four-step protocol in this article, document, classify, set a written deadline, and apply the gate, converts an ambiguous vendor gap into a documented engineering decision. It is not adversarial oversight; it is the same audit-trail discipline that criterion 3 from the 10-Point AI Vendor Audit requires of the vendor. Applying it to your own process is the engineering standard for production AI procurement.