10 Questions to Ask Before You Sign an AI Vendor Contract

Why the Contract Matters More Than the Demo

Every AI vendor demo is optimized for the best-case scenario. The model responds correctly. Latency is low. The integration looks clean. What the demo cannot show you is what happens at 3 AM when the primary model fails, who is on-call and what their escalation procedure is, whether you have the right to roll back to a prior model version, or what the exit clause looks like if the relationship stops working.

The contract governs all of that. And once you sign, your leverage to negotiate those terms disappears. A team that skips SLO negotiation has no contractual basis to demand remediation when reliability drops. A team that does not ask about data handling has no written commitment to enforce when a breach disclosure arrives. The window to negotiate is before signature; after signature, you are working with whatever boilerplate the vendor's legal team drafted in their favor.

The 10 questions below map directly to the 10 criteria in sincllm's 10-Point AI Vendor Audit, the same framework used to evaluate production AI systems. They are organized so you can send them in advance of a vendor meeting and require written answers. Every question has a corresponding "what good looks like" and a "red flag answer" so you can score across vendors on the same rubric rather than relying on recency or relationship bias.

Frameworks like the NIST AI Risk Management Framework's GOVERN function, ISO/IEC 42001:2023's supplier relationship requirements, and the EU AI Act's obligations for high-risk AI providers and deployers all establish vendor due diligence as a buyer responsibility, not a vendor courtesy. Asking these questions is not adversarial; it is the standard.

The 10 Questions

Each question maps to one of the 10 audit criteria from the 10-Point AI Vendor Audit. For each, a specific, measurable, contractually bounded answer is the pass condition. A vague commitment is a red flag.

1. Who monitors every critical path, and what is the SLO?

Why it matters: Monitoring on every critical path (Audit Criterion 1) is the baseline for any production AI system. If the vendor cannot name the specific endpoints monitored, the alert thresholds, and the service-level objective they commit to contractually, there is no basis for remediation when reliability falls short.

Good answer: "We monitor inference latency, error rate, and availability on every endpoint in your integration. Our committed SLO is 99.5% availability measured monthly, with a documented remediation credit schedule. Here is the monitoring dashboard you will have access to."

Red flag answer: "We have best-in-class infrastructure and strong uptime historically." (No number, no monitoring scope, no contractual commitment.)

2. What are the error budgets, and what triggers an escalation?

Why it matters: Error budgets and SLOs (Audit Criterion 2) define the acceptable failure envelope. Without them, every incident becomes a negotiation rather than a procedure. You need to know what threshold triggers a human escalation, what the escalation path is, and what happens if the budget is exhausted before the end of the measurement window.

Good answer: "Our error budget for your tier is 0.5% per month. When the budget is 50% consumed in a given period, you receive an automated alert. When it is exhausted, the on-call engineering lead is paged and you receive an incident declaration within 30 minutes."

Red flag answer: "We will reach out if there are issues." (No threshold, no budget, no defined escalation trigger.)

3. Do you own the source code, model weights, and audit trail?

Why it matters: Source-code ownership and audit trail (Audit Criterion 3) determine your exit options and your compliance posture. If the vendor owns the weights and you have no access to the audit trail, you cannot demonstrate regulatory compliance independently, and you cannot migrate without starting from zero. The OWASP LLM Top 10 (2025) identifies Excessive Agency (LLM06) and Overreliance (LLM09) as risks that are bounded by ownership and audit controls, not by vendor trust.

Good answer: "You own the fine-tuned weights for any model trained on your data. You have read access to the audit trail at all times. On contract termination, we provide a full export of weights, prompt templates, and audit logs within 30 days."

Red flag answer: "The model is hosted on our platform; we handle that infrastructure." (No ownership statement, no audit-trail access, no export commitment.)

4. How does the vendor detect and alert on model drift?

Why it matters: Drift detection (Audit Criterion 4) is how you know when the model that worked during evaluation is no longer the model running in production. Without drift detection, quality degradation goes unnoticed until a user reports it or a downstream system fails silently.

Good answer: "We run automated output-quality checks against a held-out evaluation set weekly. If the quality metric drops more than a defined threshold, you receive an alert within 24 hours with the specific failing cases."

Red flag answer: "Our models are continuously updated for quality." (No detection mechanism, no alert, no threshold defined.)

5. What fallback path activates when the primary model fails?

Why it matters: Fallback paths (Audit Criterion 5) are the difference between a recoverable failure and a service outage. A production AI system without a defined fallback is a single point of failure. This question surfaces whether the vendor has designed for the failure case or has assumed the primary path will hold.

Good answer: "If the primary inference endpoint is unavailable, traffic routes automatically to a secondary endpoint within 30 seconds. If both are unavailable, the integration returns a graceful degradation response defined in the integration spec. We can show you the runbook."

Red flag answer: "We have high availability. Downtime is rare." (No fallback path, no secondary endpoint, no degradation behavior defined.)

6. How does the vendor detect cost anomalies before they become a surprise invoice?

Why it matters: Cost-anomaly alarms (Audit Criterion 6) protect you from unbounded spend when a model behaves unexpectedly, when token usage spikes due to a prompt change, or when a usage pattern changes in production. The absence of alarms means the first signal you get is an invoice.

Good answer: "We send an alert when your usage exceeds 80% of your contracted volume in a billing period. You can set a hard cap at a specified threshold. We provide daily spend summaries and a webhook for spend events."

Red flag answer: "You can monitor usage in your dashboard." (No alert, no hard cap, no proactive notification.)

7. What is the model-update cadence, and can you roll back to a prior version?

Why it matters: Model-update cadence and rollback (Audit Criterion 7) determine whether the behavior you validated in testing is the behavior running in production next month. A silent model update that changes output distribution can break downstream systems, trigger hallucinations in contexts that were previously stable, or shift the model's compliance posture without notice.

Good answer: "Major model updates are announced 30 days in advance. You can pin your integration to a named model version. If a new version degrades your outputs, you can roll back to the prior pinned version within 48 hours by submitting a support request."

Red flag answer: "We update our models regularly to improve performance." (No cadence, no notice, no rollback right.)

8. What does on-call and incident response look like for your integration specifically?

Why it matters: On-call and incident response (Audit Criterion 8) define what happens when your integration fails in production and you need a human with context, not a ticket queue. Generic SLAs about "support response time" are not the same as a named escalation path for a production AI incident in your specific integration. See sincllm's AI Incident Readiness Audit for how to evaluate whether a vendor has real incident controls.

Good answer: "For production incidents affecting your integration, we have a named on-call engineer assigned during onboarding. Incident declarations are issued within 30 minutes of detection. Post-mortems are delivered within 5 business days with root cause and remediation timeline."

Red flag answer: "Our support team is available 24/7." (No escalation path, no on-call assignment, no post-mortem commitment.)

9. How is your data handled, where is it stored, and who can access it?

Why it matters: Data handling and privacy boundaries (Audit Criterion 9) determine your regulatory exposure. If the vendor stores your data in a jurisdiction that conflicts with your compliance requirements, trains on your data without explicit consent, or grants access to third-party subprocessors without disclosure, those are contractual and regulatory risks that must be resolved before signing. Use sincllm's free adversarial validator tool to test prompt-injection defenses before you sign, as prompt injection is one vector through which data boundaries are violated at runtime.

Good answer: "Your data is stored in [named region] only. We do not train on customer data without explicit written consent. All subprocessors are disclosed in our DPA. You can request a data deletion report at any time."

Red flag answer: "We take security very seriously." (No region, no training policy, no subprocessor disclosure.)

10. What does a clean handover look like, and what does the exit clause say?

Why it matters: Documented handover and no lock-in (Audit Criterion 10) determine whether you can leave when you need to. Vendor exit is engineering-feasible when the contract allows it: see proof that a vendor exit is engineering-feasible when the contract allows it. A contract that does not define the exit procedure, the data export timeline, or the handover documentation set creates de facto lock-in regardless of the vendor's intentions. This question is uncomfortable for vendors precisely because it matters most.

Good answer: "On termination, we provide: (1) full export of your model weights and training data within 30 days, (2) all prompt templates and integration documentation, (3) a runbook for migrating to an alternative provider or self-hosted stack. The exit clause in Schedule B specifies the 30-day data retention window and the export format."

Red flag answer: "We hope you will not need to leave, but we can discuss that when the time comes." (No export commitment, no timeline, no documentation set defined.)

Red Flag Answers to Watch For

The table below gives you a quick-scan reference for the vendor meeting. A red-flag answer does not disqualify a vendor automatically; it is a signal to push for specifics. If a vendor cannot replace a red-flag answer with a concrete, written commitment, that is a disqualifying finding.

How to Use This Checklist in a Vendor Meeting

The questions work best as a written pre-meeting submission, not as a verbal list read aloud in a room where the vendor can give polished improvised answers.

Step 1: Send the questions 48 hours in advance. Email the vendor your 10 questions explicitly labeled as "required written responses for contract evaluation." This signals that verbal answers are not sufficient and gives the vendor time to involve the technical team rather than the sales team.

Step 2: Require written answers. Verbal answers in a meeting are not enforceable and often do not reflect what the contract actually says. Written answers create a record you can cross-reference against the contract draft. If a vendor's written answer contradicts what their contract says, that is a finding you need to resolve before signing.

Step 3: Score across vendors on the same rubric. Use the "What Good Looks Like" column in the table above as your scoring standard. Assign a pass or fail to each answer for each vendor. Recency bias (the last vendor you spoke to seems best) is a known procurement failure mode; a consistent rubric prevents it.

Step 4: Push on every red-flag answer. A red-flag answer is not disqualifying by itself. Ask the vendor to replace it with a specific, measurable commitment. If they cannot, note that as an unresolved finding. An unresolved finding on Criterion 3 (ownership) or Criterion 10 (exit clause) is a contract risk that requires escalation to legal before signing.

Printable pre-signing checklist: Copy the list below, distribute to your evaluation team, and fill in the vendor response field for each question.

What Comes After the Contract

Getting satisfactory answers to all 10 questions before signing is the leverage moment. The contract terms you negotiate now govern every future incident, every cost dispute, and every exit scenario. That is why this checklist exists and why getting written commitments on these criteria is non-negotiable for any production AI deployment.

But the contract is not the end of the evaluation. It is the beginning. Once you sign, the question shifts from "what did the vendor commit to?" to "is the vendor meeting those commitments in production?" That is a different question, and it requires a different tool.

The 10-Point AI Vendor Audit is the runtime monitoring version of this pre-signing checklist. It uses the same 10 criteria, organized as a repeatable production-engineering audit you can run on a quarterly basis or after any significant change: a model update, a new integration, a pricing renegotiation, or an incident post-mortem. The pre-signing checklist establishes the standard. The audit verifies the vendor is meeting it.

The audit is a free 16-page PDF, gated by email. It takes 15 minutes per vendor. If you have just negotiated a contract using these 10 questions, the audit is the next step to ensure those contractual commitments are actually being honored in production.

Conclusion

The contract is the only moment in the vendor relationship where you hold all the leverage. Before signature, you can negotiate SLOs, rollback rights, data handling commitments, exit clauses, and incident response procedures. After signature, you are working with whatever the vendor's standard boilerplate says, which is written to protect the vendor, not the buyer.

These 10 questions are not comprehensive legal guidance; they are engineering-grounded due diligence criteria, reviewed against the same criteria used in production AI systems and benchmarked against the NIST AI RMF GOVERN function, ISO/IEC 42001:2023 supplier requirements, and the EU AI Act's obligations for high-risk AI deployers. Run them before every AI vendor engagement. Require written answers. Score across vendors on the same rubric. And use the 10-Point AI Vendor Audit to verify those commitments are being met after you sign.