LLM API Retry and Backoff: Reliable Client Design

Build an explicit retry classification

Status codes alone are insufficient when providers differ. Keep the classification adapter beside the provider client, version it, and include response headers or documented error type where available. Never parse a human-readable message as the only control signal.

Use backoff, jitter, deadlines, and budgets together

Exponential backoff increases the delay after repeated failure; jitter randomizes it so many clients do not retry in lockstep. The OpenAI Cookbook recommends random exponential backoff for rate-limit errors and warns that unsuccessful requests still contribute to per-minute limits. A loop without a cap can therefore worsen the condition it is trying to recover from.

Apply four boundaries: maximum attempts, maximum elapsed time, the caller's end-to-end deadline, and a shared retry budget for the dependency. Stop when any boundary is exhausted. A request with no time left for a useful answer should not begin another slow attempt merely because its local attempt count is low.

Use provider timing guidance when present, but still add bounded randomization if the contract allows it. Coordinate concurrency and admission so retries do not jump ahead of new work indefinitely. During a broad outage, circuit breaking or load shedding can fail fast while occasional probes test recovery.

Every retry consumes user latency. Include attempts and sleep time in the end-to-end latency trace, and compare recovery benefit with deadline misses and fallback quality.

Protect end-to-end effects, not only API calls

Generating another answer is often safe from a storage perspective, but the surrounding workflow may send a message, charge an account, create a ticket, or invoke a tool. Attach an application operation identifier to consequential work, persist its terminal result, and make repeated delivery return the existing outcome rather than repeat the effect.

Ambiguous failures need special treatment. A connection can drop after the provider or tool accepted work but before the client observed success. If the downstream interface cannot deduplicate, automatic retry may be unsafe. Read the authoritative state first or escalate the ambiguity.

Streaming also changes semantics. If generation stops after partial content reached the user, a fresh attempt is a new generation, not a continuation unless the provider offers explicit resume semantics. Do not append it blindly. Mark the first stream incomplete, restart visibly when appropriate, and run the full output validation contract on the accepted result.

Retry failure cases

• Retry every error: invalid requests and denied access consume quota without becoming valid. Classify first.
• Deterministic delays: a fleet retries at the same intervals and recreates the spike. Add jitter.
• Nested retry multiplication: SDK, service, job, and agent each retry independently. Assign one owner and a shared budget.
• No end-to-end deadline: local attempts continue after the user or upstream job has given up. Propagate cancellation and remaining time.
• Duplicate tool effects: an ambiguous result causes the same action twice. Require idempotency or state readback.
• Fallback hides regression: a backup model returns lower-quality results while transport success looks healthy. Evaluate fallback quality through model-routing gates.

A practical implementation sequence

1. Write the provider error map. Translate documented statuses, error types, and timing guidance into retryable, terminal, or ambiguous classes.
2. Propagate a deadline. Give each attempt the remaining user or job time and stop before useful completion becomes impossible.
3. Apply randomized backoff. Bound minimum and maximum delay, attempts, elapsed time, and shared dependency retries.
4. Add end-to-end idempotency. Protect tool calls and durable effects with an operation identifier, terminal record, and state readback.
5. Test fault sequences. Inject rate limits, connection loss before and after acceptance, partial streams, server failures, invalid input, denial, cancellation, and outage.
6. Instrument the policy. Capture classification, attempt, delay, provider guidance, deadline, fallback, final status, and duplicate prevention in LLM observability.

Decision summary

A reliable retry client answers four questions before another call: is the failure transient, is repetition safe, is there time and budget left, and will another attempt help under current load? If any answer is no or unknown for a consequential effect, stop or inspect state instead of looping.