Token Budget Watchdog: Embedded-Systems Patterns for LLM Operations

Embedded systems have a non-negotiable property: the response must arrive before the deadline. A car's anti-lock braking system has 5 milliseconds to decide. A pacemaker has microseconds. An aircraft flight controller has tens of milliseconds. These deadlines are not preferences. They are physics. Miss them and people die.

To make this work, embedded engineers spent fifty years developing a discipline: real-time systems engineering. Watchdog timers that reset the system if a deadline is missed. Interrupt handlers that pre-empt lower-priority work. Priority inversion protocols that prevent low-priority tasks from blocking high-priority ones. Deadline-driven scheduling that selects the algorithm to run based on time remaining, not on what produces the best answer.

None of this exists in typical LLM application code. A typical LLM call is a synchronous HTTP POST with a 30-second timeout, no fallback, no rate-of-change monitoring, no priority class. When the call slow-runs or fails, the application discovers it after the fact and surfaces an error to the user. Or worse, the call hangs and the user just sees a spinner forever.

What the Watchdog Demonstrates

The free Token Budget Watchdog is a live demonstration of embedded-systems thinking applied to LLM ops. You provide:

• Prompt — what you want the AI to produce
• Deadline (ms) — the hard time budget for the entire operation
• Max tokens — the upper bound on output length

The system then runs your prompt through a multi-model fallback chain (Nemotron 120B → Gemma 31B → Gemma 26B → MiniMax → Liquid → router). For each attempt:

1. Compute remaining budget (deadline minus elapsed time)
2. If less than 500ms remaining, abort with watchdog timeout
3. Otherwise, attempt the call with a timeout matched to remaining budget
4. If the call succeeds, return the result with telemetry
5. If the call fails (rate limit, error, timeout), log the failure and try the next model

The output telemetry shows: which model resolved it, latency, tokens used, percentage of deadline consumed, watchdog status, and the full list of fallback events that happened along the way.

The Watchdog Timer Pattern

In embedded systems, the watchdog is a hardware timer that the application MUST kick periodically to prevent system reset. If the application hangs, the watchdog fires and resets the system to a known-good state. The application is forced to remain responsive.

The same pattern applies to LLM calls. The "system reset" in this context is "abandon the slow model and try the next one in the fallback chain." The deadline is the hardware analog of the watchdog's timeout. The fallback chain is the recovery path. The user gets an answer (possibly from a smaller model) within the deadline, instead of a 30-second spinner followed by an error.

Why Free-Tier Rate Limits Matter for the Demo

OpenRouter free-tier models have aggressive rate limits. At any given moment, one or two of the chain might be 429-blocked. This is exactly the production reality clients hit when running real LLM workloads — providers have outages, rate limits, latency spikes, and routing problems. The Watchdog tool exposes these failures rather than hiding them. When you run it, you often see the fallback chain skip past 1-2 rate-limited models before landing on one that responds. That is what production LLM ops looks like.

A "normal" tool would catch the 429 and silently fail. The Watchdog catches the 429 and tries the next model. The user gets an answer. The telemetry shows what happened.

The Deeper Pattern: Hooks Are Interrupt Handlers

Embedded systems build everything around interrupts: external events that pre-empt the main program flow. A button press triggers an interrupt handler. A sensor reading triggers an interrupt handler. The main loop is interruptible and the interrupt service routines (ISRs) are short and predictable.

Modern AI agent systems can use the same pattern. Hooks (like the ones I write for the Claude Code harness) are interrupt handlers. They fire on events: tool call, model response, completion. The main agent loop is the equivalent of the embedded main loop. The hooks pre-empt to enforce constraints — budget caps, safety checks, format validation. Without hooks, the main loop has no observability. With hooks, every event produces a measurable response.

From a wiki synthesis I built mapping embedded systems to AI: "Hooks ARE interrupt handlers. Stuck detection = watchdog timer. Budget caps = real-time deadlines. Priority inversion = when a Haiku agent blocks an Opus agent. Context window = constrained memory. MCP tools = DMA (direct memory access bypassing main context)."

Use Cases

1. Test fallback behavior under different deadlines — set a tight deadline (2000ms) and watch which models in your chain can actually meet it
2. Demonstrate production AI ops to stakeholders — show non-engineers what a real LLM call looks like with telemetry, not just "send and pray"
3. Calibrate your own production fallback chain — the architecture shown is reproducible. Use the same pattern in your codebase

From Web LLM to True Embedded

The Watchdog demonstrates real-time patterns for cloud-LLM calls. The actual embedded version of this — TinyML on microcontrollers (ESP32, STM32, RP2040), where the AI runs on the chip, the deadlines are in microseconds, and there is no network — is service #26 in the catalog. Same patterns. Different layer of the stack. Battery-aware, deterministic, no cloud roundtrip.

For LLM applications that need real-time guarantees — voice agents, real-time assistants, latency-sensitive UIs — see the paid service. The embedded-systems discipline scales from microcontrollers to GPU clusters. The patterns are the same. The deployment is custom.