Token Budget Watchdog: Embedded-Systems Patterns for LLM Operations

Embedded systems have one rule: the answer must arrive before the deadline. A car's anti-lock braking system has 5 milliseconds to decide. A pacemaker has microseconds. An aircraft flight controller has tens of milliseconds. These deadlines are not preferences. They are physics. Miss them and people die.

To make this work, embedded engineers spent fifty years building a discipline: real-time systems engineering. A watchdog timer resets the system if a deadline is missed. An interrupt handler stops lower-priority work so urgent work can run. Priority inversion protocols stop low-priority tasks from blocking high-priority ones. Deadline-driven scheduling picks the algorithm based on time left, not on what gives the best answer.

None of this exists in typical LLM code. A typical LLM call is a plain HTTP request with a 30-second timeout. There is no fallback, no speed monitoring, no priority class. When the call is slow or fails, the app finds out too late and shows the user an error. Or the call just hangs. The user stares at a spinner forever.

What the Watchdog Demonstrates

The free Token Budget Watchdog is a live demo of embedded-systems thinking applied to LLM ops. You give it three things:

• Prompt — what you want the AI to produce
• Deadline (ms) — the hard time limit for the whole operation
• Max tokens — the top limit on how long the output can be

The system runs your prompt through a multi-model fallback chain (Nemotron 120B → Gemma 31B → Gemma 26B → MiniMax → Liquid → router). For each attempt:

1. Compute remaining budget (deadline minus elapsed time)
2. If less than 500ms remain, stop with a watchdog timeout
3. Otherwise, try the call with a timeout that matches the remaining budget
4. If the call works, return the result with telemetry
5. If the call fails (rate limit, error, timeout), log the failure and try the next model

The output shows which model answered, latency, tokens used, how much of the deadline was spent, watchdog status, and every fallback event that happened along the way.

The Watchdog Timer Pattern

In embedded systems, the watchdog is a hardware timer. The app must kick it on a regular schedule or it fires and resets the system to a known-good state. This forces the app to stay responsive.

The same pattern works for LLM calls. The "system reset" here means: drop the slow model and try the next one in the fallback chain. The deadline is the LLM version of the watchdog timeout. The fallback chain is the recovery path. The user gets an answer (maybe from a smaller model) within the deadline, instead of a 30-second spinner and then an error.

Why Free-Tier Rate Limits Matter for the Demo

OpenRouter free-tier models have strict rate limits. At any given moment, one or two models in the chain may be 429-blocked. This is exactly what clients face in production. Providers have outages, rate limits, latency spikes, and routing problems. The Watchdog shows these failures instead of hiding them. When you run it, you often see the chain skip past 1 or 2 rate-limited models before finding one that responds. That is what production LLM ops looks like.

A normal tool would catch the 429 and fail silently. The Watchdog catches the 429 and tries the next model. The user gets an answer. The telemetry shows exactly what happened.

The Deeper Pattern: Hooks Are Interrupt Handlers

Embedded systems are built around interrupts. An interrupt is an outside event that stops the main program so urgent code can run. A button press triggers an interrupt handler. A sensor reading triggers an interrupt handler. The main loop can be interrupted at any time. The interrupt service routines (ISRs) are short and predictable.

Modern AI agent systems can use the same pattern. Hooks (like the ones I write for the Claude Code harness) are interrupt handlers. They fire on events: tool call, model response, completion. The main agent loop is the embedded main loop. The hooks cut in to enforce constraints (budget caps, safety checks, format validation). Without hooks, the main loop has no observability. With hooks, every event produces a measurable response.

From a wiki synthesis I built mapping embedded systems to AI: "Hooks ARE interrupt handlers. Stuck detection = watchdog timer. Budget caps = real-time deadlines. Priority inversion = when a Haiku agent blocks an Opus agent. Context window = constrained memory. MCP tools = DMA (direct memory access bypassing main context)."

Use Cases

1. Test fallback behavior under different deadlines — set a tight deadline (2000ms) and see which models in your chain can actually meet it
2. Show production AI ops to stakeholders — let non-engineers see what a real LLM call looks like with telemetry, not just "send and pray"
3. Calibrate your own production fallback chain — the architecture shown is repeatable. Use the same pattern in your codebase

From Web LLM to True Embedded

The Watchdog shows real-time patterns for cloud LLM calls. The true embedded version, TinyML on microcontrollers (ESP32, STM32, RP2040), is service #26 in the catalog. With TinyML, the AI runs on the chip. Deadlines are in microseconds. There is no network. Same patterns. Different layer of the stack. Battery-aware, deterministic, no cloud roundtrip.

If your LLM app needs real-time guarantees (voice agents, real-time assistants, latency-sensitive UIs), see the paid service. The embedded-systems discipline scales from microcontrollers to GPU clusters. The patterns are the same. The deployment is custom.