How to Record and Review LLM Token Usage

To investigate LLM costs, set an observation period and record prompts, responses, model settings, token counts and billed charges. Check the record for missing calls and distinguish measured totals from estimates before drawing conclusions.

Review Responses per Prompt

For a week-long token review, keep daily counts of prompts, responses and completed tasks. Compare like-for-like workloads and retain the records needed to explain changes in the response-to-prompt ratio.

Record how many responses each task requires and classify corrections, clarifications and accepted outputs. Their token usage and cost need to be measured for the chosen workload.

Compare workload costs using recorded token usage, model settings, responses, and retries for both versions.

What I Was Actually Measuring

I added a logging hook to Claude Code. It recorded every prompt going in and every response coming out. It counted tokens on both sides. I tracked five things:

Exchange rate — responses per prompt
Input token volume — how much context each request carried
Output token volume — how much the model generated
SNR (signal-to-noise ratio) — quality of prompt vs. noise
Haiku overhead — cost of the scatter hook per call

Treat the proposed SNR score as a prompt-inspection heuristic. It does not establish the share of useful information or predict output quality without a validated measurement method.

I used SNR and exchange-rate records when reviewing the scatter hook.

The 80K Token Incident

Inspect long conversations for repeated context, clarification and unsuccessful retries. Retain per-call token records before attributing their growth to a prompt or hook.

Compare total task costs with a defined baseline and include context carried forward on each call. A hypothetical shorter conversation is not an observed saving.

Use the recorded failure patterns to decide which prompt or workflow change to test.

The Haiku Math

Record scatter-call count, model, token usage and billed cost for the same observation window. No weekly scatter-cost total or per-call result is established here.

Include scatter-call overhead and main-model token usage when comparing the same workload.

The hook organizes a prompt into six fields: PERSONA, CONTEXT, DATA, CONSTRAINTS, FORMAT and TASK. Inspect the generated context and measure subsequent clarifications rather than assume a reduction.

What Would Have Happened Without Measurement

Without a comparable baseline, the cost of an unchanged workflow is unknown. Compare candidate changes on the same task set and retain the records needed to explain the result.

Use responses per prompt alongside output quality, token usage and complete operating cost. Structured instructions still require output checks and may require follow-up questions.

If you run any production LLM workflow and you have not measured your exchange rate, do it this week. The number may surprise you.

// Keep reading

• What an AI Fallback Path Actually Looks Like in a Production System
• AI Monitoring Checklist: What Every Production System Needs on Every Critical Path
• SLO Design for AI Systems: How to Set Error Budgets That Survive a Vendor Update

// Production AI Engineering

Build AI systems that hold up in production.

sinc-LLM designs, audits, and stabilises production AI infrastructure: from vendor evaluation and cost accountability to incident controls and MCP architecture.

See what we do →