LLM Observability Metrics: What to Instrument in Production

The production LLM metric map

Begin with questions an operator must answer during a regression or incident. Emit dimensions that separate model, prompt, retrieval, tool, and product changes without placing raw user content into every metric label.

No single metric proves quality. A low error rate can coexist with incorrect answers; a strong evaluation score can coexist with timeouts; a fast response can still invoke the wrong tool. Keep the families joinable through trace identifiers and release metadata.

Design one trace around the user outcome

Create a parent span for the product request, not merely for the model API call. Add child spans for context assembly, retrieval, reranking, model calls, output validation, tool execution, fallback, and downstream state changes. The trace should end only when the user-visible workflow reaches a terminal result.

The OpenTelemetry registry for generative AI attributes provides common telemetry names for model operations. Use the conventions supported by your instrumentation version, record that version, and keep product-specific attributes in a documented namespace rather than overloading standard fields.

Version everything that can change behavior

At minimum, attach the application release, model identifier, prompt or policy version, output schema, retriever and index version, tool contract version, and evaluation version. Without those joins, a dashboard can show a regression while leaving the operator unable to identify what changed.

Separate content capture from operational telemetry

Metrics should use bounded labels, not raw prompts, responses, document text, email addresses, or user IDs. Traces can carry carefully controlled content when the use case permits it, but content capture needs explicit access, retention, redaction, and sampling rules. A safer default is to record hashes, sizes, classifications, and document identifiers while storing sensitive payloads only in an approved diagnostic path.

Join online signals with offline evaluations

Runtime telemetry reveals drift, error clusters, and new task shapes. Evaluations determine whether those changes are acceptable. Sample representative or failed traces into a reviewed evaluation corpus, remove sensitive content, label the failure category, and promote confirmed incidents into regression cases.

Observability failure cases

• The model-call dashboard: provider latency and token usage look healthy while retrieval, validation, or downstream actions fail. Trace the product path, not one dependency.
• Averages without distributions: a small class of very slow or failing requests disappears inside the mean. Inspect distributions and segment by route, model, task, and release.
• Logs without correlation: retrieval, model, and tool events cannot be joined to one user outcome. Propagate trace and run identifiers across every boundary.
• Raw content in metric labels: sensitive data leaks into a high-cardinality system that is expensive to query and hard to govern. Use bounded classifications and protected diagnostic storage.
• Unversioned prompts and indexes: quality changes but the operator cannot identify the behavioral change. Version configuration and data artifacts beside code releases.
• Every anomaly becomes a page: operators learn to ignore alerts that do not map to user impact or an action. The alert signal-to-noise guide shows how to route symptoms to actionable conditions.
• Success means HTTP 200: transport success is mistaken for task success. Join service metrics with evaluations and product outcomes.

A staged implementation sequence

1. Map the critical path. Draw the request from user action to terminal product outcome. Name every model, retrieval, tool, validation, and storage boundary that can fail.
2. Define the trace contract. Standardize identifiers, span names, timestamps, status, error classes, versions, policy decisions, and resource-usage fields before adding dashboards.
3. Instrument boundaries first. Capture start, completion, failure, timeout, cancellation, retry, and fallback at each dependency. Confirm that context propagates across asynchronous work.
4. Derive bounded metrics from traces. Build service, latency, usage, retrieval, tool, safety, quality, and product-outcome views with labels that remain stable and queryable.
5. Write the content-handling policy. Decide what prompt, response, and retrieved content may be captured, who can read it, how it is redacted, and when it is deleted.
6. Connect alerts to actions. Tie service objectives and quality gates to user impact, then document the owner, diagnostic query, fallback, and escalation for each alert. Use the production monitoring checklist to review path coverage.
7. Run an incident replay. Starting from one alert, verify that an operator can identify the affected release and stage, inspect a representative trace, determine user impact, and choose rollback, fallback, or further investigation.