AI Monitoring Alerts That Actually Page: The Signal-to-Noise Problem in LLM Observability

The Two Alert Failure Modes (and Why Both Are Deadly)

Most engineering teams running LLM systems in production have instrumented some form of monitoring. The problem is not whether monitoring exists. The problem is that alert quality breaks in one of two ways, and teams rarely recognize which failure mode they are in until an incident surfaces via a user complaint instead of an on-call page.

Alert fatigue. The on-call queue fills with pages: every latency spike, every token overage, every vendor 429. Ninety-five percent are noise. The engineer who gets paged at 2 AM six nights in a row for events that resolved themselves starts treating every page as noise. The real incident, when it arrives, looks like the rest. It gets dismissed. The team learns about it from users the next morning.

Silent failure. The team has been burned by noisy alerts before, so they set conservative thresholds. Nothing pages. Latency is climbing, hallucination rate is increasing, cost per resolved task is doubling. The monitors are green. The vendor dashboard is green. Users start reporting wrong answers and slow responses, and the engineering team has no record of when degradation started or how long it had been building.

LLM observability is harder than traditional service monitoring for a specific reason: output non-determinism. A 500 error is unambiguous. A wrong answer is not detectable by a metric the way an error code is. That asymmetry means you cannot instrument correctness the way you instrument availability, and it forces a sharper discipline around which metrics are actually informative. If your current monitoring setup is paging on noise or missing real degradation, a production AI engineer can audit your monitoring layer and map it against the controls that fix both failure modes.

Why the Signal-to-Noise Frame Applies to LLM Alerting

Signal-to-noise ratio is a core concept in electrical engineering: the ratio of the power of a meaningful signal to the power of background noise. A communication system works when the SNR is high enough that the receiver can detect the signal reliably. When SNR falls below the detection threshold, the signal is lost in the noise floor.

The framing is not a metaphor borrowed from a blog post. Seven years of electrical engineering work in Luanda and a BSEE from the University of South Florida put SNR and matched-filter receiver design at the center of how I think about detection problems. Applied to LLM observability, the parallel is precise: the alert system is the receiver, production metrics are the channel, and real degradation is the signal. The noise floor is everything else: per-request latency jitter, token-count fluctuation from variable input lengths, transient vendor throttling resolved by retry logic. These are non-informative components. They carry no information about real system health. Alerting on them raises the apparent noise level and buries the signal.

Real degradation, on the other hand, has a characteristic shape: it is sustained, it appears across multiple correlated metrics, and it trends rather than spikes. SLO breach is sustained latency exceeding an agreed threshold across a rolling window, not a single slow request. Error budget burn rate is a trend signal: the budget is being consumed faster than the agreed rate. Cost-per-task increase is a sustained economics shift, not a single expensive call.

The design question for your alert system is the same one an EE asks when designing a receiver: what is the minimum detectable signal above the noise floor, and what is the shape of the signal you are trying to detect? Every subsequent section answers one part of that question.

The Three Signals That Always Deserve a Page

1. SLO Breach: Latency Exceeds the Agreed Threshold for the Critical Path

The common failure in SLO alerting is choosing the wrong metric or the wrong aggregation. Alerting on mean latency produces a metric that is buffered by fast requests; a sustained slowdown in the critical path can raise mean latency by 40% while still looking acceptable. Alerting on a single slow request produces noise: every system has one-off slow calls caused by cold starts, model provider variation, or network jitter.

The correct alert triggers on sustained SLO breach: p95 or p99 latency over a rolling window (5 minutes or 15 minutes, depending on traffic volume) compared against a pre-agreed SLO per critical path. "Per critical path" is load-bearing. A global latency SLO averaged across all requests obscures the fact that your highest-value user path is consistently breaching while low-priority background tasks are running fast.

This maps directly to the 12-Control AI Incident Readiness Audit: Control 1 (monitoring on every critical path) and the related SLO design criteria. A well-run production system has named SLOs per path, not a single global latency metric. If you do not have per-path SLOs defined, alert calibration is blocked: you cannot set a threshold before you know what the threshold is meant to protect.

See the SLO design and error budgets guide for the complete framework for naming and measuring SLOs across your LLM pipeline before calibrating these thresholds.

2. Error Budget Burn Rate: Consuming the Budget Faster Than the Agreed Rate

Raw error-count alerts are a step up from single-request alerting, but they still have a calibration problem: a spike in errors during a high-traffic period looks identical to a sustained degradation during low traffic, even if the operational consequences are opposite. Error budget burn rate solves this by telling you not whether errors are happening, but whether the error budget will be exhausted before the next SLO window closes at the current rate of consumption.

The burn-rate model, drawn from SRE literature including Google's SRE practices and DORA research, operates on two thresholds. A fast-burn alert fires when the error budget will be exhausted in less than one hour at the current rate: this is a page, requiring immediate response. A slow-burn alert fires when the budget will be exhausted in less than 24 hours: this is a ticket, requiring same-day investigation. Fast burn catches acute failures. Slow burn catches gradual degradation that raw error counts would miss until the budget was already gone.

Burn rate alerting fires earlier and with higher fidelity than raw error counts because it incorporates the rate of change. A single request failure in a 30-day window is negligible burn; 10 failures in 10 minutes on a system with a 99.5% SLO is a fast-burn alert. The alert distinction is meaningful; the raw error count (10) is the same in both cases.

3. Cost-Per-Task Spike: The Economics of the Pipeline Are Breaking

Cost-per-API-call is noise. It varies with input length, model tier, and batch size. A single long input that runs 20% over the average call cost is not an incident; it is expected variance in a system that processes variable-length inputs. Alerting on every call that exceeds an average cost threshold produces high-frequency noise with no operational meaning.

Cost-per-resolved-task is signal. It measures the economic efficiency of the end-to-end pipeline: how much does it cost to produce one completed, correct output? This metric is stable under normal conditions because input variance is averaged over a workload cohort. When cost-per-resolved-task rises sustainably, something structural is breaking: a model behavior change making the model more verbose without improving resolution, a retry storm indicating upstream errors, or a caching failure forcing repeated computation.

Consider a concrete scenario: a model update makes the provider's model more verbose. Per-call token cost rises. But the model is also more accurate, so the resolution rate improves and fewer retries occur. Cost-per-resolved-task stays flat or improves. This is not an incident. Now consider the same scenario where verbosity increases but resolution rate does not improve. Cost-per-resolved-task rises with flat resolution. That is an incident: the pipeline economics are breaking without a compensating quality gain. This distinction maps to the AI Cost Reality Check's Criterion 1: cost per resolved task, which is the spend-audit lens that separates meaningful cost signals from per-call noise.

The Three Noise Sources to Filter, Not Page On

4. Per-Request Latency Jitter

Every networked LLM call has a noise floor in its latency distribution. Model provider response time varies with server load, geographic routing, and prompt complexity. A single request that takes 3x the mean latency is not an incident; it is a sample from the tail of a distribution that always had a tail. Paging on individual slow requests pages on the noise floor itself, not on signal above it.

The correct handling is percentile tracking over a rolling window. The p95 or p99 latency for the last 5 minutes tells you whether the tail is expanding systematically, which is signal, versus whether a single request was slow, which is noise. Stop alerting on single-request latency. Start alerting on rolling-window percentile latency against your per-path SLO.

5. Minor Token Count Overage

Token counts vary with input structure. A prompt that processes a longer document, a more complex query, or a multi-turn conversation history will use more tokens than the baseline average. A single request that runs 15% over a per-request token budget is not an incident: it reflects the natural variance of the input distribution.

Token count overage becomes a signal only as a trend metric: if average tokens per request is climbing week-over-week without a corresponding improvement in output quality or resolution rate, that is worth a ticket. It is not worth a real-time page. Roll up token count as a daily or weekly trend metric reviewed in your calibration loop, not as a real-time alert that fires on individual requests.

6. Temporary Vendor Throttling or Rate-Limit Events

Retry logic exists precisely to handle transient throttling. A 429 response from your model provider that is resolved by exponential backoff and a successful retry is a system working as designed. Paging on every 429 event creates noise that obscures the actual signal: sustained throttling that has exhausted the retry budget and is producing real user-facing failures.

The correct alert triggers on one of two conditions: rate limiting that has persisted beyond the configured retry window (indicating a structural throttle, not a transient one), or exhaustion of the retry budget resulting in actual request failures that count against the error budget. The first is a ticket. The second feeds the burn-rate alert already described above.

The Matched-Filter Approach: Alert on Patterns, Not Points

In EE receiver design, a matched filter is a detector tuned to the expected shape of the target signal rather than to any generic threshold crossing. The matched filter maximizes the signal-to-noise ratio at the detection output by correlating the incoming signal against a template of what the target signal looks like. Random noise does not correlate with the template, so it contributes little to the detector output. The target signal correlates strongly, so it rises above the noise floor at the detector output even when it is buried in noise at the input.

Applied to LLM alerting, the matched-filter equivalent is a multi-condition alert rule: fire when latency is trending up AND cost-per-task is rising AND accuracy proxy (measured by eval coverage or resolution rate) is declining, all simultaneously over the same rolling window. The key property is correlation across independent metrics. Random noise in a complex system rarely drives all three metrics in the same direction at the same time. Real degradation, which is a structural problem in the pipeline, does: a model behavior change raises latency, raises per-task cost, and reduces resolution rate together. The correlated triple-signal fires the matched-filter alert. Any single metric spiking in isolation does not.

Practical implementation requires multi-condition alert rules in your observability stack. Single-metric threshold rules are easier to configure, which is why most teams default to them, but they produce the alert-fatigue pattern described at the start of this article. The matched-filter architecture is more work to configure and requires a calibration loop to tune the window sizes and correlation thresholds, but it reduces false positives because the alert condition is structurally harder for noise to satisfy.

The control-theory framing underlying this calibration approach is covered in detail in the control-theory framing that underlies threshold calibration in production AI. The matched-filter model here is the alerting-layer application of the same stability and feedback principles described there.

You can also check your pipeline's current failure-mode visibility before tuning alert thresholds using the free stability auditor tool. If the auditor surfaces gaps in failure-mode coverage, those gaps are the starting point for the calibration work described in the next section.

What Good Alert Calibration Looks Like in Practice

The framework above translates into a concrete set of alert rules and a calibration loop. Table 1 classifies each metric type. Table 2 is the alert calibration reference you can adapt directly.

Table 1: Signal vs. Noise Classification for LLM Monitoring

Table 2: Alert Calibration Reference

The calibration loop that keeps this table accurate has three steps. First, measure the false-positive rate monthly: count pages that did not result in a confirmed incident, divided by total pages. A false-positive rate above 20% means thresholds need tightening. Second, review and adjust thresholds quarterly based on observed traffic patterns and any changes to the model or pipeline. Third, run a retrospective after every paged incident asking two questions: did the alert fire at the right time (not too early to be noise, not too late to be useful), and was the severity classification correct?

This discipline is what underlies sincllm's own production benchmark of 99% pipeline reliability across 500 or more transcripts on sr-demo-ai.com. Reliability at that level requires calibrated alerting, not just more monitors. Uncalibrated alerting produces either fatigue (more monitors, more noise, less trust) or silence (conservative thresholds, false security). The calibration loop is what makes the monitoring investment operationally useful.

NIST AI RMF 1.0's MANAGE function addresses ongoing monitoring, incident response, and alerting requirements for deployed AI systems as part of a production AI risk management posture. ISO/IEC 42001:2023's performance monitoring and corrective action sections are similarly relevant for teams implementing these alert disciplines under a formal AI management system. OWASP LLM Top 10 (2025) identifies LLM04 (Model Denial of Service) as the attack class a well-calibrated cost-per-task alert detects early, and LLM08 (Vector and Embedding Weaknesses) as a drift-detection case that the matched-filter correlated alert can surface before it becomes a full incident.

Before calibrating alert thresholds, confirm that every critical path is instrumented. The prerequisite work is covered in the companion checklist for instrumenting every critical path before calibrating alert thresholds. Calibration without complete instrumentation produces accurate thresholds on incomplete coverage: you may have excellent alert quality on the paths you are watching while other paths fail silently.

Does Your Current Alert Setup Have These Problems?

The following four questions are a quick diagnostic for your current monitoring posture. Answer each honestly. "Unsure" is a valid answer and is often the most informative one.

Two or more "Unsure" answers means your monitoring layer has not been calibrated against the signal types described in this article. That is not a criticism of the team that set it up: most LLM monitoring setups start from "instrument everything and alert on everything," which is the right starting posture for a new system. Calibration is the next step, and it requires applying the SNR discipline described here.

Conclusion

Alert quality in LLM production systems is an engineering discipline, not a configuration task. The distinction matters because configuration implies a one-time setup: pick some thresholds, enable some alert rules, and you are done. Engineering implies a loop: define the signal you are trying to detect, characterize the noise floor you are working against, tune the detector, measure false-positive rate, and recalibrate on evidence.

The signal-to-noise problem in LLM observability is solvable. The three signals that always deserve a page (SLO breach, burn-rate exhaustion, cost-per-task spike) are measurable and have well-defined detection conditions. The three noise sources that should never page in isolation (per-request jitter, single-token overage, transient throttling) have clear filtering rules. The matched-filter correlated alert reduces false positives by making the alert condition structurally harder for independent noise to satisfy. The calibration loop keeps the system honest over time as the model, the traffic, and the pipeline evolve.

The same tools electrical engineers use to detect signals in noisy channels apply directly to LLM observability. The discipline is portable. Applying it to your current alert setup is the difference between a monitoring layer that pages on real incidents and one that is either noisy or silent.