Time to First Token vs Throughput: Measure LLM Speed
Time to first token measures how long an interactive user waits before meaningful generation begins. Throughput measures how much work the serving system completes over time. Track both, plus generation cadence and end-to-end latency, because improving aggregate capacity can leave an individual request slower.
Define each metric before comparing systems
| Metric | Operational definition | What it represents | Common ambiguity |
|---|---|---|---|
| Queue delay | Admission timestamp to start of model processing | Contention, scheduling, and capacity pressure | Often folded into TTFT without being reported separately |
| Time to first token | Chosen request-start boundary to first non-empty generated token | Initial interactive wait, including boundaries the measurement retains | Client, gateway, and server clocks may begin at different points |
| Time per output token | Generation duration after the first token divided by subsequent generated tokens | Cadence of a streamed response | Some tools report its reciprocal or include the first token |
| End-to-end latency | User-action boundary to usable terminal result | The complete product wait | May omit retrieval, validation, tools, or rendering |
| Per-request generation rate | Generated tokens divided by that request's measured generation time | Speed experienced within one response | Not the capacity of the whole server |
| System throughput | Completed requests or processed tokens across all requests per unit time | Serving capacity under a named load | Input and output work may be combined without labels |
| Goodput | Work completed while meeting declared quality and latency constraints | Capacity that is useful to the product | A benchmark may count fast but invalid or late results |
NVIDIA's inference benchmarking guide defines TTFT as prompt processing plus generation of the first token and notes that practical measurements can include queueing and network time. Use its definitions as a starting point, then document the exact clock boundaries in your client and server.
A number without workload context is not portable
Prompt length changes prefill work; generated length changes decode work; concurrency and arrival bursts change queueing. Model architecture, precision, accelerator, memory, serving engine, scheduler, network, and warm state also matter. A single request against an idle server answers a different question from sustained traffic at a declared arrival rate.
Report distributions for TTFT, output cadence, and end-to-end latency. High percentiles show whether a minority of users wait behind long requests or resource pressure. Segment by input-length and output-length bands rather than averaging a short classification with a long report. The context-window management guide explains why context should be curated for evidence quality as well as speed.
For throughput, name the numerator: requests, input tokens, output tokens, or total processed tokens. Name the denominator and measurement window, including warm-up and drain behavior. Also disclose error, timeout, cancellation, and validity rates; dropping difficult work is not a capacity improvement.
Throughput and latency compete through scheduling
Batching can keep accelerators busy by processing multiple requests together. Dynamic serving introduces a policy question: when does a new request join, how are prefill and decoding interleaved, and how are long generations prevented from monopolizing capacity? More concurrent work may improve aggregate throughput while increasing queue delay or generation gaps for some requests.
The vLLM paper identifies key-value cache memory management as a limit on batch size and presents PagedAttention to reduce fragmentation and duplication in the evaluated system. That result motivates measuring memory and scheduler behavior, but it does not establish that every deployment inherits the paper's reported performance.
Read continuous batching for LLM inference before choosing scheduler controls. Compare operating points at equal quality, prompt mix, output mix, and service constraints. The best point is the one that meets user latency and reliability while delivering enough goodput—not the largest raw throughput value.
Measurement failure cases
- TTFT from the wrong boundary: server timing looks fast while gateway and client buffering dominate. Capture both server and user-observed clocks.
- First event is empty: a stream-open event is treated as a generated token. Require the first non-empty model content.
- Mean-only reporting: a smooth average hides long queues for a request slice. Publish percentiles and distributions.
- Throughput without quality: malformed, timed-out, or truncated responses remain in the numerator. Report goodput and failure categories.
- Mismatched workloads: systems receive different prompt lengths, output caps, or concurrency. Replay the same request manifest.
- Client and server clocks mixed: clock skew creates impossible stage durations. Use duration measurement within one clock domain or calibrated tracing.
A practical measurement sequence
- Write metric formulas. Define timestamps, non-empty first content, inclusion rules, units, percentiles, and terminal states.
- Build a request manifest. Preserve task classes, prompt lengths, output limits, streaming mode, quality assertions, and cancellation behavior.
- Choose an arrival model. Test idle, steady, and burst conditions that reflect the product rather than only maximum saturation.
- Capture stage timing. Measure queue, prefill, first content, decode, validation, and product completion under correlated traces.
- Sweep load gradually. Observe where TTFT, cadence, errors, and goodput change as concurrency rises.
- Verify product constraints. Compare quality and reliability before accepting a capacity gain, using the seven latency levers to address the dominant stage.
Decision summary
Choose TTFT when the question is “when does useful output begin?”, per-request cadence when the question is “how smooth is generation?”, throughput when the question is “how much load can the system serve?”, and end-to-end latency when the question is “when is the task done?” Keep all four tied to one workload and one quality contract.
Primary and official sources
- NVIDIA: LLM Inference Benchmarking Concepts — official definitions for TTFT, inter-token latency, and related benchmarking boundaries.
- Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” — primary research on KV-cache management and serving throughput.
- OpenAI API: Latency Optimization — official guidance on application and model latency levers.