Continuous Batching for LLM Inference: Tradeoffs
Continuous batching lets a serving scheduler admit and retire requests as generation progresses instead of holding a fixed group until every sequence finishes. It can improve accelerator use, but the engineering decision is a scheduling policy across queue delay, generation cadence, memory, fairness, cancellation, and deadlines.
Continuous batching is a bundle of controls
| Control | Benefit sought | Risk introduced | Measure |
|---|---|---|---|
| Admission rate | Keep compute occupied as slots become available | Queue growth, memory pressure, and unstable tail latency | Queue age, admitted sequences, rejection, and timeout |
| Concurrent sequence limit | Increase aggregate work in flight | Less memory headroom and slower cadence per request | Resident sequences, memory use, TTFT, and time per output token |
| Prefill scheduling | Begin new prompts without waiting for all decodes to finish | Large prefills interrupt streaming progress | Prefill duration, decode gaps, and prompt-length slices |
| Work budget per iteration | Balance prompt and generation work | A poor budget favors one class and delays another | Processed input and output units by iteration and class |
| Priority and aging | Meet product tiers or deadlines while preventing starvation | Low-priority traffic may wait indefinitely | Wait time and deadline success by priority and age |
| Preemption policy | Recover capacity for urgent or feasible work | Repeated computation, churn, and unpredictable latency | Preemptions, recomputation, evictions, and completion |
| Length and deadline limits | Bound resource occupancy and user wait | Truncation or rejection can reduce task quality | Finish reason, validity, task success, and user recovery |
KV-cache memory governs feasible concurrency
During autoregressive generation, each active sequence retains key-value cache state that grows with sequence length. Variable prompt and output lengths make this memory dynamic. If allocation wastes capacity through fragmentation or duplication, fewer sequences fit and the scheduler has less freedom.
The vLLM paper presents PagedAttention and a serving system designed to reduce KV-cache memory waste and support flexible sharing. Its reported throughput results belong to the evaluated models, workloads, and comparison systems. For your deployment, measure actual memory occupancy, allocation failures, preemption, and recomputation rather than copying a published multiplier.
Context policy also affects serving. Long prompts increase prefill work and cache occupancy; long generation limits reserve uncertainty about future memory. Pair scheduler tuning with context-window management and realistic length distributions. Keep enough headroom for variance, runtime overhead, and graceful cancellation.
Make fairness and overload product decisions
First-in-first-out is simple but does not guarantee good user outcomes when request lengths vary. Strict priority can protect an interactive tier while starving background work. Shortest-job policies may improve completion counts but require length estimates and can disadvantage large legitimate tasks. Aging can raise waiting requests over time. Choose a policy from declared service objectives.
Overload behavior must be explicit. Options include bounded queueing, admission rejection with retry guidance, reduced generation limits, alternate capacity, or degradation to a simpler workflow. Retrying blindly pushes more work into the congested system. Track queue depth together with age and deadlines; a stable count can still hide requests that never advance.
Use TTFT versus throughput metrics to report both user wait and aggregate capacity. Include goodput: only results that meet deadline, validity, and quality constraints should count toward useful serving performance.
Continuous batching failure cases
- Maximum batch benchmark: throughput rises while interactive TTFT and generation cadence become unacceptable. Keep service constraints in the result.
- Long-prefill interruption: large prompts repeatedly delay active decodes. Inspect prompt-length slices and scheduling gaps.
- Priority starvation: background or lower-tier requests never run. Add aging, quotas, or separate capacity and verify maximum waits.
- Memory cliff: a small concurrency increase causes preemption or allocation failure. Monitor headroom before saturation.
- Cancellation leak: disconnected users retain queue or cache resources. Propagate cancellation through admission and execution.
- Retry storm: rejected work returns immediately and worsens overload. Coordinate backoff and retry budgets with admission policy.
A practical tuning sequence
- Define service objectives. Name TTFT, generation cadence, end-to-end deadline, quality, error, and fairness constraints by workload class.
- Replay realistic arrivals. Preserve prompt lengths, output lengths, streaming, cancellations, priorities, and burst behavior.
- Establish a conservative baseline. Capture throughput, goodput, queue age, memory, preemption, and tail latency before tuning.
- Change one scheduler control. Sweep admission, concurrency, or work budget within memory headroom while holding the workload constant.
- Probe overload and fairness. Mix long and short requests, priority classes, cancellations, slow clients, and deadline expiry.
- Gate on user and system outcomes. Accept only operating points that meet quality and reliability constraints, using LLM latency optimization to investigate the remaining bottleneck.
Decision summary
Continuous batching is appropriate when shared serving must use accelerator capacity across variable arriving requests. Its value comes from an observed operating point where goodput improves without violating latency, memory, fairness, or quality constraints. The scheduler policy is part of product behavior and deserves versioned tests.
Primary and official sources
- Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” — primary research on KV-cache memory and high-throughput serving.
- NVIDIA: LLM Inference Benchmarking Concepts — official definitions for inference latency and throughput measurements.