LLM Context Window Management: Select, Order, and Evict

Give each context class an explicit policy

Select for decision value, then make roles visible

Start from the next model decision. Include material that can change that decision or prove its result: governing instructions, current goal, relevant evidence, required schemas, eligible tools, and necessary state. Anthropic's context-engineering guidance describes context as a finite resource and recommends finding the smallest collection of high-signal tokens likely to produce the desired outcome.

Use just-in-time retrieval when the model can locate material safely and the added tool latency is acceptable. Lightweight references such as document identifiers, file paths, or query handles can preserve discoverability without loading whole objects. For stable document stores, RAG chunking strategies can improve the unit of selection, but retrieval still needs permission and freshness checks.

Order content by role rather than dumping sources together. Clearly distinguish instructions, user data, quoted evidence, tool results, and examples. Place stable prefixes consistently when provider caching is useful, while keeping variable or user-specific content in its intended boundary. The goal is legibility to both model and operator, not a universal ordering trick.

Compress and evict with evidence-preserving tests

Compaction replaces a long history with a shorter representation. Preserve decisions, commitments, unresolved issues, authoritative references, identifiers, errors, and the current plan. Mark the summary as derived material and retain a path to the source turns. Compression is lossy unless proven otherwise for the task.

Test recall before optimizing brevity. Build long traces with delayed dependencies, changed requirements, contradictory revisions, tool errors, and facts that become important later. Compare the uncompacted baseline with candidate summaries on the same downstream decisions. Anthropic's guidance similarly recommends beginning with recall and improving precision after critical details survive.

Evict material that is reproducible, superseded, or low-value: verbose successful tool payloads, duplicate explanations, obsolete drafts, and stale retrieval. Keep denial reasons, failed attempts that affect retry policy, and the latest authoritative state. Use semantic caching for response reuse only under its separate equivalence and freshness contract.

Context management failure cases

• Fill to capacity: every available document enters the request because it fits. Evaluate relevance and distraction, not only size.
• Summary becomes authority: a derived compaction silently replaces the original source. Preserve provenance and reload paths.
• Latest turn wins: a recent instruction overrides an older binding constraint. Separate authority from recency.
• Stale tool result: a prior read remains in context after external state changes. Attach freshness and revalidation rules.
• Permission after retrieval: unauthorized text reaches the model and is filtered only later. Enforce visibility before context assembly.
• Compression benchmark is easy: tests check recent facts but not delayed dependencies or conflicting revisions.

A practical implementation sequence

1. Inventory context classes. Record owner, authority, visibility, freshness, priority, size, and reload method for each class.
2. Define the next-decision contract. Name the information required for the model's next output or action and the invariants it must preserve.
3. Build a selection baseline. Start with a small manually reviewed context and measure task, evidence, latency, and cost outcomes.
4. Add retrieval and ordering rules. Make source roles visible, enforce access before inclusion, and capture why each item was selected.
5. Introduce compaction cautiously. Preserve provenance and compare summaries with full-history behavior on delayed-dependency tests.
6. Monitor context behavior. Track input length, selected sources, compaction events, reloads, failures, and latency through LLM latency optimization.

Decision summary

Manage context by utility under constraints: select what can change the decision, identify what governs versus informs, compress with fidelity evidence, and evict only when information can be reproduced or no longer matters. Larger windows expand capacity; they do not remove the need for curation.