Context Window is the maximum number of tokens a model can attend to within a single request, bounding the combined size of the system prompt, user input, retrieved passages, tool outputs, and the model's own in-progress response. It is a hard architectural limit set by the attention mechanism's memory and compute footprint, not a soft suggestion. Early production models in 2023 shipped with roughly 4K-token windows, which forced engineers to compress context aggressively or offload retrieval to external systems. By 2025-2026, frontier models routinely advertise 1M-token or larger windows, and several open-weights releases have pushed into the multi-million range. The headline number refers to input capacity; output is usually capped separately and lower. A larger window does not linearly expand useful recall: attention degrades over long contexts, costs scale with the square of sequence length under standard attention, and latency rises accordingly. The window is best understood as a budget to be allocated deliberately rather than a container to be filled.
How it works
Inside a Transformer, every token attends to every other token within the window via the attention mechanism, producing an N×N cost in time and memory under naive attention. Architectural tricks such as sparse attention, sliding-window layers, and ring/pipeline parallelism reduce the practical cost for long sequences, but the model still must hold key-value state for every token it can attend to. The window is fixed at serving time by the model's training and the inference stack's configuration; prompts exceeding it are truncated, rejected, or split into multiple requests. Output tokens draw from the same budget, so a 200K window with a 150K-token prompt leaves little room for a long answer.
Why it matters for AI engineers
Token count drives billing, latency, and memory on roughly linear-to-quadratic curves, so stuffing a 1M window when 20K would do is a direct cost and p95-latency hit. Long contexts also suffer context rot: models lose fidelity on early tokens, miss mid-sequence details, and become more susceptible to prompt injection buried in retrieved documents. Engineers should size context to the task, prefer RAG or reranking over brute-force inclusion, use prompt caching to amortize repeated prefixes, and treat the window as one constraint among several rather than a substitute for retrieval design.
Context Window vs. alternatives
| Concept | Scope | Persistence | Typical use |
|---|---|---|---|
| Context Window | Single request | Per-request, cleared after | In-context reasoning |
| KV Cache | Single session | Short-lived, reusable | Latency reduction across turns |
| RAG | Across requests | External store | Grounding on large corpora |
| Memory (agent) | Across sessions | Durable | Long-term agent state |
Related terms
Definitions are the start. Ask the Research Desk for a cited, multi-source brief on Context Window — real sources, verified claims, delivered in minutes.
Ask the Research Desk →