Prompt Caching

Prompt Caching is a feature offered by frontier-model providers that stores a shared prompt prefix in a server-side cache so that subsequent requests reusing that prefix are billed at a fraction of the normal input-token price and return with lower latency. Anthropic launched prompt caching for the Claude family in August 2024, OpenAI followed with automatic caching in October 2024, and Google added context caching for Gemini around mid-2024. Cached input tokens typically cost around 10% of their base price on Anthropic and 50% on OpenAI, while the initial cache-write pass costs slightly more than a standard input. Caches are short-lived—on Anthropic the default time-to-live is five minutes, refreshed each time the prefix is hit—and require a minimum prefix length of 1,024 tokens on most models. Developers mark cacheable spans with explicit breakpoints, and the provider matches the incoming request's prefix against stored entries to decide whether a hit occurs.

How it works

When a request arrives, the provider hashes the prompt prefix up to each configured cache breakpoint and checks for a matching entry in its key-value cache. On a hit, the cached key-value tensors are reused directly, skipping the prefill computation that normally dominates time-to-first-token. On a miss, the provider runs prefill as usual and writes the resulting tensors back to the cache for future reuse, charging a small write premium. Anthropic exposes up to four explicit breakpoints per request, letting developers cache a system prompt, a tool schema, and a few-shot block independently; OpenAI's variant is automatic and prefix-based with no manual breakpoints. The cache only persists for a short window—five minutes on Anthropic—so high-frequency or long-running workloads benefit most.

Why it matters for AI engineers

For agent workloads that resend the same system prompt, tool definitions, and conversation history on every turn, prompt caching can cut input costs by an order of magnitude and shave seconds off each response. The savings compound across long agentic loops where the prefix grows with every tool call, making previously uneconomical multi-step runs viable. Engineers should structure prompts so the stable, cacheable content sits at the front and volatile content (user messages, tool results) lands after the last breakpoint, since any edit to the cached prefix invalidates the whole entry. The five-minute TTL also means bursty or queued workloads need a keep-alive strategy—a single request every few minutes—to avoid cache eviction. Treat the cache as a best-effort resource: design for graceful degradation when hits don't land, and never assume cached state is private to one user.

Prompt Caching vs. alternatives

Related terms