Context Windows and Prompt Caching: The Two Numbers That Decide Your LLM Bill
August 9, 2026 · AI API Index
Of all the numbers on a model's spec sheet, two end up dominating real-world bills: the context window, and the cached-input price. Teams that engineer around these two routinely run the same workload for a third of what a naive implementation costs. Here's how the billing actually works and what to do about it.
Context gets re-billed on every request
A 1M-token context window, now standard on flagships like Claude Opus 5 and GPT-5.6 Sol, is a capacity limit. It says nothing about cost. Everything sitting in context is billed as input again on every request. In a twenty-turn conversation the whole history gets resent twenty times, which is why input cost grows quadratically with conversation length. Drop a 500K-token codebase into context on a $5-per-million model and you're paying $2.50 per request before the model has written a single word. A thousand requests, $2,500.
Caching makes repeated tokens close to free
Providers cache the prefix of your prompt. If the first N tokens of a request match a recent one, those tokens get the cached rate, and the discount is steep: Opus 5 charges $0.50 cached against $5.00 standard, GPT-5.6 Terra $0.20 against $2.00, and DeepSeek V4 Pro an absurd $0.0036 against $0.435, which is over 99% off. When I compare models for prefix-heavy workloads I look at the cached-input column on the table before the headline price.
Because the match is prefix-based, the order of your prompt is a billing decision:
- Stable content first: system prompt, tool definitions, policies, reference documents.
- Then session-stable content, like conversation history, whose prefix repeats each turn.
- Volatile content last: the current user message, request-specific data.
A single volatile token early in the prompt invalidates the cache for everything after it. The classic self-inflicted wound is a timestamp in the system prompt. I've seen that one mistake erase the entire caching discount for a whole deployment.
What it looks like in practice
Take an agent carrying a 30K-token prefix (system prompt, tools, docs), adding 2K fresh tokens per request, on a $5-per-million model with 10% cached pricing:
| No caching | 90% of prefix cached | |
|---|---|---|
| Prefix cost / request | $0.150 | $0.0285 |
| Fresh input / request | $0.010 | $0.010 |
| Per 100K requests | $16,000 | $3,850 |
Same model. Same prompts. 76% smaller invoice. At DeepSeek-style 99% caching the prefix nearly vanishes from the bill.
Stuff the context, or retrieve?
With million-token windows it's tempting to skip retrieval and shove the whole corpus in. Sometimes that's right. If the corpus is stable and gets discussed across many requests, caching turns a 200K-token corpus into roughly $0.10 per request instead of $1.00, and you keep perfect recall with zero retrieval infrastructure. Retrieval starts winning when the corpus is huge, changes frequently (every change is a cache miss), or when each query only touches a small slice of it. Plenty of production systems end up hybrid, caching a stable core and retrieving the long tail. One piece of fine print worth reading: some providers charge a higher per-token rate above a request-size threshold, which tilts very long prompts back toward retrieval.
Rules I'd actually follow
- Order prompts stable to volatile. Audit for early-position timestamps and UUIDs; one is enough to break everything downstream.
- Summarize or truncate conversation history somewhere around twenty turns. Quadratic growth doesn't care that each individual turn looked cheap.
- For prefix-heavy workloads, compare models by cached price. The ranking changes.
- Model your blend in the calculator with the cache-hit-rate field before committing to anything.