The context window
A model's context window is how much text it can "see" at once — your conversation history, pasted documents, system instructions, and its own reply, all measured in tokens (a token ≈ ¾ of an English word). Modern local models advertise 32K or 128K-token windows.
What the spec sheets undersell: using that window costs memory beyond the model itself.
The KV cache
As a model reads text, it stores intermediate results (keys and values — hence "KV") for every token, so it doesn't recompute the whole conversation for each new word. This KV cache lives in the same VRAM or unified memory as the weights, and it grows linearly with context length.
Rough scale for modern 8B-class models: about 1 GB per 8K tokens of context at full precision. That means:
| Context in use | KV cache (8B model, approx.) |
|---|---|
| 4K tokens | ~0.5 GB |
| 32K tokens | ~4 GB |
| 128K tokens | ~16 GB — more than the model itself at Q4 |
Bigger models scale proportionally: a 70B model's KV cache at 32K context is tens of gigabytes.
Symptoms of running out
If a model loads fine but crashes or slows dramatically mid-conversation, the KV cache is the usual suspect — it grew until memory ran out. Runtimes like Ollama reserve cache space for the configured context length at load time, which is why raising the context setting can make a previously-working model refuse to load at all.
Practical guidance
- Don't configure a bigger window than you use. Setting 128K "just in case" reserves gigabytes you may need for a better model instead.
- 8K–16K covers most chat and coding sessions comfortably.
- Long-document work is when to raise it — and when to re-check that everything still fits, using the context selector in our Can I Run It? calculator.
- Some runtimes support KV cache quantization (storing the cache at 8-bit or 4-bit), which halves or quarters this cost at minor quality risk — worth enabling for long contexts on tight memory.