The counterintuitive truth
When a language model generates text, your GPU's compute power barely matters. What matters is how fast memory can feed it.
Here's why: to produce each new token, the model must read essentially all of its weights once. A 7B model at Q4 is about 4.8 GB — so generating 60 tokens per second means streaming 4.8 GB × 60 ≈ 288 GB through memory every second. The multiply-add math is easy for any modern GPU; the reading is the bottleneck.
The napkin formula
generation tok/s ≈ (memory bandwidth GB/s ÷ model size GB) × ~0.65
The 0.65 is real-world efficiency — overhead, KV cache reads, and imperfect scheduling eat about a third of theoretical bandwidth.
Tested in BreadLab: an RTX A4000 has 448 GB/s of bandwidth. For a 7B model at Q4 (~4.8 GB with overhead): 448 ÷ 4.8 × 0.65 ≈ 61 tok/s predicted. Our measured result: 63 tok/s. The formula also predicted the scaling — halving model size roughly doubles speed (we measured 203 / 124 / 63 tok/s for 1.5B / 3B / 7B).
What this means when buying hardware
- For generation speed, buy bandwidth, not FLOPS. A used RTX 3090 (936 GB/s) beats many newer cards for LLM work.
- A faster CPU won't fix a slow model. On CPU-only setups, dual-channel DDR4 (
51 GB/s) caps a 7B Q4 model near 7 tok/s no matter how many cores you have. DDR5 (90 GB/s) roughly doubles that. - Quantization is a speed upgrade, not just a fit trick — Q4 runs ~2× faster than Q8 because it streams half the bytes.
Where the formula doesn't apply
Prompt processing (reading your input before generating) is compute-bound, not bandwidth-bound — it batches many tokens per pass and runs 10–100× faster than generation. That's why a long prompt starts responding after a short pause, then generates at a steady rate. Mixture-of-experts models also break the formula in your favor: they only read the "active" fraction of their weights per token.
Run the numbers for your own hardware in the Can I Run It? calculator — it uses exactly this formula, validated against real benchmarks published in Lab Notes.