The short version
A language model is billions of numbers (weights). Quantization stores those numbers with fewer bits each, making the model file smaller and faster to run — at a small cost in quality.
The original training format is usually FP16 (16 bits per weight). A 7-billion-parameter model at FP16 is about 14 GB. Quantized to 4 bits, the same model is roughly 4–5 GB and produces nearly the same output.
Reading the names
Formats you'll see on model downloads, from biggest to smallest:
| Format | Bits per weight (approx.) | Quality |
|---|---|---|
| FP16 / BF16 | 16 | The reference — what the lab trained |
| Q8_0 | ~8.5 | Effectively indistinguishable from FP16 |
| Q6_K | ~6.6 | Excellent, hard to notice any loss |
| Q5_K_M | ~5.7 | Very good |
| Q4_K_M | ~4.85 | The community default — small, still strong |
| Q3_K_M | ~3.9 | Noticeable quality tradeoffs |
| Q2_K | ~3.35 | Desperate measures — often incoherent |
The _K and _M suffixes refer to newer mixed-precision schemes that keep the most important weights at higher precision. In practice: Q4_K_M is the default for a reason — it roughly quarters the memory needed versus FP16 while keeping quality most people can't tell apart in normal use.
Why it matters for your hardware
Memory is the wall in local AI. Quantization is the lever that decides whether a model fits:
- 7B model: 14 GB at FP16 → ~4.7 GB at Q4 — fits a 8–12 GB GPU
- 32B model: 64 GB at FP16 → ~20 GB at Q4 — fits a 24 GB GPU
- 70B model: 140 GB at FP16 → ~42 GB at Q4 — needs unified memory or multiple GPUs
Quantization also makes models faster: generation speed is limited by how quickly weights stream through memory, so a model half the size generates roughly twice as fast on the same hardware.
Rule of thumb
Start at Q4_K_M. If you have memory to spare, step up to Q5 or Q6 for a small quality bump. Only go below Q4 when it's the difference between running a model and not running it — a bigger model at Q4 usually beats a smaller model at Q8 anyway.
To see what fits your machine, try our Can I Run It? calculator — it does this math for any model, quant, and GPU combination.