Glossary

Quantization

Last updated: 2026-07-20

Quantization compresses a neural network by storing its weights at lower numeric precision — 8, 4, even 2 bits instead of the 16 used in training. The weights become slightly coarser approximations of themselves, and in exchange the model shrinks dramatically: a 70B-parameter model needs ~140 GB of VRAM at 16-bit, ~35 GB at 4-bit. That’s the difference between a GPU cluster and a single consumer card — quantization is the single biggest reason open-weight models run on ordinary hardware at all.

Why it works

Trained weights carry more precision than the model actually needs; much of those 16 bits is noise. Quantization exploits that slack — mapping each weight to a compact grid of values, with scale factors keeping the important ones sharp. Smart schemes protect the outliers that matter most, which is why modern 4-bit quants typically lose only a few percent on benchmarks. Below 4 bits the slack runs out and degradation gets visible fast: subtler reasoning slips first, long generations drift. A useful heuristic from the llama.cpp community: a bigger model at 4-bit usually beats a smaller one at full precision in the same memory.

The names you’ll see

Model files on Hugging Face advertise their format: GGUF (llama.cpp’s format, the standard for local CPU/GPU inference — tags like Q4_K_M mean ~4 bits per weight), AWQ and GPTQ (GPU-serving formats), FP8/INT8 (the precision datacenter APIs commonly serve at). When you download a model in LM Studio or Ollama, picking a quant level is picking a quality-memory trade — Q4_K_M is the conventional sweet spot, Q8_0 near-lossless, anything below Q3 a compromise you should test.

What it doesn’t do

Quantization compresses inference; it isn’t smaller training (that’s QLoRA’s job — fine-tuning on top of a quantized base). And a quantized model is the same model, just slightly blurrier — same knowledge, same failure modes, marginally less reliable on the hardest tasks. Hosted APIs quantize behind the scenes too; you’re rarely served full 16-bit weights, you just never see the label.

Sources

← All glossary terms