Quantization compresses a neural network by storing its weights at lower numeric precision — 8, 4, even 2 bits instead of the 16 used in training. The weights become slightly coarser approximations of themselves, and in exchange the model shrinks dramatically: a 70B-parameter model needs ~140 GB of VRAM at 16-bit, ~35 GB at 4-bit. That’s the difference between a GPU cluster and a single consumer card — quantization is the single biggest reason open-weight models run on ordinary hardware at all.
Why it works
Trained weights carry more precision than the model actually needs; much of those 16 bits is noise. Quantization exploits that slack — mapping each weight to a compact grid of values, with scale factors keeping the important ones sharp. Smart schemes protect the outliers that matter most, which is why modern 4-bit quants typically lose only a few percent on benchmarks. Below 4 bits the slack runs out and degradation gets visible fast: subtler reasoning slips first, long generations drift. A useful heuristic from the llama.cpp community: a bigger model at 4-bit usually beats a smaller one at full precision in the same memory.
The names you’ll see
Model files on Hugging Face advertise their format: GGUF (llama.cpp’s format, the standard for local CPU/GPU inference — tags like Q4_K_M mean ~4 bits per weight), AWQ and GPTQ (GPU-serving formats), FP8/INT8 (the precision datacenter APIs commonly serve at). When you download a model in LM Studio or Ollama, picking a quant level is picking a quality-memory trade — Q4_K_M is the conventional sweet spot, Q8_0 near-lossless, anything below Q3 a compromise you should test.
What it doesn’t do
Quantization compresses inference; it isn’t smaller training (that’s QLoRA’s job — fine-tuning on top of a quantized base). And a quantized model is the same model, just slightly blurrier — same knowledge, same failure modes, marginally less reliable on the hardest tasks. Hosted APIs quantize behind the scenes too; you’re rarely served full 16-bit weights, you just never see the label.