Models· 2 min
What is quantization?
Quantization shrinks model weights from 16-bit to 8-, 4- or even 2-bit numbers to run faster on smaller hardware.
Key idea
Lower precision = less memory and compute per weight, with small quality loss when done carefully (GPTQ, AWQ, GGUF).
Example
A 70B Llama model in fp16 needs ~140GB VRAM; the same model in 4-bit fits on a single 48GB GPU.
What to remember
- Enables local / on-device LLMs (Ollama, LM Studio)
- GGUF is the go-to format for llama.cpp-style inference
- Combine with LoRA for cheap fine-tuning (QLoRA)