Tool
Which quantization format?
GGUF, AWQ, GPTQ, EXL, FP8, FP4, NF4 — what each one is, the size-versus-quality trade-off, and which engines actually read it. The layer that sits between "which model" and "which engine."
If you skip the cards
The short version.
- Running locally, one user GGUF Q4_K_M. It runs on anything and the quality is excellent for the size. Step to Q5/Q6 if VRAM allows.
- Serving many users on a GPU AWQ or FP8 on vLLM/SGLang. FP8 if you're on Ada/Hopper/Blackwell and want near-lossless; AWQ for 4-bit at scale.
- Max quality on one consumer card EXL2/EXL3 — dial the exact bits-per-weight to fill your 24 GB and no more.
- Fine-tuning a big model on little VRAM NF4, via QLoRA. It's a training base, not a serving format — merge and re-quantize afterwards.
The trilogy
Format picked — now size it and serve it.
Quantization sets bits-per-weight; the VRAM calculator turns that into GPUs, and the inference explorer picks the engine that reads your format.