AI: quantization
This course covers quantization of model weights: storing parameters in fewer bits so a model fits on the hardware available. It starts with the memory arithmetic for weights and the KV cache, then compares precision formats such as BF16, FP8, INT8 and INT4. It works through the scale and zero point mapping, explains how outliers break it, and describes methods like GPTQ and AWQ and formats like GGUF. It ends with task-based evaluation in your own language and a procedure for deciding whether and how to self-host.
What you'll learn
- ✓Calculating whether a model's weights and KV cache fit in memory
- ✓The differences between FP32, FP16, BF16, FP8, INT8 and INT4
- ✓Quantizing with a scale and zero point, and how outliers distort the range
- ✓How GPTQ and AWQ differ, and choosing among GGUF, MLX and GPU checkpoints
- ✓Evaluating quantized models per task and per language against thresholds set in advance
- ✓Deciding whether to self-host and serving a local model behind a provider interface
The same idea, one level down
You already did this once. The pgvector course stored embeddings as halfvec to halve the index, and as one bit per dimension for a first-pass search. Both traded a little accuracy for a lot of memory.
Read the full lesson free →Curriculum
7 lessonsThe same idea, one level down
Quantization stores model weights in fewer bits, and bits per parameter decide which hardware can run a model.
The arithmetic of what fits
Computes weight memory and KV cache overhead, and shows why a larger model at lower precision usually wins.
Precision formats
Compares floating-point and integer formats and shows where quality drops first, including languages other than English.
How quantization works, and breaks
Works through the scale and zero point mapping, group granularity, and how outliers destroy precision for other weights.
Methods and formats
Separates methods like GPTQ and AWQ from formats like GGUF and MLX, and covers calibration data.
Measuring the loss honestly
Explains why perplexity falls short and how to evaluate quantized models per task, language and refusal behavior.
Choosing, serving and deciding
Gives a procedure for deciding whether to self-host, choosing a quantization, serving it and recording its version.