Large models are becoming increasingly capable, but deployment costs are also rising. Quantization and distillation are currently the two most mainstream technical routes for "making models smaller and faster."
Quantization: Reducing Precision
The core idea is to compress model weights from high-precision formats (FP32/FP16) to low-precision formats (INT8/INT4 or even lower).
Post-Training Quantization (PTQ)
The simplest and most direct approach — reduce precision immediately after training. It typically compresses model size by 75-80% with acceptable accuracy loss. Suitable for rapid deployment.
Quantization-Aware Training (QAT)
The model adapts to low-precision conditions during training. It yields better results than PTQ but requires retraining.
Quantization-Aware Distillation (QAD)
A new method recently developed by NVIDIA — the student model learns two things simultaneously: adapting to quantization errors and aligning with the full-precision teacher model. Their NVFP4-QAD research demonstrated that accuracy can be recovered even at FP4 precision.
Distillation: Knowledge Transfer
A small model (student) mimics the behavior of a large model (teacher). The student model typically achieves 90-95% of the teacher's performance while being much smaller in size.
DeepSeek R1's distilled version is a typical case — the 32B distilled model matched o1-mini.
Pruning: Directly Cutting Parameters
Identify and remove unimportant parameters (weights, neurons, or entire layers). Typically 30-50% of parameters can be removed with almost no performance loss.
Speculative Decoding: Parallel Verification
A small "draft model" quickly generates multiple candidate tokens, and the large model verifies them in parallel. The principle is similar to "rough sketch first, then refine," significantly reducing latency.
Trends
NVIDIA predicts that 2026 will be "a year of divergence between frontier models and efficient models." Combining multiple optimization techniques is becoming mainstream — not choosing one, but using quantization + distillation + pruning together.
On the hardware side, ASIC accelerators, chiplet designs, and analog inference chips are maturing, and GPU is no longer the only inference option.
Source: CocoLoop, NVIDIA Technical Blog