Quantization
Lower-precision storage and runtime strategies that trade memory, bandwidth, and latency against possible quality loss.
Inference
Model
Concept
- Activation quantization
Quantizing live intermediate activations during execution instead of only shrinking stored weights.
- Calibration
Running representative inputs through a model to estimate the ranges or scales a quantized deployment will need.
- Dynamic quantization
Choosing some quantization scales or converted values at runtime instead of fixing every low-precision detail ahead of deployment.
- KV cache quantization
Compressing the cached keys and values kept across autoregressive decode steps to reduce long-context serving memory.
- Post-training quantization
Converting an already trained model to lower precision after training so it can use less memory or run more efficiently.
- Quantization
Storing or computing with fewer bits so models use less memory and sometimes run more efficiently, with tradeoffs around quality, calibration, and hardware support.
- Quantization-aware training
Training or fine-tuning with simulated low-precision effects so a model can adapt before deployment.
- Weight-only quantization
Compressing stored model weights into fewer bits while most runtime activations stay at a higher precision.
- Why 4-bit models are not exactly 4x faster
Why lowering model weights to 4-bit often cuts memory more than it cuts end-to-end latency or throughput.
Paper
Training
- FP4 Quantization-Aware Training
A DeepSeek-V4 training step that prepares the model for very low precision by exposing it to quantization effects during training.
- On-Policy Distillation
A DeepSeek-V4 post-training routine where the student is refined on trajectories produced under its current policy rather than only on frozen offline traces.
- Specialist Training
A DeepSeek-V4 post-training pattern that sharpens separate capability areas before they are merged back into the final stack.
System
- Deployment
A serving system practice that turns a trained model plus its runtime stack into a live, versioned inference service on real hardware.
- Inference Engine
The serving runtime layer that takes model weights plus live requests and turns them into token outputs by coordinating kernels, scheduling, memory movement, and cache state on real hardware.