Quantization: how much precision do you actually need?
Modern AI runs on neural networks, and a neural network is just a huge list of millions of numbers called weights. Those numbers are what the model learned during training — together, they are the model.
Two things decide whether that model is any use on a small device: how big it is, and how accurate it is. Both come down to a single choice: how much precision you keep in each of those numbers.
What “precision” means
Precision is how exact each weight is. Round the numbers off and the model gets smaller, and maybe a little less accurate. That trade is quantization.
By default a weight is a 32-bit float (FP32), 32 binary digits, more exactness than it usually needs. FP16 (16-bit) keeps fewer decimals: the numbers barely change and the model halves in size.
INT8 (8-bit) goes further, snapping each weight to a much coarser grid of values, coarse enough that it can start to change what the model predicts. More generally FP16 is a narrowing, while INT8 is an approximation.
The result
To make this concrete, we ran a simulation on a real example: a model that looks at a chest X-ray image and checks it for 14 different diseases. It is a pretrained DenseNet-121 from torchxrayvision, and we ran it on a $249 Jetson Orin Nano Super.
Accuracy is macro-AUROC: roughly, how reliably the model ranks a genuinely diseased scan above a healthy one: 1.0 is perfect, 0.5 is a coin toss (averaged over the 14 diseases, tested over 2000 images).
| Precision | Throughput | Speedup | Macro-AUROC | Change vs FP32 | Model size |
|---|---|---|---|---|---|
| FP32 | 191 img/s | 1.0× | 0.7405 | — | 35.4 MB |
| FP16 | 513 img/s | 2.7× | 0.7405 | 0.000 | 14.9 MB |
| INT8 | 1036 img/s | 5.4× | 0.6868 | −0.054 | 8.8 MB |
FP16 leaves the accuracy exactly where it was: 0.7405, identical to full FP32, but it halves the model size, and increases throughput by over 2x.
INT8 also doubles throughput and nearly halves the file size, but costs 0.054 macro-AUROC, which represent about 7% of the model’s ability to tell a sick scan from a healthy one.
The boundary
There is no single right precision. It depends on the problem: how small and how fast the model has to be, and how much accuracy you can afford to give up.
Quantization is a dial you tune to that, from a near-lossless FP16 to an aggressive INT8 that buys another 2×. Here, FP16 is our default: nearly 3× the throughput of full precision, with the accuracy untouched.