Quantisation represents model values with reduced numerical precision to change resource requirements. It can reduce memory needs, but its effect on quality and speed depends on the model, method and hardware. Evaluate the exact version rather than assuming every smaller file behaves equivalently.
What gets smaller
A model stores numerical values. Quantization represents some of those values with fewer bits, potentially reducing memory requirements while introducing approximation. The practical effect varies by method, model and hardware; a smaller file alone does not prove a better experience.
Compare the same work
Run the original and quantized versions on representative inputs. Include difficult extraction cases, unfamiliar terms and the longest documents you expect. Measure correctness alongside memory use and response time. A change that barely affects an average score may still matter for one critical field.
Choose an operating point
For a local assistant, fitting comfortably in memory may avoid severe slowdowns. For another task, a loss of accuracy may outweigh that benefit. Record the model version and quantization format with your results so the comparison can be reproduced. Keep a small regression set for future updates. The goal is a model that meets the required quality on available hardware, rather than the smallest model file you can find.
A worked example
Compare two supported quantisations of the same model on one machine using identical prompts. Include a precise extraction task and a longer explanation. Record memory, latency and errors. If one configuration fits available memory but makes more mistakes on critical fields, the reduced footprint is a trade-off to assess, not an automatic improvement.
Questions readers ask
Does quantisation always make a model faster without changing quality?
No. Results depend on implementation, hardware and task. Measure both operating behaviour and quality for the exact configuration you intend to use.
Follow the evidence.
AI-assisted explanation with linked technical references. Examples are illustrative learning exercises, not reported customer deployments or experimental results.
Public sources support the factual background. Analysis and illustrative exercises are identified in the text. How we assess AI · Editorial standards
Spot a factual error or an outdated detail?



