Join our Newsletter — 33% off our NHI Course
Home› Guides› LLM Quantisation Explained: Smaller Models, Trade-offs and Integrity…
Guide AI Platforms

LLM Quantisation Explained: Smaller Models, Trade-offs and Integrity Checks

PDF for newsletter subscribers · free to read online

← All guides
By NHI Mgmt Group, edited by Lalit Choda Updated 15 November 2026 17 min read
On this page

LLM quantisation is the practice of storing and running a large language model's weights at lower numerical precision, for example 8-bit or 4-bit numbers instead of 16-bit floating point, so the model needs less memory and often runs faster. It is the reason a model that needs a data centre GPU in its original form can run on a single workstation card or a laptop. It also creates a new artefact: a quantised file that is not the model the publisher evaluated, that is often produced by a third party, and that can behave differently from its full-precision parent. This page explains how quantisation works, the main methods and file formats, the accuracy and hardware trade-offs, and the integrity checks a team should run before it trusts a quantised model. It is written for engineers choosing a deployment format and for security and platform teams who approve which model files may run. Last checked: 2026-11-15.

This is an independent reference page from NHI Mgmt Group based on public provider documentation; it is not sponsored or endorsed by any provider.

Key takeaways

  • Quantisation trades precision for memory. Hugging Face's documentation puts an 8B-parameter model at about 16 GB in bf16, about 8 GB at 8-bit and about 4 to 5 GB at 4-bit, with some accuracy cost that varies by method.
  • 8-bit is close to lossless, below 4-bit is not. Hugging Face's benchmarks on Llama 3.1 group 8-bit methods as very close to bf16, 4-bit methods as high accuracy, and sub-4-bit methods as showing a noticeable drop, especially at 2 bits.
  • A quantised model is a different artefact. Research published as "Exploiting LLM Quantization" showed a model can be built to look benign in full precision and behave harmfully only after common quantisation, so evaluating the parent does not clear the child.
  • Provenance metadata exists, but it is self-declared. Hub fields such as base_model and base_model_relation: quantized, and GGUF keys such as general.quantized_by and general.source.repo_url, describe lineage; nothing in them proves it.
  • Treat quantised files like any third-party binary. Pin hashes, prefer files you quantised yourself from a verified parent, sign the result, and evaluate the exact file you deploy. See the Securing AI Model Weights guide for storage and access controls.

What is LLM quantisation?

A trained model is mostly a very large collection of numbers (weights). Training usually happens in 32-bit or 16-bit floating point (FP32, FP16 or bfloat16). Quantisation maps those values onto a smaller set of representable numbers, such as 256 levels for 8-bit integers or 16 levels for 4-bit, and stores a small number of scaling constants so the original range can be approximately reconstructed during computation. The llama.cpp documentation describes the effect plainly: reducing precision shrinks the model and can speed up inference, but may introduce accuracy loss, which is usually measured in perplexity or Kullback-Leibler divergence.

Most modern schemes quantise in blocks. A block of weights (for example 32 or 256 values) shares a scale, and sometimes a minimum, so outliers in one part of the matrix do not ruin precision everywhere else. GGUF's legacy Q4_0 type, for example, stores 32 weights per block and reconstructs each one as the quantised value multiplied by a block scale, while the newer K-quants group blocks into super-blocks with their own quantised scales.

Key terms

TermMeaning
Weight-only quantisationOnly the stored weights are reduced; activations are computed at higher precision (often written W4A16: 4-bit weights, 16-bit activations).
Weight and activation quantisationBoth weights and activations use low precision (for example FP8 W8A8 in vLLM's LLM Compressor support), which needs hardware kernels for that format.
CalibrationRunning a small sample dataset through the model to measure which weights or channels matter most before choosing how to round them. GPTQ and AWQ use calibration; bitsandbytes does not.
Importance matrix (imatrix)In llama.cpp, a file of activation statistics used to decide where precision matters; the GGUF IQ types are defined as using one.
Bits per weight (bpw)The average storage cost per weight including scales; Q4_K_M on Llama 3.1 8B measures about 4.89 bpw in the llama.cpp tables, not exactly 4.
Pre-quantised checkpointA file already converted by someone (you or a third party) and loaded as-is.
On-the-fly quantisationLoading a full-precision checkpoint and quantising it in memory at load time.
KV cache quantisationReducing the precision of the attention cache built during generation, which Hugging Face documents separately from weight quantisation for long generations.

How much memory does quantisation save?

The saving follows from the arithmetic: bf16 uses 2 bytes per parameter, 8-bit uses 1 byte, and 4-bit uses about half a byte plus scale overhead. The llama.cpp quantize documentation publishes measured sizes for Llama 3.1 converted to Q4_K_M.

Model (Llama 3.1)Original sizeQ4_K_M sizeSource
8B32.1 GB4.9 GBllama.cpp quantize README
70B280.9 GB43.1 GBllama.cpp quantize README
405B1,625.1 GB249.1 GBllama.cpp quantize README

As of November 2026. "Original size" in that table is the FP32 conversion input. Two points are easy to miss. First, Hugging Face notes that most GGUF files loaded into Transformers are dequantised to full precision as they load, so they use as much memory as an unquantised model; the memory saving applies in GGML-based runtimes such as llama.cpp. Second, weights are not the whole bill: long contexts add KV cache memory, which is why both Hugging Face and vLLM document KV cache quantisation as a separate option.

What are the main quantisation methods?

The methods below are the ones most practitioners meet. Bit widths, hardware and fine-tuning support are taken from Hugging Face's "Choose a quantization method" page; descriptions come from each method's paper or documentation. Methods are listed alphabetically.

MethodBits (per Hugging Face)How it worksCalibrationFine-tuning in Transformers
AWQ4Activation-aware weight-only quantisation; the paper reports that protecting about 1% of salient weights greatly reduces error, and scales those channels rather than using mixed precisionYes (activation statistics, no backpropagation)Yes, with gptqmodel 5.0.0 or later
bitsandbytes4, 8Quantises at load time; 4-bit NF4 is the data type introduced with QLoRANoYes (the usual QLoRA route)
FP8 (fine-grained and FBGEMM)88-bit floating point; needs CUDA compute capability 8.9+ (fine-grained) or 9.0+ (FBGEMM), or Intel XPUNot stated on the pageNo
GGUF1 to 8File format for GGML-based runtimes with many block quantisation types (Q4_K, Q5_K, IQ types and others)Optional imatrixNo
GPTQ2, 3, 4, 8One-shot quantisation using approximate second-order information; the paper reports quantising 175B-parameter models in about four GPU hoursYesYes
HQQ1, 2, 3, 4, 8Hugging Face lists it as a route below 4 bits without calibrationNoYes
MXFP4 and NVFP444-bit floating point formats; MXFP4 needs compute capability 7.5+, NVFP4 is listed for one Blackwell GPUNot stated on the pageNo
torchao4, 8PyTorch-native quantisation; Hugging Face notes it supports automatic compilationDepends on scheme8-bit schemes only

As of November 2026; Hugging Face lists more than 20 methods in total. On accuracy, Hugging Face's own summary of Llama 3.1 8B and 70B benchmarks is that 8-bit methods are about 2x smaller than bf16 and very close to it in accuracy, 4-bit methods (AWQ, GPTQ, HQQ, bitsandbytes NF4, SINQ) are about 4x smaller with high accuracy, with AWQ and GPTQ often leading, and below 4 bits the drop becomes noticeable. Its advice is to benchmark on your own task and hardware before choosing.

QLoRA deserves a separate mention because it links quantisation to fine-tuning. Its authors describe fine-tuning a 65B model on a single 48 GB GPU by backpropagating through a frozen 4-bit base model into LoRA adapters, using NF4, double quantisation (quantising the quantisation constants) and paged optimisers. The security consequences of adapters are covered in the Fine-Tuning and LoRA Security Guide.

What is GGUF and how do its quantisation types differ?

GGUF is the binary format used by llama.cpp and other GGML-based engines. Unlike tensor-only formats such as safetensors, a GGUF file carries both tensors and a standardised set of key-value metadata, including architecture, tokenizer and, optionally, a chat template stored as a Jinja string in tokenizer.chat_template. The specification requires the file to start with the magic bytes GGUF and describes version 3. Hugging Face's GGUF page lists the quantisation types; a selection is below, with measured figures from the llama.cpp README for Llama 3.1 8B.

GGUF typeNominal bpw (Hugging Face)Measured bpw, Llama 3.1 8B (llama.cpp)Size (GiB)Structure
F161616.0014.96Half-precision float, no quantisation
Q8_088.507.95Legacy round-to-nearest, 32 weights per block
Q6_K6.56256.566.14Super-blocks of 16 blocks of 16 weights, 8-bit block scale
Q5_K_M5.5 (Q5_K)5.705.33K-quant mixture based on Q5_K
Q4_K_M4.5 (Q4_K)4.894.58K-quant mixture based on Q4_K
IQ4_XS4.254.464.174-bit, 256-weight super-blocks, importance matrix
Q3_K_M3.4375 (Q3_K)4.003.74K-quant mixture based on Q3_K
IQ2_XXS2.062.382.232-bit, 256-weight super-blocks, importance matrix

As of November 2026. The _S, _M and _L suffixes are mixtures: llama.cpp keeps some sensitive tensors at a higher type, which is why measured bpw is above the nominal figure (its --pure option disables the mixtures). The README warns that --allow-requantize, which re-quantises already quantised tensors, "can severely reduce quality compared to quantizing from 16bit or 32bit", so a good quant starts from a 16-bit or 32-bit source.

How do you choose a quantisation level?

The choice is a set of factual constraints, not a single right answer.

  • Runtime and hardware first: the format must match the engine. vLLM documents, for example, that AWQ and Marlin kernels do not run on Volta GPUs, that FP8 W8A8 needs Ada or Hopper (and also runs on AMD GPUs), and that GGUF runs on Volta and newer. Apple silicon users are pointed by Hugging Face to Metal quantisation or GGUF.
  • Memory budget: weights plus KV cache for your longest context must fit. Hugging Face's quantization picker estimates this per precision.
  • Task sensitivity: code generation, maths and long structured outputs tend to expose small errors; measure on your own evaluation set rather than relying on perplexity alone.
  • Fine-tuning plans: if you will train adapters, pick a method Hugging Face marks as fine-tuning capable (bitsandbytes, GPTQ, HQQ and others); GGUF is inference-only in Transformers.
  • Silent fallbacks: vLLM notes it may fall back from W8A8 to weight-only execution when a kernel is unavailable, and that setting VLLM_STRICT_QUANT_SCHEME=1 forbids this. Pin the setting if your capacity plan depends on the scheme.
SituationUsual starting pointTrade-off to test
Production GPU serving with headroom16-bit or FP8 on supported hardwareThroughput gain against any accuracy delta on your evaluation set
Fit a large model on one GPU4-bit AWQ or GPTQ checkpointCalibration data quality; regression on domain tasks
Local or edge inference on CPU or Apple siliconGGUF Q4_K_M to Q6_KSpeed against quality; lower types for very small memory
Fine-tuning on limited hardwarebitsandbytes 4-bit NF4 (QLoRA)Adapter quality; merged artefact provenance
Extreme compression2-bit or 3-bit (HQQ, IQ2/IQ3, AQLM)Noticeable accuracy loss; validate every critical task

Does quantisation change model behaviour or safety?

Yes, in two ways. The ordinary way is accuracy loss: rounding changes outputs, and the effect grows as bits fall. That is why llama.cpp measures perplexity and KL divergence against the original, and why any safety or refusal evaluation done on the full-precision model should be repeated on the quantised file.

The deliberate way is quantisation-conditioned behaviour. In "Exploiting LLM Quantization" (Egashira, Vero, Staab, He and Vechev, arXiv 2405.18137), the authors fine-tune a model on a malicious task, compute the set of full-precision weights that all round to the same quantised model, then repair the full-precision model within those constraints so it looks benign. The result behaves well at full precision and badly once quantised with widely used methods. They demonstrated vulnerable code generation, content injection and over-refusal, and noted that an adversary could host the benign-looking full-precision model on a community hub so that users deploying a quantised version are exposed. In MITRE ATLAS terms this sits under AML.T0018 Manipulate AI Model and the AML.T0010.003 Model Repository route of AML.T0010 Supply Chain Compromise. OWASP's LLM03:2025 Supply Chain and LLM04:2025 Data and Model Poisoning entries cover the same class of risk.

The practical lesson is that the deployed precision is the one that must be tested. A clean evaluation report on the original weights says nothing reliable about a quantised derivative, whether you made it or downloaded it.

How do you check a quantised model's integrity and provenance?

Most quantised models on public hubs are produced by community members rather than the original publisher. Metadata helps you trace them, but it is written by whoever produced the file.

EvidenceWhere it livesWhat it tells youLimits
base_model and base_model_relationHugging Face model card YAMLDeclared parent and relationship (quantized, adapter, merge, finetune); the Hub can infer the relationDeclared by the uploader
general.quantized_by, general.source.repo_url, general.base_model.*GGUF metadata keysWho quantised it and the claimed source repository and parentsOptional and self-declared
general.quantization_version and tensor typesGGUF metadata and tensor infoWhich quantisation format version and types the file usesDescribes format, not origin
tokenizer.chat_templateGGUF metadata or tokenizer configThe Jinja template that shapes every promptCan differ from the publisher's template; review it
File hashYour intake record, hub file listingDetects any change after you pinned itProves sameness, not trustworthiness
Signature over the model directoryOpenSSF model signing (Sigstore bundle)Who signed exactly these files; verification recomputes every file hashOnly as good as the signer you trust

The OpenSSF model_signing tool provides sign, verify and digest subcommands and can sign keylessly through Sigstore, with a private key, with certificates or with a PKCS #11 device. Signing your own quantised output closes the gap between "we quantised a verified parent" and "this is the file we quantised". See the Model Signing and OpenSSF Model Signing Guide and Model Signing vs Model Scanning.

Format matters too. GGUF and safetensors store tensors rather than executable code, whereas pickle-based PyTorch files can run code when loaded. GGUF files are still parsed by native code in the runtime, so keep llama.cpp-based engines patched and treat files from unknown sources as untrusted input. The MLOps Security Guide covers serialisation safety and loader settings in depth.

How to adopt quantised models safely

  1. Decide who may produce quants. Prefer quantising in your own pipeline from a parent whose hash or signature you verified. Accept third-party quants only from an allowlist of publishers, recorded in your AI-BOM. See the AI Supply Chain and AI-BOM Guide.
  2. Quarantine and inspect. Download into a quarantine location, record the hash, read the GGUF or model card metadata (declared parent, quantiser, licence, chat template) and reject files with missing or contradictory lineage.
  3. Quantise from 16-bit or 32-bit sources. Avoid requantising existing quants; record the method, version, type and any imatrix or calibration dataset, because calibration data influences the result.
  4. Evaluate the exact file. Run task accuracy, safety and refusal tests, and a comparison against the full-precision parent on the same prompts. Large divergence on specific behaviours is a signal to investigate, not just a quality issue.
  5. Sign and register. Sign the quantised output, store it in your model registry as a distinct version linked to its parent, and let serving load only signed, registered versions. See the Self-Hosted LLM Security Guide.
  6. Roll out and roll back. Deploy behind a flag or to a canary pool, monitor output quality and refusal rates, and keep the previous version ready to restore.

Detection and measurement

  • Lineage coverage: percentage of deployed quantised models with a recorded, verified parent and a signature over the deployed file.
  • Divergence score: KL divergence or task-level agreement between each quantised model and its parent on a fixed evaluation set, tracked per release.
  • Safety delta: change in refusal and harmful-output rates between full-precision and quantised versions on the same red-team prompts.
  • Unregistered model files: count of .gguf, .safetensors or quantised checkpoint files found on inference hosts that are not in the registry.
  • Scheme drift: number of serving instances running a different quantisation scheme from the one approved (for example after a kernel fallback).

Framework mapping

ConcernNIST SP 800-53 Rev. 5NIST CSF 2.0MITRE ATLAS / OWASP
Verify file integrity before loadSI-7, SI-7(6) Cryptographic ProtectionPR.DS-01AML.T0018 Manipulate AI Model
Record parent model and quantiserSR-4 Provenance, CM-8 System Component InventoryID.AM-08OWASP LLM03:2025 Supply Chain
Allowlist third-party quant publishersSR-3, SR-11 Component AuthenticityGV.SC-07AML.T0010.003 Model Repository
Evaluate the deployed precisionSA-11 Developer Testing and EvaluationPR.PS-06OWASP LLM04:2025 Data and Model Poisoning
Detect unapproved model files and scheme driftCM-7, SI-4DE.CM-09AML.T0011 User Execution

Common failure patterns

  • Clearing the parent, shipping the child: the full-precision model passed review, so a community GGUF of it was assumed safe. The quantisation exploit research shows why that inference fails.
  • Trusting the model card: base_model and quantized_by fields were treated as proof of origin, although any uploader can set them.
  • Requantising a quant: a 4-bit file was converted again to another type, compounding error that llama.cpp explicitly warns about.
  • Perplexity as the only test: a small perplexity change hid regressions in code generation or refusals that only task-level tests reveal.
  • Unpinned downloads: serving pulled a repository by name, so a re-uploaded file with the same name replaced the evaluated one without any change record.
  • Ignoring the chat template: a quant shipped a different embedded template from the publisher's, changing system prompt handling and tool formatting.

Practitioner checklist

  • Record the parent model, its verified hash or signature, and the quantisation method, version, type and calibration or imatrix data for every quantised model.
  • Quantise from 16-bit or 32-bit sources in your own pipeline where possible; allowlist any third-party quant publishers.
  • Pin every quantised file by hash and load only signed versions from your registry.
  • Re-run accuracy, safety and refusal evaluations on the exact quantised file, and compare against the parent on the same prompts.
  • Review embedded metadata, including licence and chat template, before approval.
  • Prefer tensor-only formats (GGUF, safetensors) and keep inference runtimes patched.
  • Pin the quantisation scheme in serving configuration and alert on fallbacks.
  • Track lineage coverage, divergence and safety delta as release metrics.

Standards and references

Related NHI Mgmt Group resources: Securing AI Model Weights · Open-Weight Model Security Guide · Self-Hosted LLMs vs Hosted LLM APIs · AI Model Scanning and Supply Chain

PDF for subscribers

Take this guide with you

Every section, table and checklist in one PDF, with links to the sources and related guides. Subscribe to our weekly NHI & AI Identity Journal and we’ll email you a download link.

Free weekly newsletter

Subscribe to the NHI & AI Identity Journal

The latest on NHI and Agentic AI security – articles, research, breaches, news and events every week.

Bonus 33% off our NHI Course when you subscribe.

NHIMG Editorial Note
Researched and written by the NHI Mgmt Group editorial team using AI tools, under the editorial direction of Lalit Choda. Facts are drawn from the sources listed on this page and checked by our automated quality process. Last updated 15 November 2026. Spotted an error? Use our contact page.
NHI Mgmt Group, the independent authority on Non-Human Identity, IAM and Agentic AI security. nhimg.org

PDF Downloads for Newsletter Subscribers

Get the guide

Enter your details and we’ll email you a download link.

The full guide is free to read online. The PDF is a thank you for subscribing.

Fill in all fields marked * and tick the PDF box to continue.