What is the difference between full KV cache quantization and outlier token tracing?

How the two methods differ in what they protect

Both techniques are trying to reduce the memory and bandwidth cost of the KV cache, but they make different trade-offs. Full KV cache quantization applies compression to every cached token, so the whole cache gets smaller at once. Outlier token tracing keeps the bulk of the cache compressed, while preserving the small set of unusually sensitive values in full precision so attention math stays more stable.

The practical distinction is scope versus selectivity. Full quantization is simpler and often delivers the biggest savings, but it assumes the error introduced by compressing every token stays acceptable. Outlier tracing adds a decision layer: identify the tokens that are likely to suffer most from quantization error, then exempt only those from the lowest-precision path.

That means the first method is easier to reason about operationally, while the second is more adaptive to the shape of the token distribution. In workloads where a small number of tokens dominate error, selective preservation can maintain quality with less memory overhead than keeping the entire cache at a higher precision.

Why quality loss appears differently in practice

The main failure mode of full KV cache quantization is that the same compression policy is applied to ordinary tokens and unusual tokens alike. If the cache contains values that are numerically extreme or otherwise hard to approximate, the approximation error can spread into the attention scores and distort the model’s next-token behavior.

Outlier token tracing is designed to reduce that specific failure mode by separating the exceptional tokens from the rest. It does not eliminate quantization error, but it narrows the set of tokens exposed to the most aggressive compression, which helps preserve the parts of the cache most likely to influence attention sharply.

In other words, full quantization bets that the average loss is low enough, while outlier tracing bets that you can identify the small number of tokens where loss would be disproportionate. That distinction matters most when model quality is sensitive to rare but important activations, not just to average reconstruction error.

When to prefer one approach over the other

Use full KV cache quantization when memory pressure is the dominant constraint and you can tolerate a broader quality trade-off. It is often the better first pass if the system needs a uniform implementation, predictable compression behavior, and maximum footprint reduction.

Use outlier token tracing when the workload needs more quality preservation without giving up most of the memory benefit. It is especially useful when a few tokens consistently carry outsized influence on attention and the model’s output degrades noticeably under blanket quantization.

The deciding question is not which method is more sophisticated, but which one better matches the model’s sensitivity profile. If the model is robust to uniform approximation, full quantization may be enough. If quality loss is concentrated in rare tokens, selective tracing is usually the safer compromise.

Risk and Threat Considerations

Quantization choice can affect reliability, determinism, and downstream trust in model output. The risk is not usually security compromise in the classical sense, but degraded inference quality, unstable attention behavior, or inconsistent results under high load when compression is too aggressive for the token distribution.

Failure mechanism: Full cache quantization can introduce enough approximation error into unusual tokens that attention weights drift, especially when the cache is large or the model is already operating near a precision boundary. Outlier tracing reduces that risk by preserving the tokens most likely to carry disproportionate influence.

Impact: If the wrong method is chosen for the workload, users may see hallucination-like degradation, poorer retrieval fidelity, or output variance that is hard to diagnose because the cache still appears to be functioning normally.

Practitioner Guidance

What to verify: Compare both methods on the same prompt set, then inspect not just average throughput and memory use, but the tail of quality loss on prompts that contain rare terms, long contexts, or sharp attention shifts. Those are the cases most likely to expose whether blanket quantization is too blunt.

Decision rule: If your acceptance criteria are driven by peak memory reduction alone, start with full KV cache quantization. If you need a better quality-to-memory balance, especially on sensitive prompts, use outlier token tracing or a similar selective preservation strategy.

Practitioner takeaway: The right choice depends on where your error budget sits, full quantization is the simpler and more aggressive compression strategy, while outlier tracing is the safer option when a small number of tokens disproportionately affect output quality.

Related resources from NHI Mgmt Group

• Why does KV cache quantization create accuracy risk in long context LLMs?
• What are the signs that KV cache quantization is hurting LLM output quality?
• What breaks when a sanctions program relies only on exchange names instead of tracing successor entities and token flows?
• How should teams reduce KV cache memory without degrading LLM inference quality?