Why does better prediction make compression and language modelling look similar?

Why This Matters for Security Teams

Better prediction makes compression and language modelling look similar because both depend on the same core task: assigning the right probability to the next symbol. When the model is more accurate, it can represent uncertainty with fewer bits in compression and with lower loss in language prediction. That shared math is not just academic; it shapes how teams think about machine-generated text, logs, telemetry, and the secrets embedded in those workflows.

For security teams, the practical issue is that better predictive systems can also make automated content generation, detection, and exfiltration faster and cheaper. If a system can anticipate common patterns, it can compress them efficiently, but it can also reproduce them efficiently, including credentials, tokens, and other sensitive strings if they are exposed in training or runtime context. NHI Management Group notes in the Ultimate Guide to NHIs that only 5.7% of organisations have full visibility into their service accounts, which means many teams are already managing machine-generated outputs without full identity context. Current guidance from NIST SP 800-53 Rev 5 Security and Privacy Controls reinforces that visibility and accountability are not optional when automated systems touch sensitive data.

In practice, many security teams encounter the risks only after a model has already reproduced sensitive patterns at scale, rather than through intentional data governance.

How It Works in Practice

Compression and language modelling both estimate probability distributions over possible next symbols, but they use that estimate for different ends. In compression, a symbol that is highly probable gets fewer bits; a surprising symbol gets more. In language modelling, the same probability distribution is used to choose or score the next token. The better the prediction, the smaller the expected coding cost and the better the model appears at forecasting text.

That symmetry becomes important in security engineering because models learn from patterns that may include secrets, service account names, API key formats, or routine operational language. If those patterns are frequent, the model compresses them well and may also regenerate them with high confidence. That is why identity and secrets governance matter even in seemingly abstract AI systems. NHI Management Group’s Ultimate Guide to NHIs highlights that 96% of organisations store secrets outside of secrets managers in vulnerable locations, which increases the chance that machine-learning systems will ingest them.

• Compression perspective: lower entropy means fewer bits are required to encode the same sequence.
• Language-modelling perspective: lower entropy means the model assigns higher likelihood to the correct next token.
• Security perspective: better prediction can make sensitive patterns easier to reproduce if they enter the model context.
• Operational response: treat prompts, logs, embeddings, and training corpora as sensitive data sources, not just model inputs.

Policy should therefore cover data minimisation, secret scanning, and access controls for any system that can observe or generate operational text, aligned with the control expectations in NIST SP 800-53 Rev 5 Security and Privacy Controls. These controls tend to break down when teams assume model outputs are harmless because the system “only predicts text” and ignore the fact that prediction quality can preserve sensitive structure.

Common Variations and Edge Cases

Tighter probability modelling often increases operational complexity, requiring organisations to balance prediction quality against privacy and leakage risk. That tradeoff is especially sharp in systems that mix compression, retrieval, and generation, because each stage can reinforce the same learned patterns in different forms. Current guidance suggests treating any pipeline that stores, tokenises, or replays operational text as a potential exposure path.

There is no universal standard for this yet, but several edge cases are clear. A model trained on highly repetitive infrastructure logs may appear excellent at compression-like prediction while still being unsafe if those logs contain tokens, account names, or authentication artefacts. Likewise, a model that performs well on benchmark perplexity can still be poor at secure handling of secrets because predictive accuracy does not imply governance. The key operational distinction is that compression rewards low uncertainty, while security rewards low exposure.

For practitioners, that means the right control set is not just about model quality. It also includes source-data classification, secret redaction before training, and review of any system that can reconstitute sensitive text at runtime. This is where the identity layer matters as much as the model layer, especially when service accounts and API keys are part of the data exhaust. In high-churn environments, the gap between what the model predicts and what the organisation can safely disclose widens quickly.

Related resources from NHI Mgmt Group

• Why do passkey rollouts often look better on mobile than on desktop?
• Why do manual approvals make lifecycle automation look less effective than it is?
• How should security teams govern AI adoption when maturity scores look better than reality?
• Why do retries sometimes make outages worse instead of better?