Token efficiency may outperform entropy in secrets scanning

At a glance

What this is: This is a research-backed analysis of whether token efficiency can improve post-regex secrets filtering by spotting rare, non-natural-language strings better than entropy.

Why it matters: It matters because secrets scanning teams need filters that reduce false positives without missing exposed credentials, especially where API keys, tokens, and other secrets appear in code and tickets.

By the numbers:

• Only 44% of organisations are currently using a dedicated secrets management system.
• 88% of security professionals are concerned about secrets sprawl, with 49% of those in larger organisations described as "very concerned".
• 44% of NHI tokens are exposed in the wild, being sent or stored over platforms like Teams, Jira tickets, Confluence pages, and code commits.

👉 Read Aikido's analysis of token efficiency for secrets scanning

Context

Secrets scanning often starts with a wide regex match, but the real challenge is deciding which captured strings are genuinely sensitive and which are merely unusual-looking text. Token efficiency is being tested here as a rarity signal for secrets scanning, which matters because the first filter determines how much noise reaches the rest of the pipeline.

For IAM and NHI governance teams, this is not just a detection-tuning problem. Secrets, tokens, and service credentials are identity artefacts, and poor filtering leaves standing exposure in code, tickets, and collaboration tools. The underlying question is whether scanners can better distinguish secret-like values from ordinary but uncommon strings without adding operational drag.

Key questions

Q: How should security teams reduce false positives in generic secrets scanning?

A: Start by removing credential types that are public by design, then add contextual checks for keys whose risk depends on platform permissions or surrounding code. The goal is not to suppress more findings overall, but to preserve analyst time for credentials that can actually be used to access sensitive systems.

Q: Why do machine identities need different governance than human accounts?

A: Machine identities lack the human anchors that make access reviews reliable, such as managers, HR events, and predictable offboarding. Their ownership is often ambiguous, their usage is hidden inside workloads, and their permissions can expand quietly. Governance must therefore rely on evidence, accountability, and lifecycle controls built for systems, not people.

Q: What do security teams get wrong about entropy-based secrets detection?

A: They often treat entropy as if it measures secrecy, when it really measures unpredictability. A string can be high-entropy and harmless, or low-entropy and still be a credential. Entropy is useful as a rough filter, but it should not be the only signal deciding what is worth investigating.

Q: How can teams decide whether a new secrets-scanning filter is actually better?

A: Compare precision, recall, and false-positive volume on labelled real-world data, then test by secret category. A better filter is one that preserves true positives while reducing triage noise enough for the team to act quickly. If analysts still cannot keep up, the control is not operationally useful.

Technical breakdown

Why entropy is a blunt filter for secrets detection

Entropy measures unpredictability, so it works reasonably well for strings that look random. The problem is that many non-secret values also look random, while some real secrets are short, patterned, or mixed with words. That makes entropy a weak proxy for secret-likeness once regex has already collected candidates. In practice, entropy can both miss low-entropy passwords and over-score dependency fragments, IDs, or other odd-looking text. Token efficiency tries to solve that by measuring how well a string maps to a tokenizer vocabulary, which is closer to rarity than randomness.

Practical implication: treat entropy as a rough filter, not the deciding control, when tuning secrets scanners.

How token efficiency reflects rarity in natural language

Byte-pair encoding tokenizers compress common words and subwords into fewer tokens, while unusual strings break into many short tokens. Token efficiency is the ratio of character length to token count, so higher values usually indicate text that the tokenizer recognizes as more natural-language-like. Lower values suggest the string is uncommon, fragmented, or secret-like. That is useful in secrets scanning because candidate secrets are often statistically unusual compared with source code or human text. The method does not prove a string is a secret, but it provides a more useful post-capture signal than randomness alone.

Practical implication: use token efficiency as a secondary ranking signal after regex capture, then validate with policy and context filters.

Why a word filter still matters after token-based scoring

Neither entropy nor token efficiency is enough on its own. Words embedded inside a candidate, especially repeated ordinary terms, can indicate that the string is not a secret even if it looks unusual. The article shows that a simple word filter materially improves both methods by removing obvious false positives. That matters because secrets scanners operate in messy source repositories where emails, URLs, dependency strings, and copy-pasted snippets all appear alongside credentials. A layered model remains the right architecture: capture broadly, score for rarity, then apply targeted lexical exclusions.

Practical implication: combine token efficiency with word-based exclusions before deciding whether a candidate reaches review.

Threat narrative

Attacker objective: The attacker seeks to use exposed secrets and tokens for unauthorised access, repository compromise, or downstream service abuse.

1. Entry occurs when a secret, token, or API key is captured from source code or other developer content after regex matching. Escalation follows if the scanner cannot separate true credentials from rare-looking non-secrets, allowing dangerous values to remain untriaged. Impact occurs when exposed secrets are left in repositories or collaboration tools long enough to be abused for downstream access.
2. The attacker objective is to exploit leaked credentials or tokens before they are detected, revoked, or rotated.

NHI Mgmt Group analysis

Token efficiency is best understood as a rarity control, not a secrecy detector. The important shift here is from asking whether a string is random to asking whether it is statistically unusual in real code and language. That distinction matters because many exposed credentials are not perfectly random, and many random-looking strings are harmless. For practitioners, this reinforces the value of layered detection logic rather than reliance on a single numeric threshold.

Secrets scanning is really identity hygiene for machine credentials. API keys, tokens, and certificates are non-human identities in operational form, so scanning quality directly affects NHI exposure management. When false positives swamp triage, real credentials stay active longer than they should. That makes scanner tuning part of identity governance, not just a developer tooling problem.

Rare-looking strings are not the same as risky strings. A tokenizer can help prioritise candidate secrets, but context still decides whether a finding matters. The governance gap is assuming that one statistical signal can replace policy, source context, and lifecycle control. For security teams, the conclusion is clear: rare-text scoring should improve triage, but not replace credential lifecycle controls.

Static secret exposure window: this article highlights the gap between detection and remediation when long-lived credentials sit in code or tickets. The practical lesson is that scanning quality and secret rotation have to be treated as a linked control plane, not separate workflows.

Post-regex filtering is where precision gets won or lost. Regex defines the candidate pool, but secondary scoring decides whether the organisation can keep pace with the volume of findings. That means token efficiency may help teams reduce analyst fatigue without weakening coverage, provided the threshold is calibrated against real repository data. Practitioners should evaluate scanner filters against their own secret patterns, not just benchmark datasets.

What this signals

Token efficiency may improve the quality of secrets triage, but it does not solve secret sprawl on its own. The governance issue is that organisations still struggle to inventory and retire the identities behind the secrets, which means better detection must be paired with faster lifecycle action. The static secret exposure window is shrinking only if discovery and revocation are linked in the same operating model.

For IAM and NHI teams, the practical signal is that secrets scanning is becoming part of broader credential governance. If your pipeline cannot distinguish rare-but-benign strings from actual credentials, you will continue to overburden reviewers while missing high-risk exposures. That makes scanner calibration, rotation workflows, and offboarding controls mutually dependent rather than separate controls.

For practitioners

• Tune post-regex thresholds against your own secret corpus Measure token efficiency and entropy on confirmed secrets from your repositories, then set thresholds separately for API keys, passwords, and tokens. Validate against false positives from dependency strings, URLs, and configuration fragments before deployment.
• Layer lexical exclusions after rarity scoring Add word-based filters for obvious non-secrets such as repeated dictionary terms, safe file types, and known benign patterns. This reduces noise without relying on a single numeric cutoff to do all the work.
• Treat discovered secrets as identity artefacts Route confirmed findings into credential rotation and offboarding workflows, not just ticket queues. A scanner that identifies secrets faster is only useful if the related token, key, or certificate is revoked before it can be reused.
• Benchmark precision and recall separately by secret type Do not rely on aggregate F1 alone. Test whether the filter behaves differently for passwords, short API keys, long tokens, and mixed-format secrets so you can avoid overfitting to one class of credentials.

Key takeaways

• Token efficiency is a useful rarity signal for secrets scanning, but it should be treated as a filter, not a proof of compromise.
• The article’s main operational lesson is that precision matters as much as recall, because false positives can delay response to real exposed credentials.
• For identity teams, the real control objective is to connect detection quality with rotation, revocation, and offboarding workflows.

Key terms

• Token Efficiency: Token efficiency is the ratio of a string’s character length to the number of tokens produced by a tokenizer. In secrets scanning, a low score often indicates an unusual, secret-like string that does not map cleanly to common language patterns.
• Entropy Filter: An entropy filter scores how unpredictable a string is, with more random-looking values receiving higher scores. It is useful for narrowing candidate secrets, but it cannot reliably distinguish a true credential from any other uncommon-looking text.
• Post-regex Filtering: Post-regex filtering is the second-stage analysis applied after a pattern matcher has captured candidate secrets. The purpose is to reduce false positives by scoring, excluding, or prioritising values before human review or automated remediation.
• Secrets Sprawl: The uncontrolled proliferation of sensitive credentials — API keys, tokens, passwords, certificates — across codebases, cloud environments, CI/CD pipelines, and configuration files. In 2024, over 50 million leaked secrets were found on the dark web.

What's in the full article

Aikido's full blog covers the operational detail this post intentionally leaves for the source:

• Exact benchmark methodology for CredData reconstruction and filtering
• Token-efficiency threshold tuning details for short secrets and newline-containing strings
• Full benchmark tables comparing token efficiency, entropy, and word filters
• Code-level implementation notes for the Betterleaks filter

👉 Aikido's full post covers the benchmark tables, implementation notes, and threshold logic in detail.

Deepen your knowledge

The NHI Foundation Level course, the industry's only accredited NHI security programme, covers NHI governance, secrets management, and machine identity security for practitioners building defensible identity controls. It is designed for teams that need to connect detection, lifecycle management, and access governance across their programmes.