Token efficiency may outperform entropy in secrets scanning

At a glance

What this is: Aikido argues that token efficiency is a better post-regex filter for candidate secrets than entropy because it captures how unusual a string is, not just how random it looks.

Why it matters: Secrets detection teams should rethink their false-positive filters, because treating rarity as a signal can improve precision without giving up most true positives.

By the numbers:

• Token Efficiency: Precision=57.3%, Recall=98.6%, F1=0.725 on CredData.
• Entropy: Precision=21.1%, Recall=70.4%, F1=0.325 on CredData.
• TE + Word Filter: Precision=80.4%, Recall=95.8%, F1=0.874.
• Entropy + Words Filter: Precision=76.6%, Recall=67.1%, F1=0.715.

Context

Secrets scanners often need a second-stage filter after regex capture because broad pattern matching pulls in many strings that are not actually secrets. The core governance problem is not whether a string looks random, but whether the detector can distinguish rare credential-like material from ordinary text and benign identifiers.

Entropy has been used as a proxy for randomness, but randomness and rarity are not the same thing. In NHI and secrets governance, that matters because false positives slow triage while false negatives leave exposed credentials in code, pipelines, and logs. The article tests whether token efficiency can improve that decision point without turning the scanner into a different class of detector.

Key questions

Q: What fails when entropy is used as the main secrets filter?

A: Entropy mainly measures randomness, so it can miss the difference between a true secret and a string that only looks noisy. In practice that creates too many false positives and still leaves short or structured secrets underdetected. A better post-capture filter should assess whether the string is rare in natural text, not only whether it is random.

Q: Why does rarity work better than randomness for candidate secrets?

A: Because many secrets are not maximally random, but they are unusual in ordinary source code and documentation. Rarity captures how far a string sits from natural-language distribution, which is closer to the real triage problem after regex has already found a candidate. That makes rarity a more discriminating signal than entropy alone.

Q: What are the signs that a secrets scanner is over-relying on entropy?

A: High false-positive volume, noisy review queues, and poor performance on short or structured secrets are the clearest signs. If your scanner catches many values that look random but do not behave like credentials, your filter is too blunt. The control problem is not discovery, but classification after capture.

Q: Should teams combine token efficiency with other secrets controls?

A: Yes. Token efficiency is strongest as a filter after regex capture, but it works better when paired with suppression logic, rule tuning, and validation against labelled data. That combination reduces false positives while preserving broad coverage, which is the practical goal for secrets scanning programmes.

Technical breakdown

Why entropy is an imperfect secrets filter

Shannon entropy measures unpredictability within a string, which is useful when you want to know whether character distribution looks random. In secrets scanning, that is only a partial proxy. Many real secrets are short, structured, or mixed-format, while many non-secrets such as UUIDs, encoded values, and odd dependency strings can also look random. Entropy therefore tends to blur a key distinction: whether a string is statistically unusual in human-written code and text, not merely noisy. For post-regex filtering, that is a material gap because the scanner has already decided the string is worth inspecting. The question becomes classification quality, not candidate discovery. Practical implication: treat entropy as a weak signal for candidate ranking, not as the primary gate for secrets triage.

Practical implication: use entropy as a supporting signal, not the main decision rule, after regex capture.

How byte-pair encoding captures rarity

Byte-pair encoding, or BPE, tokenizes text based on learned frequency patterns. Common words and subwords become compact tokens, while rare or unnatural strings fragment into many smaller tokens. That makes the token count a useful proxy for how far a string deviates from natural language and normal source text. In the article’s framing, token efficiency is simply string length divided by token count. Higher efficiency means the tokenizer recognizes the text well; lower efficiency means it does not. For secrets scanning, that is attractive because secrets often behave like out-of-vocabulary text, especially when they contain key-like prefixes, unusual character mixes, or encoded fragments. Practical implication: BPE-based rarity scoring can help a scanner separate believable text from suspicious candidate secrets.

Practical implication: score candidate secrets by token efficiency before final alerting or suppression.

Why token efficiency can improve post-regex precision

The article’s key technical point is that token efficiency is not trying to detect secrets from scratch. It assumes regex has already captured a candidate and then asks whether that candidate resembles everyday language enough to be trusted. That shift matters because many false positives survive entropy checks yet remain poor fits for natural text models. Token efficiency therefore functions as a better rarity discriminator than entropy in this use case, especially for generic secrets where structure is weak and labels are noisy. The result is fewer false positives without collapsing recall, which is exactly the kind of control improvement detection pipelines need. Practical implication: place rarity-based filtering after capture and before escalation to review or blocking.

Practical implication: insert rarity-based filtering between capture and analyst review to cut noise without losing broad coverage.

NHI Mgmt Group analysis

Token efficiency is a governance signal, not just a machine-learning tweak: the article is really about better control over candidate secret triage. When scanners rely on entropy alone, they are treating randomness as the proxy for exposure risk, which is too crude for modern codebases. The stronger model is whether a value looks like something the repository should ever contain. For secrets operations, that shifts the control question from randomness to recognisability, which is a more defensible basis for exception handling and alert prioritisation.

Rarity beats randomness when the control point is post-capture: regex already does the broad collection work, so the filter should answer a different question. Entropy is useful for spotting noisy strings, but it is not tuned to human-language distribution and therefore misses the difference between uncommon text and true credential-like material. Token efficiency aligns more closely with that boundary, which is why it can improve precision while preserving recall. Practitioners should treat that as an argument for control layering, not a replacement mindset.

Secrets governance needs fewer generic thresholds and more context-aware filtering: the article shows that one-size-fits-all entropy cutoffs create unnecessary analyst load and inconsistent detection quality. That is especially relevant where repository content mixes code, documentation, encoded values, and secrets. A context-aware filter that understands tokenisation behaviour offers a cleaner way to narrow candidate lists before human review. The practical lesson is to tune scanners around the kinds of strings your environment actually produces, not around an abstract notion of randomness.

Open-source secrets scanning should optimise for precision after discovery, not purity before it: the article’s benchmark work shows that post-regex classification can materially change operational outcomes. In broader identity and NHI governance, the lesson is familiar: discovery is only the start, and control quality depends on what happens next. If teams cannot separate valid secrets from noisy lookalikes efficiently, they will underinvest in the very signals that find exposed credentials. The practitioner conclusion is to invest in smarter filtering where triage cost is highest.

Token efficiency gives secrets teams a named concept worth tracking: rarity-aware filtering: this is the idea that candidate secrets should be judged by how atypical they are in natural text, not only by how random they appear. That framing is useful because it can be measured, tuned, and compared across pipelines. It also creates a more stable basis for deciding which findings deserve escalation. Practitioners should evaluate their secret scanners on rarity-aware filtering quality, not just on raw detection volume.

What this signals

Rarely looking like everyday text is often a better indicator of candidate secrets than looking random. That distinction matters because scanners operate after capture, when the real task is separating useful findings from noise. Teams should expect better triage quality when rarity-based filters sit between regex and analyst review.

Token efficiency is a useful control concept because it matches how teams actually manage secrets workflows. Discovery creates volume, but triage determines whether the programme stays operational. Practitioners should test whether their current thresholds reduce review noise without creating blind spots for short credentials and structured tokens.

For practitioners

• Adopt rarity-aware post-regex filtering Measure candidate secrets by token efficiency after regex capture, then compare the resulting precision and recall against your current entropy threshold.
• Tune separate thresholds for short values Use stricter handling for short strings and known password-like patterns, because token efficiency and entropy behave differently when the candidate is only a few characters long.
• Add a word-based suppression layer Combine rarity scoring with a simple word filter to reduce false positives from strings that contain multiple common words or obvious natural-language fragments.
• Benchmark on your own secret corpus Validate the filter against labelled examples from your repositories, pipelines, and build logs so the threshold reflects your actual code and secret mix.
• Review generic secret rules for entropy dependence Identify rules that rely heavily on entropy alone and test whether token efficiency improves alert quality without reducing recall below an acceptable operational threshold.

Key takeaways

• Token efficiency gives secrets scanning a better post-capture filter than entropy because it measures rarity in a way that maps more closely to how credentials and lookalike strings behave in real repositories.
• The article’s benchmark results show a large precision gap between token efficiency and entropy, while recall remains high enough to preserve broad detection coverage.
• Teams should evaluate secrets scanners on labelled local data and combine rarity scoring with suppression rules, rather than trusting entropy as the primary decision gate.

Key terms

• Token Efficiency: Token efficiency is the ratio of a string’s character length to the number of tokens produced by a tokenizer. In secrets scanning, a low score often indicates an unusual, secret-like string that does not map cleanly to common language patterns.
• Entropy: Entropy is the lower bound on how many bits are needed to represent a stream of data without losing information. In this context, it measures uncertainty in the symbol distribution, so lower entropy means more predictable data and greater compression potential.
• Post-regex Filtering: Post-regex filtering is the second-stage analysis applied after a pattern matcher has captured candidate secrets. The purpose is to reduce false positives by scoring, excluding, or prioritising values before human review or automated remediation.
• False Positive: A false positive is a scanner result that looks like a secret but is not actually sensitive. In secret governance, false positives matter because they consume analyst time, weaken trust in alerts, and can delay response to the findings that truly change exposure and access risk.

Deepen your knowledge

NHI Mgmt Group covers identity security, NHI governance, and agentic AI through independent research, practitioner guides, and the NHI Foundation Level course. Explore nhimg.org for resources that connect identity governance to the broader security disciplines your programme depends on.