Topic starter
02/08/2026 11:57 am
TL;DR: Byte-pair encoding token efficiency can filter candidate secrets more effectively than entropy after regex capture, showing higher precision and F1 on CredData while keeping recall at 98.6%, according to Aikido’s analysis. The result suggests secrets scanning teams should rethink rarity signals, not just randomness, when tuning detection pipelines.
Editorial analysis by NHI Mgmt Group, based on content published by Aikido: “Rare Not Random: Using Token Efficiency for Secrets Scanning”.
By the numbers:
- Token Efficiency: Precision=57.3%, Recall=98.6%, F1=0.725 on CredData.
- Entropy: Precision=21.1%, Recall=70.4%, F1=0.325 on CredData.
- TE + Word Filter: Precision=80.4%, Recall=95.8%, F1=0.874.
Key questions
Q: What fails when entropy is used as the main secrets filter?
A: Entropy mainly measures randomness, so it can miss the difference between a true secret and a string that only looks noisy.
Q: Why does rarity work better than randomness for candidate secrets?
A: Because many secrets are not maximally random, but they are unusual in ordinary source code and documentation.
Q: What are the signs that a secrets scanner is over-relying on entropy?
A: High false-positive volume, noisy review queues, and poor performance on short or structured secrets are the clearest signs.
Practitioner guidance
- Adopt rarity-aware post-regex filtering Measure candidate secrets by token efficiency after regex capture, then compare the resulting precision and recall against your current entropy threshold.
- Tune separate thresholds for short values Use stricter handling for short strings and known password-like patterns, because token efficiency and entropy behave differently when the candidate is only a few characters long.
- Add a word-based suppression layer Combine rarity scoring with a simple word filter to reduce false positives from strings that contain multiple common words or obvious natural-language fragments.
Bottom line: Token efficiency gives secrets scanning a better post-capture filter than entropy because it measures rarity in a way that maps more closely to how credentials and lookalike strings behave in real repositories.
Explore further
View Full Forum → | NHI Foundation Course → | Our Services → | Read the full analysis →
02/08/2026 1:05 pm
Token efficiency is a governance signal, not just a machine-learning tweak: the article is really about better control over candidate secret triage. When scanners rely on entropy alone, they are treating randomness as the proxy for exposure risk, which is too crude for modern codebases. The stronger model is whether a value looks like something the repository should ever contain. For secrets operations, that shifts the control question from randomness to recognisability, which is a more defensible basis for exception handling and alert prioritisation.
A question worth separating out:
Q: Should teams combine token efficiency with other secrets controls?
A: Yes. Token efficiency is strongest as a filter after regex capture, but it works better when paired with suppression logic, rule tuning, and validation against labelled data. That combination reduces false positives while preserving broad coverage, which is the practical goal for secrets scanning programmes.
👉 Read our full editorial: Token efficiency may outperform entropy in secrets scanning