Notifications
Clear all
Topic starter
05/07/2026 6:43 pm
TL;DR: Tokenizers sit at the core of LLM interaction, and HiddenLayer shows how glitch tokens, invisible Unicode, and TokenBreak techniques can manipulate prompts, evade classifiers, and disrupt guardrails before a model ever evaluates the content. The security boundary now extends beyond the model itself to preprocessing, normalization, and token-level inspection.
NHIMG editorial — based on content published by HiddenLayer: Tokenization Attacks on LLMs: How Adversaries Exploit AI Language Processing
Questions worth separating out
Q: How should security teams defend LLMs against tokenization attacks?
A: Defence should start before the model sees the prompt.
Q: Why do tokenization attacks bypass prompt-injection classifiers?
A: They work because the classifier and the target model may not interpret the same text in the same way.
Q: What breaks when hidden Unicode is allowed into AI workflows?
A: Hidden Unicode breaks the assumption that human review and machine review are seeing the same payload.
Practitioner guidance
- Normalize all inbound text before evaluation Strip or canonicalise invisible Unicode, variation selectors, and other non-printing character classes before prompts reach classifiers, routing logic, or model calls.
- Test classifier and model tokenization together Run the same malicious and near-miss prompts through both the guardrail layer and the target model, then compare verdicts for divergence caused by token boundary changes.
- Add token-aware prompt-injection cases to QA Build a regression set that includes glitch tokens, Unicode payloads, and word-splitting variants so security testing covers evasion, not only obvious malicious strings.
What's in the full report
HiddenLayer's full research covers the operational detail this post intentionally leaves for the source:
- Tokenizer examples showing how specific prompt variants change model and classifier behaviour in practice
- Step-by-step illustrations of invisible Unicode payload construction and why they evade human review
- TokenBreak examples that demonstrate how small text perturbations alter prompt-injection verdicts
- Discussion of how HiddenLayer positions AI-native defences against tokenizer abuse in production workflows
👉 Read HiddenLayer's research on tokenization attacks against LLMs →
Tokenization attacks in LLMs: are your guardrails catching them?
Explore further
05/07/2026 6:58 pm
Tokenization is now a governance boundary, not just a model implementation detail. The article shows that adversaries can attack the layer that converts text into tokens before any policy decision is made. That means security teams cannot treat prompt content as trustworthy simply because it looks normal to a human reviewer. The practical conclusion is that LLM governance must extend to preprocessing, not only inference-time controls.
A few things that frame the scale:
- 28.65 million new hardcoded secrets were detected in public GitHub commits in 2025 alone, a 34% year-over-year increase and the largest single-year jump ever recorded, according to The State of Secrets Sprawl 2026.
- 24,008 unique secrets were exposed in MCP configuration files in 2025 alone, the protocol's first year of widespread adoption, according to The State of Secrets Sprawl 2026.
A question worth separating out:
Q: How can organisations tell if their LLM guardrails are actually working?
A: They should test whether benign-looking variants of the same malicious prompt receive consistent verdicts across sanitizers, classifiers, and the model itself. If a small token change flips the decision, the control is brittle and vulnerable to evasion rather than genuinely effective.
👉 Read our full editorial: Tokenization attacks expose a new bypass layer in LLM security
