Unicode Security: Detect Homoglyphs, Invisible Characters and Normalisation Flaws

Unicode security is the discipline of handling text so that characters which look identical, or cannot be seen at all, cannot be used to deceive people or bypass software checks. Unicode contains more than one way to write what looks like the same string: Cyrillic "а" and Latin "a", a fullwidth "Ａ" and a plain "A", or a word with a zero width space hidden inside it. Attackers use these differences for lookalike domains and usernames, source code that reads differently from how it compiles (Trojan Source), and hidden instructions for AI models. This guide explains confusables, invisible and bidirectional characters, normalisation (including NFKC for usernames) and the Unicode Consortium's own security standards, then gives developers and security teams concrete controls. Domain-specific defences are in the Typosquatting and Lookalike Domains Guide and prompt-specific ones in ASCII Smuggling and Invisible Prompts.

Key takeaways

• Visual equality is not string equality. UTS #39 defines confusable detection with a "skeleton" so systems can tell when two different strings would look the same to a person.
• Invisible characters are real characters. Zero width spaces, joiners, bidirectional controls and the Tags block (U+E0000 to U+E007F) change how text renders or is interpreted while showing nothing.
• Normalise once, early and consistently. Pick NFC or NFKC deliberately, apply it before validation, and use the same rule at registration, login and lookup.
• NFKC is not a security filter. It folds fullwidth and ligature forms but leaves Cyrillic lookalikes alone; you still need confusable and script checks.
• UTR #36 is stabilised. The Unicode Consortium says its guidance is superseded in parts by UTS #39, UTS #55 and UAX #31, so build on those.

What it is

Unicode assigns code points to characters from every writing system plus symbols and control characters. Security problems arise from three properties of that design.

• Confusables (homoglyphs): different code points that look the same or nearly the same. Unicode's confusables data maps, for example, CYRILLIC SMALL LETTER A (U+0430) to LATIN SMALL LETTER A, and Cyrillic es, ie and er to Latin c, e and p. A homoglyph attack substitutes one for another (CAPEC-632 Homograph Attack via Homoglyphs).
• Invisible and format characters: characters of general category Cf (format) that render as nothing or only affect layout, such as ZERO WIDTH SPACE (U+200B), ZERO WIDTH NON-JOINER (U+200C), ZERO WIDTH JOINER (U+200D), WORD JOINER (U+2060), ZERO WIDTH NO-BREAK SPACE (U+FEFF) and SOFT HYPHEN (U+00AD).
• Multiple encodings of the "same" text: precomposed and decomposed accents, fullwidth forms, ligatures and compatibility characters such as KELVIN SIGN (U+212A), which canonically decomposes to Latin "K". Normalisation exists to reconcile these, but each normalisation form makes different choices.

How it works

Confusable detection with UTS #39

UTS #39, Unicode Security Mechanisms (version 18.0.0, revision 34, dated 27 August 2026), computes a skeleton for each string: apply NFD, remove default ignorable characters, map each character to its prototype from the confusables data file (confusables.txt), then apply NFD again. Two strings are confusable when their skeletons match. The standard is explicit that the skeleton is for internal comparison only, never for display or as a normalised form to store. It distinguishes:

IDN and punycode

Internationalised domain names are converted label by label into an ASCII form for DNS, with the prefix xn--; Chromium's documentation gives öbb.at becoming xn--bb-eka.at. Browsers decide per label whether to display Unicode or the punycode form. Chrome's published policy, in place since Chrome 51, shows punycode when UTS #46 conversion fails or a label contains characters not allowed in identifiers under UTS #39, among other rules. Display policies reduce but do not remove risk, which is why lookalike domain monitoring still matters.

Bidirectional controls and Trojan Source

Unicode's bidirectional algorithm lays out mixed left-to-right and right-to-left text. Explicit controls such as RIGHT-TO-LEFT OVERRIDE (U+202E), the embeddings (U+202A, U+202B), POP DIRECTIONAL FORMATTING (U+202C) and the isolates (U+2066 to U+2069) reorder how text appears. The Trojan Source research by Nicholas Boucher and Ross Anderson (University of Cambridge) showed that placing these characters in comments and string literals can make source code display differently from how compilers read it, producing apparent "commented-out" code that executes, early returns hidden in comments, and "stretched" strings. It was tracked as CVE-2021-42574, with a homoglyph identifier variant as CVE-2021-42694. The same override trick has long been used to disguise file extensions in file names (see MITRE ATT&CK T1036 Masquerading).

Tags and hidden text for AI models

The Tags block (U+E0000 to U+E007F) includes invisible counterparts of ASCII letters, such as TAG LATIN CAPITAL LETTER A (U+E0041). Most interfaces render nothing, yet tokenisers can pass them to language models, which is how hidden instructions are smuggled into prompts, documents and tool output (MITRE ATLAS AML.T0068 LLM Prompt Obfuscation). The ASCII Smuggling and Invisible Prompts guide covers detection and stripping for LLM inputs.

Normalisation forms

UAX #15 warns that the compatibility forms "must not be blindly applied to arbitrary text" because they remove distinctions that can matter, and notes that no normalisation form is closed under concatenation: joining two normalised strings can yield an unnormalised one. Strings containing unassigned code points may normalise differently in a later Unicode version, which is why identifier systems should reject unassigned characters.

Why it matters

Username handling done properly

RFC 8265 (PRECIS, October 2017, obsoleting RFC 7613) gives a tested recipe. The UsernameCaseMapped profile maps fullwidth and halfwidth characters to their decompositions, lowercases, applies NFC and enforces the Bidi Rule from RFC 5893; UsernameCasePreserved does the same without case mapping. Comparison is then an exact octet match. For passwords the OpaqueString profile maps non-ASCII spaces to U+0020 and applies NFC but no case mapping, so that a password typed on a different keyboard still matches without creating false accepts. Note that PRECIS uses NFC plus width mapping rather than NFKC; whichever you choose, the essential rule is consistency across every code path that touches the identifier, including SSO, SCIM provisioning and support tools. See the SCIM Provisioning Guide.

Common misconceptions

• "NFKC makes lookalikes safe." It folds compatibility forms such as fullwidth letters and ligatures, but Cyrillic, Greek and Latin lookalikes are distinct characters and survive normalisation. Use the UTS #39 skeleton for that.
• "Just block non-ASCII." ASCII-only is one UTS #39 restriction level and may suit internal service account names, but it excludes most of the world's users for customer-facing names and still allows "rn" versus "m".
• "Invisible characters are harmless whitespace." They change tokenisation, matching and display; in source code and prompts they carry meaning.
• "Bidi characters should be banned everywhere." UTS #55 does not ban them because legitimate right-to-left text needs them; it recommends editors display code by lexical structure and that tools flag unterminated or misplaced controls.
• "The skeleton is a canonical form." UTS #39 says it is for comparison only; storing or displaying skeletons corrupts data.
• "UTR #36 is the standard to follow." It has been stabilised and unmaintained since 2014; follow UTS #39, UTS #55 and UAX #31.

Related concepts

• Typosquatting and lookalike domains: brand protection and takedown for domains. See the Typosquatting and Lookalike Domains Guide.
• ASCII smuggling: the prompt-specific use of tag characters. See ASCII Smuggling and Invisible Prompts.
• Prompt injection: the broader class hidden text feeds. See the Prompt Injection Guide.
• Input validation and canonicalisation: NIST SP 800-53 SI-10 Information Input Validation and OWASP ASVS V1 Encoding and Sanitization.
• UAX #31 Unicode Identifiers and Syntax: which characters programming languages and protocols should allow in identifiers.

Implementing controls

1. Inventory identifier fields. List usernames, display names, tenant and workspace names, domains, file names and API object names, and decide the allowed restriction level for each.
2. Pick one normalisation rule per field and centralise it. One library function used at registration, login, password reset, provisioning and search. Reject unassigned code points.
3. Add confusable collision checks. On creation, compute the UTS #39 skeleton and reject or flag names whose skeleton matches an existing privileged or brand name.
4. Strip or flag invisible characters where they have no business. Remove default ignorable and Tags block characters from identifiers and LLM inputs; in free text, flag rather than silently delete, and keep the original for audit.
5. Guard source code. Enable compiler and linter warnings for bidi controls and confusable identifiers, add a CI check that fails on unterminated bidi controls in comments and strings, and configure review tools to show hidden characters, as the Trojan Source authors recommend.
6. Test. Add test cases with fullwidth, Kelvin sign, Cyrillic lookalike, zero width and tag characters to every identifier and validation test suite, and use Unicode's NormalizationTest.txt to verify normalisation libraries.

Metrics

• Share of identifier fields with a documented normalisation and restriction policy.
• Confusable-collision rejections per month at account and tenant creation.
• CI builds failed for bidi or confusable characters, and time to fix.
• Invisible character detections in LLM inputs and inbound documents.

Practitioner checklist

• Apply one documented normalisation rule (NFC with width mapping per RFC 8265, or NFKC with case folding) to each identifier field, before validation.
• Use the same normalisation function at registration, login, reset, SSO, SCIM and admin tools.
• Reject unassigned code points and default ignorable characters in identifiers.
• Check new usernames, tenant names and display names against existing privileged names with the UTS #39 skeleton.
• Restrict machine and service account names to ASCII-Only or Single Script.
• Fail CI on unterminated bidi controls and flag confusable identifiers in source code.
• Strip the Tags block and zero width characters from content before it reaches an LLM, and log detections.
• Monitor lookalike domains of your brands, including whole-script confusables.

Standards and references

• Unicode Consortium: UTS #39 Unicode Security Mechanisms
• Unicode Consortium: UTS #55 Unicode Source Code Handling
• Unicode Consortium: UAX #15 Unicode Normalization Forms
• Unicode Consortium: UTR #36 Unicode Security Considerations (stabilized)
• IETF: RFC 8265 Preparation, Enforcement, and Comparison of Internationalized Strings Representing Usernames and Passwords
• University of Cambridge: Trojan Source
• Chromium: Internationalized Domain Names (IDN) in Google Chrome
• MITRE CAPEC: Homograph Attack via Homoglyphs

Related NHI Mgmt Group resources: Typosquatting and Lookalike Domains Guide · ASCII Smuggling and Invisible Prompts · Prompt Injection Guide · Brand Impersonation and Lookalike Domain Takedown Playbook