ItInsightsWorld Security checklist: identify Unicode identifier risks through UTS #39 (18.0.0)

For teams defending applications against spoofed labels, Unicode® Technical Standard #39, Unicode Security Mechanisms, is a practical reference. The edition reviewed here is Version 18.0.0, dated 2026-08-27, Revision 34.

Document status is mixed: it carries a draft warning because materials may be updated, replaced, or superseded, and publication does not imply endorsement by the Unicode Consortium. It also notes that Unicode members and other interested parties reviewed it, the Unicode Consortium approved publication, and the document is stable. Because a Unicode Technical Standard is independent, claiming conformance to the Unicode Standard does not automatically mean conformance to any UTS.

Core documents behind the mechanisms

Readers should start with Unicode Technical Report #36, Unicode Security Considerations, and the Unicode FAQ on Security Issues. Unicode Standard Annex #31, Unicode Identifiers and Syntax, gives a base method for deciding what may be an identifier; UTS #18: Unicode Regular Expressions and UTR #23: The Unicode Character Property Model support property handling. For IDNA paths, Unicode Technical Standard #46, Unicode IDNA Compatibility Processing, is relevant alongside IDNA2003 and IDNA2008.

Conformance clauses in plain operational terms

A conformance claim should follow the relevant section exactly, or publish a precise list of deviations. The sections that need a deviation note include General Security Profile for Identifiers (3.1), IDN Security Profiles for Identifiers (3.2), and Email Security Profiles for Identifiers (3.3). Confusable detection (4) may be followed as written, or changed with exact added or removed mappings. Detection behavior for mixed scripts (5.1), Restriction-Level Detection (5.2), and Mixed-Number Detection (5.3) also needs clear descriptions. Section 2 sets the expectation for these documented differences.

The confusable detection functions should be implemented consistently with the section unless an implementation states what changed. That includes whether X and Y are single-script confusables, mixed-script confusables, or whole-script confusables, and whether X has whole-script confusables in set of scripts S.

What identifier characters are at risk

For this work, identifiers—IDs—are application strings that point to a single meaningful entity. A URL is a familiar example: a user must know whether a page belongs to a bank, a service, or a lookalike attacker. Machines can tell sequences apart, while humans may struggle if no character limits exist.

Restricting characters is the central countermeasure. It does not ban those characters from ordinary documents. The profile relies on Unicode Standard properties, including XID_Start and XID_Continue from the Unicode Character Database, toCasefold(X), and NFKC or NFKD normalizations. Data files follow UCD file format.

Programming language identifier profiles often add characters such as $ and remove others such as - because the latter may be used as minus; IDNA removes _ among others. Specific protocols can use the base method and define profiles that tighten or expand the set.

General Security Profile essentials (3.1)

Files under [idmod] give recommended restricted characters and a small list of additions beyond XID_Start and XID_Continue because these identifiers may appear in contexts broader than programming. Restricted characters are uncommon: not in modern use, specialized like liturgical, phonetic, or mathematical symbols, or used by very small communities. The initial selection is conservative, with future additions possible. Handling changes over time is discussed in Unicode Technical Report #36, Section 2.10.1.

An implementation following the General Profile must reject characters marked Identifier_Status=Restricted unless it documents allowed extras, which can be defined by Identifier_Type=Technical, explicit lists, or both. It may also narrow the Allowed set, for example by permitting only IDNA2008 characters. Candidates for additions include scripts from Table 7, Limited Use Scripts, of Unicode Standard Annex #31, though those scripts have not been prioritized for confusable or specialized use analysis. Canonical equivalence is applied when checking Allowed characters: <u, combining-diaeresis> may satisfy the profile if u and diaeresis are both Allowed or if ü itself is Allowed. For the format of the [idmod] files, see Section 7, Data Files.

Table 1: status and type mapping

Table 1 records why a code point is Restricted or Allowed. Identifier_Status and Identifier_Type are properties of code points. A code point can have multiple Restricted types, but multiple values are not assigned to strong restrictions: Not_Character, Deprecated, Default_Ignorable, or Not_NFKC. Among usage qualifiers, the distinction between Obsolete, Uncommon_Use, and Technical is not strict; Identifier_Status is the key signal. If no category applies, the default Identifier_Type should be Uncommon_Use.

Identifier_Status	Identifier_Type	Security reading
Restricted	Not_Character	Unassigned, private use, surrogate, or non-whitespace control characters.
Restricted	Deprecated	Characters with Deprecated=Yes.
Restricted	Default_Ignorable	Characters that are ignored by default in display or processing.
Restricted	Not_NFKC	Characters that cannot occur in NFKC-normalized strings.
Restricted	Not_XID	Characters that do not qualify as default Unicode identifiers, lacking XID_Continue=True.
Restricted	Exclusion	Script_Extensions contains a script from Table 4, Excluded Scripts, and no script from Table 7, Limited Use Scripts, or Table 5, Recommended Scripts, other than Common or Inherited.
Restricted	Obsolete	Characters no longer in modern use or not common in modern text.
Restricted	Technical	Specialized use, such as technical or liturgical symbols.
Restricted	Uncommon_Use	Uncommon characters, limited within non-Limited_Use scripts, or uncertain use.
Restricted	Limited_Use	Characters from limited-use scripts listed in Table 7, without a Table 5 script other than Common or Inherited.
Allowed	Inclusion	Exceptionally allowed characters, including Table 3a, Optional Characters for Medial, and Table 3b, Optional Characters for Continue, plus some IDNA2008 characters.
Allowed	Recommended	Characters from widely used everyday scripts listed in Table 5.

In Unicode 15.0, the Join_Control characters ZWJ/ZWNJ were removed from Identifier_Type=Inclusion. They therefore carry Identifier_Type=Default_Ignorable and Identifier_Status=Restricted. Retaining them in programming identifier profiles has usability and security consequences. Implementations that keep ZWJ and ZWNJ should declare a profile modification per Section 2, Conformance, and should enforce the constraints in Section 3.1.1, Joining Controls.

Allowed characters are usually applied as filters rather than used as-is. If C is the set supported by an identifier syntax, the effective set can be C′ = (C intersect Allowed) union Exception. Exception characters are implementation-specific, such as adding a property character with XID_Continue=False. Some implementations may also include CLDR exemplar characters for supported languages. For characters outside C′ but inside C, deployments may use appeal processes for registrations or warning messages in lookup interfaces. U+00B7 · MIDDLE DOT is recommended as an Inclusion candidate because Catalan requires it; it is an example of a non-letter symbol used inside words.

Joining controls: why blanket exclusion is not enough (3.1.1)

Some excluded characters create visible distinctions, especially Join_Control characters. A blanket ban can make correct visual spelling impossible in languages that depend on these formats. The profile therefore discusses limited contexts for joining controls: 3.1.1.1 describes permitted contexts, and 3.1.1.2 explains limitations.

Systems that support modern, customary usage should accept such characters when needed but reject unnecessary uses. The term modern, customary usage covers ordinary newspaper, journal, lay publications, street signs, commercial signage, common geographic names, and company names. It does not cover mathematical formulas, archaic scripts, isolated teaching examples, or liturgical use. The practical goal is to permit necessary visual distinction while blocking cases with no visible effect.

Choosing types (3.1.2)

When no stronger category applies, the default Identifier_Type should be Uncommon_Use. More data can change a character's status or type across versions, so deployments should have backward compatibility policies. Values are compared case-insensitively and ignoring hyphens and underbars.

Profile variants for IDN and email

The IDN and email profiles narrow identifier acceptance for DNS labels and mailbox domains or addresses. In practice, teams should treat 3.2, IDN Security Profiles for Identifiers, and 3.3, Email Security Profiles for Identifiers, as separate policy layers rather than copying a general developer identifier profile into email validation or domain registration checks. IDNA profiles also align with Unicode IDNA Compatibility Processing.

Confusable detection and visual spoofing (4)

Section 4 defines how to determine whether X and Y are single-script confusables, mixed-script confusables, or whole-script confusables. It also covers whether X has whole-script confusables within a set of scripts S. 4.1 covers Whole-Script Confusables, and 4.2 covers Mixed-Script Confusables. The operational point is that lookalikes are not only single-character typos; they can involve entire script substitutions or mixed scripts.

Detection mechanisms (5)

Detection is where a specification becomes an implementable control. Use these sections as a checklist:

1. 5.1 Mixed-Script Detection: identifies characters that should not appear together in a given identifier. Table 1a. Mixed Script Examples can be used to create regression tests.
2. 5.2 Restriction-Level Detection: maps a string against the profile restrictions.
3. 5.3 Mixed-Number Detection: catches unsafe combinations of number forms.
4. 5.4 Optional Detection: covers additional checks that deployments can enable without treating them as mandatory conformance behavior.

Development process and data inputs (6)

The development process tells teams what to expect in future data releases. Section 6 covers collection of confusables data in 6.1 and identifier modification data in 6.2. Data files are described in Section 7, including Table 2. Data File List.

Migration and persistent identifier data

Persistent identifier systems need version correspondence. Table 3. Version Correspondence explains how newer profile data relates to older versions, including Version 8.0 Migration and Version 7.0 Migration. Migrating Persistent Data is especially important for registrations, allowlists, blocklists, and cached confusable mappings.

Back matter that still matters for audits

Acknowledgments, References, and Modifications are part of the specification. References help reviewers verify Unicode Technical Report #36, Unicode Standard Annex #31, Unicode Technical Standard #46, and related Unicode documents. Modifications records version changes and helps security teams explain why an identifier previously accepted may now be restricted.

Security checklist for deploying Unicode identifier profiles

• Pin the profile version: record Version 18.0.0, dated 2026-08-27, Revision 34, so audits can separate old and new decisions.
• Define whether you implement the General Profile, an IDN profile, an email profile, or a programming identifier profile; each has different risk surfaces.
• Document any added or removed characters, mapping changes, or detection behavior changes instead of silently diverging.
• Test confusables as data, not guesses, using the documented mappings and detection functions.
• Run mixed-script and mixed-number regressions before allowing new registration flows.
• Review joining controls carefully when ZWJ or ZWNJ must appear in user-visible identifiers.
• Plan migration for persistent data before upgrading, especially for previously registered strings.

Common misconceptions

• Conformance to the Unicode Standard is enough. The document makes clear that a Unicode Technical Standard is independent, so UTS conformance must be claimed separately.
• The Allowed set should be pasted directly into a validator. In practice, Allowed acts as a filter over an existing syntax and may be narrowed for IDNA2008, email, or code identifiers.
• Restricted characters are bad characters in prose. The restriction targets identifier ambiguity; ordinary text can still use those characters.
• Joining controls can be banned forever. They can be necessary for modern, customary usage, so context-based limits are required.
• Confusable detection is just homoglyph lookup. Mixed scripts, whole-script confusables, and mixed numbers are separate detection problems.

FAQ

Why does this matter beyond DNS?

Because many systems accept Unicode identifiers. The same ambiguity logic can affect usernames, filenames, package names, account handles, and display names, even when the underlying syntax differs.

Can a profile allow fewer characters than Table 1 suggests?

Yes. The specification permits implementations to narrow the Allowed set, such as by restricting to IDNA2008 characters, and to document that choice.

What should a team do if it needs ZWJ or ZWNJ?

It should declare the profile modification, align with Section 2, Conformance, and implement the limited contexts described in Section 3.1.1, while testing that unnecessary uses are blocked.

How should changes be handled over time?

By reading the migration sections and compatibility guidance. Table 2. Data File List, Table 3. Version Correspondence, and the version migration notes give the path for data upgrades.

Action advice

1. Inventory identifier surfaces: login names, package names, domains, email local parts, file labels, and user-visible account names.
2. Choose one profile per surface: General Profile, IDN profile, email profile, or programming syntax, and record its allowed exceptions.
3. Add detection controls for mixed scripts, restriction levels, mixed numbers, and confusables, then verify with examples from Table 1a and Table 2.
4. Build an upgrade test suite for Version 8.0 Migration and Version 7.0 Migration cases before moving existing registrations.
5. Review policy every release, because Identifier_Status and Identifier_Type values can change as data improves.