Merged Second Level Reference Label Generation Rules

Overview

This document specifies an integrated collection of Label Generation Rules (LGR) for the Second Level, merged from a set of contributing Second Level Reference LGRs. This Common LGR is intended for resolution of label conflicts in cases where more than one of these LGRs are in use simultaneously in a given zone.

While the individual Reference LGRs or LGRs based on them may be used in a standalone fashion in any zone, the Common LGR is intended to be used in conjunction with any combinations from the set of Second Level Reference LGR files listed below, or the LGRs derived from them. As noted below, certain kinds of modifications to the individual LGRs might require recomputing this Common LGR.

For more details on the Reference Label Generation Rules for the Second Level and their development see “Reference Label Generation Rules for the Second Level: Overview and Summary” [Level-2-Overview].

File Format and Conventions

The format of this file, as well as that of the other Reference LGRs, follows [RFC 7940], with the following conventions:

• Any item that is common to all LGRs (or a group of LGRs) is marked with a ⍟ in the comment. This ensures that the item is recognized as not specific to a single LGR and the name for it is not prefixed during integration.
• A reflexive variant of type “out-of-repertoire-var” marks an item as not part of the current repertoire, even if it is listed in the LGR as a target for cross-repertoire variants.
• A special context rule “extended-cp” is used to gate off (disable) optional members of the repertoire as described below.
• A special variant context rule, “enabled” is used to mark variants that are enabled as written, but may be disabled.

The use of code points of script “Common” or “Inherited” is not declared via the “language” elements in the metadata. Likewise, any admixture of ASCII code points is not reflected in the language declaration.

The XML version of this file is the normative specification, the HTML version is provided for ease of reading. It contains additional explanatory text and other information such as summary counts or descriptive data extracted from the Unicode Character Database.

Merged (Common) LGR

This merged LGR has been machine generated by combining the data from all of the Reference LGRs listed below plus a preamble containing this description and the list of references used in this merged LGR. The merged LGR contains the union of the repertoire, variant mappings and Whole Label Evaluation (WLE) rules of the contributing LGRs (also referred to as Element LGRs) as described in the following sections. Data from the Element LGRs that are necessarily script-dependent, such as the type for variant mappings have been removed or replaced by default values.

Excluded from the merge are certain standalone LGRs, principally those that cater to a specific language by supporting a subset of the repertoire of one of the reference LGRs listed below. As standalone LGRs and supporting a subset of the repertoire they neither contribute to the merged repertoire or variant set.

Each of the contributing Reference LGRs provides the complete specification for determining both the validity of a label in the given script, as well as whether a proposed label is an allocatable variant for a given original label. As the set of blocked labels may contain mixed-script or cross-repertoire labels, it cannot be computed from the element LGR. The same is true for Index labels computed in collision testing. Therefore, determining collisions between labels or calculation of index labels requires the use of this merged file.

Using the Common LGR in Label Processing

When a zone supports multiple LGRs for different scripts simultaneously, there is a chance of whole-script confusable labels. The most serious of these are whole-script homographs, or labels that look so close to identical, that they can be substituted for one another without a reasonably alert user noticing the substitution. This issue can be mitigated via the cross-script variants defined in the Common LGR, but in the same pass, the merged, Common LGR can also be used to handle collisions based on variants defined within any of the scripts that contributed to the merge. See the related description in Section 3.1, “How to use the Common LGR in Processing a Label” in [Level-2-Overview].

When processing an applied for label, this merged LGR presents the complete data and specification needed for conflict checking with any existing label, independent of script or language. This is in contrast to the individual Reference LGRs, each of which presents the complete data and specification to determine the validity of any label when applied for under that reference LGR as well as the verification of any proposed allocatable variant for the label. In particular, the merged LGR contains a complete set of all cross-script and cross-repertoire variants defined for the combined set of Element LGRs. This can be used to generate an index label for each label: two labels with the same index label mutually block each other, unless one is an allocatable variant of the other (which would be determined using the selected Element LGR).

For any valid label, an index label can be computed using the Common LGR, and any label for which the index label is not unique across the registry has a collision with another label in that zone. For details of that process, see Section 3.2 “Calculating an Index Label” in [Level-2-Overview].

Note that the variant listing in this file suppresses a small number of redundant sequences as discussed in Section 4.5.8 “Implicit Cross-script Variants” in [Level-2-Overview]. Omitting them does not change the index label calculation and therefore does not affect the set of labels that are mutual variants. As a consequence, only the index label calculation is guaranteed to succeed in detecting conflicts. Attempting to enumerate all “blocked” variants may not yield all such labels. See also “Large Variant Sets” below for why a full enumeration may be impractical for other reasons.

Repertoire

The repertoire of this merged LGR is the cumulative repertoire of all the Second Level Reference LGRs from the list below that have been integrated into this version. Those repertoires, in turn were developed based on [RZ-LGR-5] which is bounded by [MSR-5], which in turn is a subset of [Unicode 11.0.0]. For further details, see Section 3.2.1, “Repertoire” in [RZ-LGR-Overview].

For the second level, the repertoire has been extended beyond MSR-5 by various sets of decimal digits and the HYPHEN-MINUS (and on rare occasion by similar script-specific punctuation marks that form an integral part of words in that script).

As the repertoires are merged, any code point with a reflexive “out-of-repertoire-var” mapping in all Reference LGRs containing that code point will be considered not part of the merged repertoire and not included in the merged repertoire. This can happen if a subset of reference LGRs is merged while containing unused cross-script mappings to scripts not included. The merge process does not remove any code point from the individual reference LGRs.

Repertoire Listing: Each code point or range is tagged with the script or scripts with which the code point is used. Each cites the version of the Unicode Standard in which the code points were first encoded. All repertoire elements, including sequences of code points, cite each Reference LGR containing that repertoire element; see “References” below.

Optional Extensions: Repertoire elements that are considered optional extensions are given the context rule “extended-cp”. This rule is defined as matching the empty label; therefore, by default, all optional extensions are disabled. To enable an individual element, if required, remove the rule.

Some code points are also tagged with script- or language-specific classifications. These tags have been prefixed with the Unicode script identifier in the merged LGR.

Sequences as Part of the Repertoire: Element LGRs may define sequences as members of the repertoire. In some instances these sequences may contain code point not individually listed. These code points are implicitly part of the repertoire, even though they may only appear in the context of that sequence. Sometimes sequences are defined as out-of-repertoire targets for cross-repertoire variant mappings, even though they are otherwise redundant. Such sequences may be included in the merged repertoire, or as an “imposed” variant due to cross-script variant transitivity. (Being redundant, they do not affect the set of available labels under that LGR.)

Comments

Comments provide additional information for some of the code points, but also for the definition of variants, classes, rules and actions. Where comments were merged from multiple sources, duplicates are omitted, but non-matching comments are both shown, separated by a vertical bar. A comment marked with ⍟ indicates an item explicitly shared among two or more Element LGRs, or an item required by integration.

Variants

The variant mappings in this LGR are based on the union of the non-reflexive variant mappings from all the element LGRs that have been integrated into this merged LGR. After computing the union, the variants are mechanically augmented, if needed, to ensure transitivity of the set. Any variants added during this process are identified in comments.

Variant Dispositions: Because the disposition of variant labels, for example, as “allocatable”, is specific to each script, information related to such dispositions cannot be expressed in the script- or language-neutral context of this merged file. Instead, all merged variant mappings are labeled as “blocked” in this document as needed for conflict checking. For a discussion of this approach, see Section 3.2.2, “Variants” in [RZ-LGR-Overview].

Variants of Digits: several reference LGRs for the second level contain both ASCII and native digits as semantic variants of each other, meaning digits of the same numerical value are variants. By transitivity, this makes all digits of the same value cross-script variants of each other across all reference LGRs as long as the LGR contains the ASCII digits as well. In order to keep the individual element LGRs readable, the effect of transitivity for these variants is only expressed in this merged file. This is possible because the ASCII digits are common to all affected LGRs and it is therefore possible to merge the variant relations to any script-specific digits defined in each. For almost all other cases of cross-script variants there is no such a common subset and they are therefore listed as “out-of-repertoire-var” variants in each affected element LGR.

In addition, some of the digits also are cross-script homoglyphs, meaning that they look identical. By also making them variants of the corresponding ASCII digit (whether or not the repertoire contains that ASCII digit already) we ensure that when transitivity is applied to the merged file, they become blocked variants across the scripts, without having to overburden each LGR with out-of-repertoire mappings to multiple targets. (Conflicts arising from digits of different value that are homoglyphs of each other or digits that are homoglyphs of letters may not be fully handled in this version of the LGR, and thus may need to be addressed by policy in other steps of the application process).

Reflexive Variants: In the merged file, any reflexive variant mappings, including “out-of-repertoire-var” mappings have been removed from any repertoire element in the merged repertoire as these mappings do not contribute to the index label calculation needed for collision checking. In contrast, each element LGR retains the full details of variant mappings as needed to determine variant disposition.

Suggested Variants: Some reference LGRs may have suggested variants that are enabled as written by using a context rule named “enabled”. They can be individually disabled by changing the corresponding “when” attribute to a “not-when” attribute in the XML. (Note: this convention interferes with integration, so the desired variants should have the attribute removed, and the undesired ones should be deleted, if it is desired to integrate the affected LGR.)

Implicit Cross-script Variants: Not all element LGRs explicitly list all variants to other scripts and repertoires, choosing instead to inherit those defined in other script LGRs implicitly. Nevertheless, as long as cross-script variants are defined in at least one LGR, they will be part of this merged LGR and are considered in effect for all LGRs that contain the relevant code points or sequences. However, in all cases, each element LGR always lists all of the applicable in-script variants. See also Section 4.5.8, “Implicit Cross-script Variants” in [Level-2-Overview]. A set of “full-variant” LGRs has been introduced that collectively contains all the unlisted cross-script variants from a set of related LGRs. It is these LGRs that are used in defining the variants in this Common LGR.

Variant Label Processing: Only the merged LGR contains the full information needed for computing the index label for a label needed for collision checking. The merged LGR therefore must be used to determine conflicts (two labels blocking each other) while the element LGRs are used to determine whether a given label is an allocatable variant for a particular original label under that LGR. See also Section 5, “Using the LGR” in [RZ-LGR-Overview].

Index Label: An index variant label, of index label for short, is akin to a key in a hash table. It is the same for all members of a set of labels (or their variants) that would have a mutual conflict. Allocatable variants share the same index label, but their conflict is resolved by allowing delegation to the same applicant. The index variant label for an invalid label is undefined, but an index label itself does not have to be a valid label. In some special cases it may be necessary to define context rules and other restrictions that ensure that any two labels that are variants of each other always generate the same index label.

Large Variant Sets: Especially for longer labels, some variant definitions may lead to variant label sets that can be very large, so large, in some cases, as to not be enumerable in real time. In other cases, while the actual number of variant labels is limited, the enumeration may in the worst case require the evaluation of a large number of different ways of partitioning a label into sequences. (In addition, the elision of certain redundant sequences may make this enumeration incomplete.)

However, given any two labels, the Common LGR is designed to support very efficient determination of whether the two labels collide (block each other). Collision detection can be implemented as an O(1), or constant time algorithm that is unaffected by the number of variants defined for each. Finding out whether a label collides with any of a large number of already delegated labels can be as fast as detecting whether a particular hash code exists in a list.

The element LGRs, in turn, support determining for a given original applied-for label whether the choice of some other label is an allocatable variant or not. Unlike an attempt to generate the full set of variants, this validation of variant disposition can be optimized so that it does not depend on the total number of allocatable variants.

Allocatable Fallback Variants: A fallback variant label is a single allocatable variant label that uses substitute code points for code points or sequences not available (or not allowed) in some contexts, that would otherwise be required for a linguistically accurate rendering of some label. A fallback my not look like the intended label, but it is generally recognized as an established “poor man’s substitute” for that label.

Fallbacks are typically defined for IDNA2003 compatibility; for ASCII fallback; or for similar reasons. For examples of fallback variants, see Section 4.5.5, in “Allocatable Fallback Variants” in [Level-2-Overview].

With variants defined as “fallback”, a single “preferred” or “intended” spelling of a label plus a single fallback variant for that label may be allocated. The fallback exclusively uses the fallback characters, where defined. Any label that mixes non-fallback and fallbacks for the same characters, would be a blocked variant. If the fallback variant is the one applied for, no other variant label is allocatable.

To implement this limitation, the variant mapping from the standard letter or sequence to its fallback is of type “fallback”, while the mapping in the other direction is of type “blocked”. A third variant type “r-original” is used as a reflexive variant to identify code points that have a fallback mapping, but that appear in their non-fallback form in the original label, and thus “map to themselves”. A set of default actions is defined for use with all second level reference LGRs, see “Default Whole Label Evaluation Rules and Actions” below. These actions resolve as “allocatable” any label where all variants are of type “fallback”, and as “valid” any label where all variants are of type “r-original”. Labels with a mix of variant types are resolved as “blocked”.

A fallback mapping does not imply that the target code points or sequences are inferior. They are simply a substitute for a more linguistically desirable spelling for the intended label. However, that does not mean that they couldn’t be part of a linguistically accurate spelling of some other label. In fact, the fallback may spell some word in its own right, sometimes it even is an alternate spelling of the same word as the intended label.

Context Rules for Variants: some of the variants defined in this LGR are “effective null variants”, that is, some code points in the source map to “nothing” in the target with all other code points unchanged. (Because mappings are symmetric, it does not matter whether it is the forward or reverse mapping that maps to “null”.) Such variants require a context rule to keep the variant set well behaved. Symmetry requires the same context rule for both forward and reverse mappings. For a discussion, see Section 6.5 in [RZ-LGR-Overview].

In other cases, the sequences or code points making up source and target are constrained by explicit context rules on the code points (or by implicit context rules defined for the adjacent code points). In such a case, any variants may require context rules that match the intersection between the effective contexts for both source and target; otherwise, a sequence might be considered valid in some variant label when it would not be valid in an equivalent context in an original label. See Section 6.4.2 in [RZ-LGR-Overview].

Overlapping Variants: Some sequences may overlap, that is, they share a common part with another sequence or code point. Accordingly, in partitioning a label into code points and sequences, more than one partition is possible. In these cases, variants have to be computed for all possible partitions. In some cases, context rules on sequences or variants are defined to curtail any unwanted side effects of such multiple partitioning, such as having each partition being part of a different variant label set, or generating a different index label. For further discussion, see Section 6.6 “Overlapped Variants” in [RZ-LGR-Overview].

The specification of variants for the Second Level Reference LGRs follows the guidelines in [RFC 8228].

Character Classes

This merged LGR includes the cumulative set of character classes from all the element LGRs that have been integrated into this version of the merged LGR. See Section 3.2.3, “Character Classes” in [RZ-LGR-Overview]. The names for any script-specific character classes have been prefixed with the Unicode script identifier.

Whole Label Evaluation (WLE) and Context Rules

This merged LGR includes the cumulative set of WLE and context rules and actions from all the element LGRs that have been integrated into this version of the merged LGR. For additional discussion, see Section 3.2.4, “Whole Label Evaluation Rules (WLE)” [RZ-LGR-Overview] and Section 4.6 “Whole Label Evaluation (WLE) and Context Rules” in [Level-2-Overview]. See also the comments given for each rule or action.

Default Whole Label Evaluation Rules and Actions

Rules: This integrated LGR includes a set of common default WLE rules, and their associated actions, applicable to IDNs. They are marked with ⍟. These include the restrictions defined in [RFC 5891] on placement of combining marks and HYPHEN-MINUS, the restrictions from [RFC 5893] on digit placement in RTL script labels, as well as the default actions defined in [RFC 7940]. For details, see Section 4.6.1 “Protocol-defined Rules” in [Level-2-Overview].

Note that other restrictions for IDNs, such as being lowercase, normalized to Unicode Normalization Form NFC, or having an A-label of length 63 or less are not explicitly enforced by this or other LGRs. It is up to the application processing a label under these LGR to perform these validation steps before input, and, in the case of variant labels calculated under these LGRs, also on output.

Actions: This LGR includes the default actions from [RFC 7940] assigning dispositions to variant labels based on the variant mapping types “allocatable” and “blocked” as well as actions for the reflexive mapping identifying a code point as “out-of-repertoire-var”. In addition, actions needed to implement protocol restrictions as defined in [RFC 5891] are also included by by default, independent of whether they are triggered by any label under the given LGR. There are a set of additional actions for consistent handling fallback variants, added where appropriate to enforce the restriction that at most one fallback label may be an allocatable variant in addition to the original label. All of these actions are marked with ⍟.

Script- or language-specific WLE Rules

The Bidi Rule of [RFC 5893] is treated as a script- or language-specific rule for LGRs using a right-to-left script, as are any rules related to digit mixing.

The names for any script-specific rules have been prefixed with the Unicode script identifier in this file.

Script- or language-specific Actions

Some actions are triggered by script- or language-specific rules or variant type values. While such actions are collected here for reference, they are inoperative in the context of the merged LGR because in the merged LGR all variant type values have been mapped to “blocked”, or, in the case of reflexive variants, removed.

Methodology and Contributors

The reference LGRs for the Second Level were developed by Asmus Freytag and Michel Suignard following the [Guidelines]. An important input to this process are the Root Zone LGRs, and the output has been reviewed in a public process. For additional details on methodology and a full list of contributors, see Section 6 “Contributors” of [Level-2-Overview]

References

In the listing of the repertoire, references starting at [0] refer to Unicode Standard versions in which the corresponding code points were initially encoded. References [101] and above correspond to the script- or language-specific LGRs that include the repertoire item. Repertoire items may have more than one reference.

In addition, the following references are cited in this document or they form a background document:

[Guidelines]

Internet Corporation for Assigned Names and Numbers, (ICANN), “Guidelines for Developing Reference Label Generation Rules for the Second Level”. (Los Angeles, California: ICANN, 27 May 2020),
https://www.icann.org/en/system/files/files/lgr-guidelines-second-level-27may20-en.pdf

[Level-2-Overview]

Internet Corporation for Assigned Names and Numbers, (ICANN),“Reference Label Generation Rules (LGR) for the Second Level: Overview and Summary” (PDF), (Los Angeles, California: ICANN, 24 January 2024),
https://www.icann.org/en/system/files/files/level2-lgr-overview-summary-24jan24-en.pdf

[MSR-5]

Integration Panel, “Maximal Starting Repertoire — MSR-5 Overview and Rationale”, 24 June 2021,
https://www.icann.org/en/system/files/files/msr-5-overview-24jun21-en.pdf

[Proposal-Devanagari]

Neo-Brahmi Generation Panel, “Proposal for a Devanagari Script Root Zone Label Generation Rule-Set (LGR)”, 22 April 2019,
https://www.icann.org/en/system/files/files/proposal-devanagari-lgr-22apr19-en.pdf

[Proposal-Myanmar]

Myanmar Generation Panel, “Proposal for a Myanmar Script Root Zone Label Generation Rule-Set (LGR)”, 17 March 2022,
https://www.icann.org/en/system/files/files/Proposal-Arabic-lgr-17mar22-en.pdf

[RFC 5891]

J. Klensin, “Internationalized Domain Names in Applications (IDNA): Protocol”, RFC 5891, August 2010,
https://www.rfc-editor.org/info/rfc5891

[RFC 5893]

Alvestrand, H., Ed., and C. Karp, “Right-to-Left Scripts for Internationalized Domain Names for Applications (IDNA)”, RFC 5893, August 2010,
https://www.rfc-editor.org/info/rfc5893

[RFC 6365]

Hoffman, P. and J. Klensin, “Terminology Used in Internationalization in the IETF”, BCP 166, RFC 6365, DOI 10.17487/RFC6365, September 2011,
https://www.rfc-editor.org/info/rfc6365

[RFC 7940]

Davies, K. and A. Freytag, “Representing Label Generation Rulesets Using XML”, RFC 7940, August 2016,
https://www.rfc-editor.org/info/rfc7940

[RFC 8228]

A. Freytag, “Guidance on Designing Label Generation Rulesets (LGRs) Supporting Variant Labels”, RFC 8228, August 2017,
https://www.rfc-editor.org/info/rfc8228

[RZ-LGR-Overview]

Integration Panel, “Root Zone Label Generation Rules (RZ LGR-5): Overview and Summary”, 26 May 2022 (PDF),
https://www.icann.org/sites/default/files/lgr/rz-lgr-5-overview-26may22-en.pdf

[RZ-LGR-5]

Integration Panel, “Root Zone Label Generation Rules (RZ-LGR-5)”, 26 May 2022 (XML),
https://www.icann.org/sites/default/files/lgr/rz-lgr-5-common-26may22-en.xml
non-normative HTML presentation:
https://www.icann.org/sites/default/files/lgr/rz-lgr-5-common-26may22-en.html

[Unicode 11.0.0]

The Unicode Consortium. The Unicode Standard, Version 11.0.0, (Mountain View, CA: The Unicode Consortium, 2018. ISBN 978-1-936213-19-1)
https://www.unicode.org/versions/Unicode11.0.0/

For details on references [0] and up and [100] and up, please refer to the Table of References below, as well as to [RZ-LGR-Overview].

Reference LGRs Merged

This file is merged from the following Second Level Reference LGRs.

• Second Level Reference Label Generation Rules for the Arabic Script (und-Arab) [101]
• Second Level Reference Label Generation Rules for the Armenian Script (und-Armn) [102]
• Second Level Reference Label Generation Rules for the Bengali Script (und-Beng) [103]
• Second Level Reference Label Generation Rules for the Chinese Script (und-Hani)* [104]
• Second Level Reference Label Generation Rules for the Cyrillic Script (und-Cyrl) [105]
• Second Level Reference Label Generation Rules for the Devanagari Script (und-Deva)* [106]
• Second Level Reference Label Generation Rules for the Ethiopic Script (und-Ethi) [107]
• Second Level Reference Label Generation Rules for the Georgian Script (und-Geor) [108]
• Second Level Reference Label Generation Rules for the Greek Script (und-Grek) [109]
• Second Level Reference Label Generation Rules for the Gujarati Script (und-Gujr) [110]
• Second Level Reference Label Generation Rules for the Gurmukhi Script (und-Guru) [111]
• Second Level Reference Label Generation Rules for the Hebrew Script (und-Hebr) [112]
• Second Level Reference Label Generation Rules for the Japanese Script (und-Jpan) [113]
• Second Level Reference Label Generation Rules for the Kannada Script (und-Knda) [114]
• Second Level Reference Label Generation Rules for the Khmer Script (und-Khmr) [115]
• Second Level Reference Label Generation Rules for the Korean Script (und-Kore)* [116]
• Second Level Reference Label Generation Rules for the Lao Script (und-Laoo) [117]
• Second Level Reference Label Generation Rules for the Latin Script (und-Latn)* [118]
• Second Level Reference Label Generation Rules for the Malayalam Script (und-Mlym) [119]
• Second Level Reference Label Generation Rules for the Myanmar Script (und-Mymr)* [120]
• Second Level Reference Label Generation Rules for the Oriya Script (und-Orya) [121]
• Second Level Reference Label Generation Rules for the Sinhala Script (und-Sinh) [122]
• Second Level Reference Label Generation Rules for the Tamil Script (und-Taml)* [123]
• Second Level Reference Label Generation Rules for the Telugu Script (und-Telu)* [124]
• Second Level Reference Label Generation Rules for the Thai Script (und-Thai) [125]

(*) Where available the “full-variant” version of each LGR was used in the merge.