This document provides a number of sets of confusables mappings for the Bengali script. This data is intended to be used in String Similarity Evaluation [SSE]. It is used together with an SSE-Common file [SSE-Common]. Generally, if it is possible to convert a given label into another label by applying the mappings listed in the SSE-Common file to one or more code points or sequences, then the two labels are considered potentially confusable.
In String Similarity Evaluation, the first step will be to discover whether two labels can be mapped into each other using the mappings in SSE-Common, which are a superset of the mappings from all SSE Data files. In a second step, a visual distance metric is determined based on similarity category values such as those specified in this document. The higher each category is, the more it contributes to the distance. Above some threshold to be determined, the two labels would be considered too distant to be confusable.
The current file is provided as a human readable summary of confusables from the perspective of the Bengali script, but is intended for documentation. All calculations for SSE should match those performed with the SSE-Common file.
This file presents an overview of all the code points and sequences in and outside the Bengali script that are deemed confusable with at least one other code point or sequence in the script, as well as those that have been defined as variants in the Root Zone LGR [RZ-LGR].
In the HTML format, this summary is contained in the “Repertoire” table. For each listed item in the table, there are one or more links to applicable Confusables Sets.
The data that define the mapping between code points and sequences, including the degree of their similarity or confusability is collected in Confusables Sets which are symmetric and transitive. They also include all variant mappings defined in the [RZ-LGR].
Each Confusables Set is presented in the same format as a “Variant Set” in the RZ-LGR, even though only some of the mappings are actually defined as variant mappings. Those are marked appropriately. The reason confusables and variants must be reviewed and processed together is that two labels may be converted into each other by applying a mix of confusable and variant mappings. Note that labels that only differ by the application of variant mappings alone would have already been identified using the RZ-LGR.
Each mapping has a string similarity category value listed in the “Similarity Category” column. The category shows the degree of similarity, with 1 being (practically or fully) identical, and higher values indicating lesser degrees of confusability. Unless otherwise categorized, the relation between two repertoire elements is considered “Distinct” (5) and is not explicitly listed.
Note: category 1 is not limited to variants; any other pair that appears (practically or fully) identical may also be given a similarity category of 1, but then with a different Reason code in the “Comment” column.
Each mapping has an associated “Reason” code that is a highlighted part of the Comment column.
| Reason Code | Explanation |
|---|---|
| Capitalization | The similarity is between two uppercase forms or between an uppercase and unrelated lowercase form. While IDNs may not contain capital letters, they may be presented to the user in uppercase, leading to confusion. |
| Confusing | A majority of users familiar with the script of either target, source or both will confuse the pair unless viewed side-by-side. |
| Fallback | One of the code points is a common fallback representation for the other. |
| Font | The confusability depends on font choice. |
| Handwriting | The confusability is caused or strengthened by users being familiar with a shape, even though it is normally only used in handwriting. |
| Other | None of the other reason codes applies. For example, this code is used when the category is homograph (1) for a mapping that exceptionally has not been defined as a variant in the RZ-LGR. Or, when the category is 4 for a mapping that is not confusing but is still used in the screening calculation. NOTE: This code MUST be accompanied by a comment explaining the nature of the confusability and reasoning for choosing the assigned category value. |
| Underlining | The presence of underlining makes these confusable. This is typically the case when the distinction between two characters rests primarily on a mark below or a descender that might be obscured when underlined. |
| Variant | The mapping is defined as a variant in the [RZ-LGR]. NOTE: the category value is set to 1 by definition. |
| Special values | |
| Not_confusing | Mappings that are marked with Reason code “not_confusing” are listed here for reference only. They will be ignored in screening or label distance calculations. |
| Injected confusable mapping | This Reason code is automatically generated when a confusable mapping was in injected from [SSE-Common]. | Imposed | This Reason code is automatically generated when a mapping has to be imposed to make a confusables set transitive after Variants have been injected. NOTE:All imposed mappings are assigned category 4 by default as they are not based on expert input. All mappings that require a different distance category have been added explicitly. |
A mapping with Reason code Not_confusing is included here primarily for reference and to document that it was considered by the expert team but explicitly resolved as not confusing despite some weak similarity. These mappings are flagged with a symbolic context “excluded-similarity” in the “Required Context” column. This context ensures that, while such mappings are documented, they are ignored in processing and do not interact with other mappings that are not so marked. (Note: For some scripts, the listing of not_confusing data may be suppressed for reasons of file size. This has no effect on any processing performed with the LGR.)
All variant mappings injected from the RZ-LGR will have a Reason code of Variant and a Similarity Category value of 1, which reflects their strong confusability.
In some cases, transitive confusables sets with the "excluded-similarity" context include mappings based on variant pairs. Despite that context, the Reason code remains Variant and the Similarity Category remains 1.
Code points added as targets of injected variants may have the term “Variant” in their comment. After variants have been injected, the mixed confusables/variant sets are made transitive again, which imposes additional confusables marked as "Imposed".
Context rules defined for injected RZ-LGR code points and variants are copied from the RZ-LGR, as are any other items and attributes referenced in or required by those rules. In a few instances the context rules in the RZ-LGR were reformulated equivalently, but in a way that better accommodates the injection process.
String similarity considers some sequences that are not part of the [RZ-LGR]. Where all members are code points from the [RZ-LGR], they are annotated as such. This annotation ignores any context constraints; it does not establish whether all sequences would indeed be valid for their RZ script.
This refers to the process by which cross-script or cross-repertoire confusables have been made consistent across scripts or repertoires. Mappings that are manually added during this process are identified by a reference.
When the elements of a sequence have confusable mappings of their own, the resulting mappings are concatenated into a new target sequence, and a mapping is listed. These additions are noted in the comments.
The data presented here are based on data collected by experts in the script. However, the data were acquired as a list of one-directional mappings. For presentation and later processing these have been mechanically completed into fully symmetric and transitive sets. Note that symmetric mappings have the same category value for forward and reverse directions, but transitive mappings show the value Imposed instead.
After Augmentation, and before applying transitivity, the existing variant mappings from the [RZ-LGR] have been injected with reason Variant, followed by applicable confusables mappings from [SSE-Common] identified as Injected confusable mapping.
For more details on the methodology common to all scripts and the special methodology used for Han ideographs, please see [101].
Each mapping has been given a reference indicating its status. See references [101] and up together with their comments in the Table of References below.
]]>