Collation Order

What is Collation Order?

Different languages around the world use different schemes for sorting or ordering textual values. It is not uncommon for computer programs to order text values based on the ascii values of each successive character. This works pretty well for English, since the 26 letters that make up the English alphabet have consecutive ascii codes, but it results in strange orderings when there are letters with accents. For example, most dictionaries would place the word "naïve" between the words "nag" and "name", but since the i with an umlat has an ascii code of 239, which comes after z which has an ascii code of 122, a straight ascii places "naïve" after "navigate".

Language-specific collation uses the rules of a particular language (e.g., Finnish, Russian, Spanish, etc) to determine the ordering of words. In almost all languages, you'll find that all variations of accented e (e.g., é, è, ë, ȇ etc.) come between d and f. There are also some orderings that change between languages. For example, in a Spanish-language collation order,"cha" would come after "cza", whereas in English the order would be reversed. In Swedish, ö > z whereas in German it is the other way around.

TextLocale

The system variable TextLocale (introduced in Analytica 4.5) holds the locale that determines collation order. This usually holds the name of the language. For example, if you want to use the "Swedish" collation order, you would set this system variable to "Swedish". You can do this by selecting Definition → System Variables → TextLocale from the Analytica menus, when no nodes are selected. The object window for the system variable appears, where you can set the definition. The value should not be surrounded in quotes.

When you set TextLocale, the value is a property of your model, not a property of your computer or your installation of Analytica. Changing the system variable does not invalidate previously computed values that compared or sorted text.

Ansi order

You also have the option of setting TextLocale to "ANSI" (or synonymously "ASCII") if you really want pure ascii-order collation or to "Regional" if you want it to use the end-user's native language (when you are sharing models between people in different countries). The "ANSI" order may be desirable if you have legacy (pre-Analytica 4.4) models containing algorithms that rely on ascii ordering. The default definition of TextLocale is "English", which gives the same order on every computer. If you set it to "Regional", you should be aware that the results of your model (those things that rely on text ordering) might not be identical for users in different countries.

Functions and operators that use TextLocale

TextLocale determines how the operators <, >, <=, and >= compare text. It also impacts how the functions SortIndex, Rank, Sort and RankCorrel order text values relative to each other.

Properties

Collation order lacks several properties you might be inclined to assume (see Unicode Collation Algorithm). I list a few of these here.

Order is not determined by the ascii values of characters -- a smaller ascii value will often come after a larger ascii value, and vise versa.

Collation order is not preserved under concatenation or substring operations in general.

x < y does not imply that xz < yz

x < y does not imply that zx < zy

xz < yz does not imply that x < y

zx < zy does not imply that x < y

This last property implies that you cannot determine the ordering in general by looking at the first N characters, even if the text differs on the first N characters. Also, there are cases where A < B and C < D, but when you concatenate these, AC > BD.

Although these properties do not hold for non-Ansi collation orders, they do hold (only) for the Ansi collation order.

In Analytica (but not in all other programs that use collation order), it does hold that exact one of x < y, x > y or x = y holds for every two text values x and y. Also, x < y or x = y if and only if x <= y, x > y or x = y if and only if x >= y, and x < y or x > y if and only if x <> y.

Case sensitivity

Analytica's comparison operators, <, <=, > and >=, compare text in a case-sensitive fashion, and so do Sort, SortIndex and Rank unless you pass their optional «caseInsensitive» parameter (FindInText, SplitText and TextReplace have a similar option). But in every collation order other than Ansi, "case-sensitive" does not mean that capital letters come before lowercase letters. Case is the last tiebreaker. Two texts are compared first by their letters ignoring case (and, in most languages, ignoring accents), then by accents, and only when they are still tied does case decide, with lowercase before uppercase. The actual algorithm doesn't literally run in three stages, and languages vary in how they treat accents, but this is the way to think about it (see Case Sensitive Collation Sort Order).

Some consequences:

'Zebra' < 'apple' is false, because z comes after a and case never enters into it. In Ansi order it is true, since every capital precedes every lowercase letter.
'a' < 'A' and 'apple' < 'Apple' are true: the letters are identical, so case decides, lowercase first.
Sorting ['Ac', 'ac', 'Ab', 'ab'] gives ab, Ab, ac, Ac. Although 'a' < 'A', that is trumped by the second letter. The Ansi order would be Ab, Ac, ab, ac.
A case-sensitive sort of many mixed-case words looks almost identical to a case-insensitive sort of the same words, which can make it appear that «caseInsensitive» has no effect. The two differ only among words that are identical ignoring case: the case-sensitive sort puts the lowercase spelling first, while the case-insensitive sort treats them as tied and keeps their original order.

For example, sorting ['Zebra', 'apple', 'zebra', 'Apple']:

Setting	Result	Why
default (TextLocale = English)	apple, Apple, zebra, Zebra	letters first; then lowercase before uppercase
`caseInsensitive: true`	apple, Apple, Zebra, zebra	letters only; ties keep their original order
TextLocale = ANSI	Apple, Zebra, apple, zebra	character codes; all capitals first

When this matters:

When you expect capitalized items to group together, for example to list proper nouns or identifiers first. Case-sensitive collation won't do that. Set TextLocale to ANSI, or sort on a key you compute (a multi-key sort whose first key is whether the text starts with a capital letter, for instance).
When a model's results depend on text order and the model is shared across countries. With TextLocale set to Regional the order follows each user's Windows language settings; the default, English, is the same everywhere.
When an algorithm compares text and assumes Ansi properties such as "x < y implies xz < yz" (see Properties above). They don't hold, and case is one of the reasons.

«caseInsensitive» only removes the final case tiebreaker; it cannot make capitals come first. Only the Ansi setting of TextLocale does that.

Equality

The equality operator (=) and not-equal operator (<>) identify text as equal (x = y) only when both contains precisely the same character sequence. In Unicode there are often multiple ways of representing the same logical character or characters. For example, an accented á can be either Chr(225), or it can be the two-character sequence a & Chr(769). "fi" can either be a two-character sequence 'f' & 'i', or the single character 'ﬁ' (Chr(64257)). The Ω character can be either Chr(937) or Chr(8486). When characters have more than one combining character, the combining characters can appear in any order. In all these cases, even though visually identical text results, they are not considered equal by Analytica's = operator because the precise sequence of characters is not identical. In all these cases, text using these characters will be adjacent when sorted, as if the text is nearly equal. When collation order alone does not distinguish between strings, Analytica breaks ties using ansi-order.

History

Introduced in Analytica 4.5.