View source for Model File Character Encoding

[[Category: Models]]
[[Category: Encoding functions]]

__TOC__


[[Analytica 4.5]] can read model files use either ASCII (aka ANSI, ISO-8859-1) or UTF-8 encodings. [[Analytica 4.4]] and earlier only only recognize ASCII. This page explains these encodings, why it matters, and how to convert between them.

==ASCII Encoding ==

When a file is stored with an ASCII encoding, one byte is used to store each character. This means that only 255 distinct characters can appear, namely <code>[[Chr]](1)..[[Chr]](255)</code>. Analytica's ASCII files use a slightly modified version of the ISO-8859-1 code page standard. Due to an oddity dating back to the early days of the Macintosh (where Analytica originated), Analytica's mapping differs from the ISO-8859-1 standard on three extended characters: [[Chr]](173), [[Chr]](178), [[Chr]](179), which in ISO-8859-1 are &#173;, &#178; and &#179; but in Analytica's ASCII format are &ne;, &le; and &ge;.

== UTF-8 Encoding ==

In the UTF-8 encoding, a single character may take up 1, 2 or 3 bytes. Each of these byte sequences map to a Unicode character having a character code between 1 and 65535. So whereas the ASCII encoding can only represent 255 distinct characters, UTF-8 can represent 65535 characters.  This large Unicode character set includes characters from eastern Asian languages (e.g., Chinese, Japanese, Korean), Arabic, Cyrillic (Russian), Greek, Scandinavian languages, etc. It also contains numerous mathematical and scientific symbols, currency symbols, cute graphical symbols, and combining characters (accents and other marks that can be combined with any other character to form compound glyphs). 

The most common characters used in English have character codes between 1 and 127, and these use a single byte in UTF-8 and are the same as in the ASCII mapping. Therefore, when a file contains only these characters, the sequence of bytes that make up the file are the same whether it is saved in ASCII or UTF-8.

The characters from [[Chr]](128) to [[Chr]](2047) are represented by two bytes, and [[Chr]](2048) to [[Chr]](65535) use three bytes. If you load a UTF-8 file into a program that interprets it as ASCII, there will show up as two or three printable but unusual characters.  From example, the character "£" appears as "Â£". 

== Character Code Differences ==

UTF-8 maps byte sequences to Unicode characters, and Unicode has its own standard for mapping numeric codes to characters. The standard agrees with ISO-8859-1 an all characters in the ranges [[Chr]](1)..[[Chr]](127) and [[Chr]](161) to [[Chr]](255), but has a different set of characters in the range [[Chr]](128) to [[Chr]](160).  In ASCII these are the characters <code>"€‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ"</code>, and in Unicode they are mostly non-printing control characters. Analytica 4.5 now uses the Unicode standard for [[Chr]](173), [[Chr]](178) and [[Chr]](179), so these three also differ between ASCII and UTF-8 model file encodings.

== Identifying Encoding ==

To mark a file as UTF-8, Analytica 4.5 records the encoding on the first line of the model file, for example
:<code>{ From user Lonnie, Model Bond_finance_model at 15-Mar-2013 1:09:29 AM, encoding="UTF-8" }</code>

A [http://en.wikipedia.org/wiki/Byte_order_mark byte-order mark (BOM)] is optional, and is another way to mark it as UTF-8, but Analytica 4.4 doesn't recognize BOMs and will report errors about unrecognized characters if you try loading into 4.4.

== Loading Analytica 4.4 models into Analytica 4.5 ==

Analytica 4.4 models are always saved in ASCII format, and since Analytica 4.5 can read ASCII encodings just fine, you have no character encoding issues to worry about when loading Analytica 4.4 models into 4.5.  For the character encoding differences, it automatically remaps them to the equivalent Unicode character, preserving the characters in the original model.  In other words, Analytica 4.5's handling of character encodings is fully backward compatible, so you don't have to worry about it.

== Loading Analytica 4.5 models into Analytica 4.4 ==

If you save a model from Analytica 4.5, and then load it into Analytica 4.4, in some cases character encoding issues can cause problems; however, these encoding problems only occur if you use extended Unicode characters (those characters that have no equivalent in the ASCII character set) in you 4.5 model.

When Analytica 4.5 saves your model, it determines whether any non-ASCII characters are present. It there are any (even just one), then it saves the file in UTF-8 format. If only ASCII characters are present, then it saves it in the same ASCII encoding used by earlier releases of Analytica. 

If you attempt to load a model saved in UTF-8 by Analytica 4.5 into Analytica 4.4, you'll be presented with a warning, telling you that there may be issues with extended characters. Because a UTF-8 sequence is a printable sequence in ASCII, the net impact on your model might be small.  Some models contain "«null»" in certain places, and these become corrupted to "Â«nullÂ»", which can cause errors to appear.

== Encoding conversion ==

If you need to convert a UTF-8-encoded model file into ASCII, read this section.  This need might occur if you've used some Unicode characters, but now need to go back to using it in Analytica 4.4.

''Warning: The conversion from UTF-8 to ASCII loses information. Since extended Unicode characters don't exist in ASCII, those extended characters are replaced by the '?' character.''

To perform the conversion, you first need in install [http://notepad-plus-plus.org/ NodePad++], a free text editor (not associated with [http://lumina.com Lumina Decision Systems]). The steps are
# Load the model file into NotePad++
# On the '''Encoding''' menu, select '''Encode in UTF-8 without BOM'''.  Skip this step if UTF-8 is already selected.
# On the '''Encoding''' menu, select '''Encode in ANSI.
# On the first line of the file, change <code>encoding="UTF-8"</code> to <code>encoding="ascii"</code>
# Go to the end of the file and look for a warning message after the final Close statement, which might look like
#:<pre style="background:white; border:white; margin-left: 1em; font-style:italic">{!-40499|MsgBox("This file is encoded in UTF-8 format and contains non-ascii characters. To properly read UTF-8 format, you need to upgrade to Analytica 4.5 or later. If your continue to use this model in this release of Analytica, unicode and extended ascii characters will become distorted and may cause parts of the model to break.",16,"Analytica 4.5 required")}</pre>
#:If you see this, remove it.
# Select '''File &rarr; Save''' (or '''Save As...''')
# Exit NotePad++

One final note. You need to do this conversion on the UTF-8 file saved from Analytica 4.5 (not a copy saved from Analytica 4.4).

== See Also ==
* [[Model file formats]]
* [[Chr]]
* [[Asc]]
* [[WriteTextFile]]
* [[Text functions]]
* [http://en.wikipedia.org/wiki/UTF-8 UTF-8 character encoding] on Wikipedia.
* [http://en.wikipedia.org/wiki/ISO-8859-1 ISO/IEC-8859-1] on Wikipedia.
* Functions [[Chr]] and [[Asc]], which map between characters and numeric character codes.