TextCharacterEncode

(new to Analytica 5.0)

Converts text to or from many common codings, including URLs, XML, UTF-i, and NFC (unicode).

TextCharacterEncode( type, text )

Converts «text» into a special encoded or unencoded form according to «type». Possible values for «type» include:

• For encoding or decoding URLs: 'URL', 'IRI', 'URL%', '-URL'
• For encoding XML or HTML: 'XML', '-XML'
• For UTF-8 encodings: 'UTF-8', 'UTF-8+', '-UTF-8'
• For Unicode normalized forms: 'NFC', 'NFD', 'NFKC', 'NFKD'.
• Hash code: 'SHA-1', 'SHA-256'
• Return «text» with no change: 'None'
• Unicode character names: 'characterName', '-characterName'

Start «type» with a minus, code>'-', to invert the encoding -- i.e. to decode the text.

Encoding text for inclusion in a URL

«Type» options 'URL', 'IRI' and 'URL%' encode data for inclusion on a URL. The option '-URL' decodes URL data.

Data is often passed in the query string portion of a URL, such as "John Doe" in the following URL:

http://acme.com/somePage?name=John+Doe

Notice that it converts the space to a '+' before inserting it in the URL. The special characters ""!*'();:@&=+$,/?#[]% each have a special meaning in a URL and so must be converted into text that does not involve those characters. The «type» value 'URL' encodes data according to the RFC-3986 standard.

If you ever need to pass a URL as a data item in another URL, you must encode all its special characters so they aren't interpreted as part of the outer URL.

This same encoding appears in other standards as well, including submitting form data in HTTP, and in for JSON.

When using TextCharacterEncode('URL', text), your text should only encode the value that will be placed after an equal sign in the query, but nothing more. For example, you should write:

'http://acme.com/somePage?name=' & TextCharacterEncode( 'URL', 'John Doe' )

not

TextCharacterEncode( 'URL', 'http://acme.com/somePage?name=John Doe' )

since in the latter case the characters ? =:/, etc. will be encoded, which you don't want.

A problem with the 'URL' encoding is that all characters except the letters, digits, and -._~ are percent encoded, making URLs very difficult to read, especially for non-English sites. The 'IRI' option (International Resource Identifier) preserves all but the reserved characters ("!*'();:@&=+$,/?#[]%"), which generally still works correctly for URLs.

The standard URL encoding changes space to a plus character. The 'URL%' option uses percent encoding for space (%20) instead.

The «type» option '-URL' converts the URL-encoded text back into the original text. It works for any of the encodings 'URL', 'IRI' and 'URL%'.

Examples

TextCharacterEncode('URL', '(1+2) = 3') → "%281%2B2%29+%3D+3"

TextCharacterEncode('URL%', '(1+2) = 3') → "%281%2B2%29%20%3D%203"

TextCharacterEncode('-URL','%281%2B2%29+%3D+3') → "(1+2) = 3"

TextCharacterEncode('-URL','%281%2B2%29%20%3D%203') → "(1+2) = 3"

TextCharacterEncode('URL', 'test@中文.com') → "test%40%E4%B8%AD%E6%96%87.com"

TextCharacterEncode('IRI', 'test@中文.com') → "test%40中文.com"

Variable email := "John_Doe@yahoo.com"

Variable website := "http://acme.com?name=johnDoe&type=student"

Variable cityToFind = "San Francisco, CA"

Variable UrlToRead := "http://dataSource.com/query?email=" & TextCharacterEncode( 'URL', email ) & "&site=" & TextCharacterEncode('URL', website) & "&city=" & TextCharacterEncode('URL', cityToFind)

UrlToRead → "http://dataSource.com/query?email=John_Doe%40yahoo.com&site=http%3A%2F%2Facme.com%3Fname%3DjohnDoe%26type%3Dstudent&city=San+Francisco%2C+CA"

Encoding text in XML or HTML

The option 'XML' for «type» encodes data for insertion in XML or HTML. Without this encoding, an XML or HTML parser will attempt to interpret special characters such as '<', '>', '&', quotes. Also, a few characters falling in control ranges (below ascii 32 or between ascii 128 and 159) will be automatically converted to entities as required be the standards.

The '-XML' does the inverse decoding.

Character names

New to Analytica 6.3

You can map an extended character to its name, or a character name to the character, for example:

TextCharacterEncode( 'Ψ', 'characterName') → 'Psi'

TextCharacterEncode( 'Psi', '-characterName' → 'Ψ'

You can use these character names when typing extended characters by typing a backslash, \, the character name, then the TAB key.

When a character has no special name assigned, TextCharacterEncode( 'Ψ', 'characterName') returns Null. Analytica reads the character names from the file "characterNames.ini" found in the Analytica installation folder.

Examples

TextCharacterEncode( 'XML', 'One < Two, Three & Four are "Bigger"' )

→ "One < Two, <b>Three & Four</b> are "Bigger""

TextCharacterEncode('-XML', One < Two, <b>Three & Four</b> are "Bigger"' )

→ 'One < Two, Three & Four are "Bigger"'

UTF-8 encoding

Set «type» to 'UTF-8' to obtain the UTF-8 encoding, or to '-UTF-8' to decode a UTF-8 encoding into the Unicode characters.

The option 'UTF-8+' prepends the UTF-8 Byte Order Mark (BOM). The '-UTF-8' decoding option always removes the BOM if it is present.

Examples

TextCharacterEncode('UTF-8', '확률 분포') → "확률 분포Œ"

Asc(SplitText(TextCharacterEncode('UTF-8', '확률 분포')))

→ [0xed, 0x99, 0x95, 0xeb, 0xa5, 0xa0, 0x20, 0xeb, 0xb6, 0x84, 0xed, 0x8f, 0xac]

TextCharacterEncode('-UTF-8', '확률 분포Œ') → "확률 분포"

TextCharacterEncode('UTF-8+', '확률 분포') → "ï»¿í™•ë¥ ë¶„í¬Œ" { '' is the BOM }

Unicode normalization

The «type» options 'NFC', 'NFD', 'NFKC', and 'NFKD' convert text into canonical Unicode normalized forms.

The Unicode standard includes special combining characters that allow ' composite character to be constructed from one or more combining characters applied to a main character. As a result, there are often multiple ways to encode the same visible glyph, which is often the case with accented characters. For example, the accented 'a' character 'á' can be obtained with either of the following:

Chr(225) → 'á'

'a' & Chr(0x301) → 'á'

Although these display the same, the first has a text length of 1, the second a text length of 2. The character Chr(0x301) is the acute accent combining character, and can be applied to any character. In fact, it is even possible to apply multiple combining characters to the same glyph.

When you want to ensure that combined characters are used (such as the one character 'á'), set «type» to 'NFC', which stands for Normalized Form Combined. If the two character sequence 'a' & Chr(0x301) appears in «text», it will be replaced with the single pre-composed character Chr(225).

When you want to ensure that composite characters are split into individual combining character constituents, set «type» to 'NFD', which stands for Normal Form Decomposed. Hence, for example, the single character 'á' will be replaced with the two character sequence :'a' & Chr(0x301).

Unicode also includes digraph ligature characters], such as the ligature 'fi', which is a single character glyph that contains both 'f' and 'i'. The «type» options 'NFKC' and 'NFKD' expand ligatures and digraphs into their individual characters. None of the four canonical Unicode encodings re-combine character sequences into pre-composed ligatures.

Note: The four Unicode normalization encodings required Windows Vista or later. When running on XP, «text» is returned unchanged.

Hash codes

New to Analytica 6.5

The 'SHA-1' hash returns a 40 hexadecimal character string and 'SHA-256' hash returns a 64 hexadecimal character string. These can be used to verify that text from two sources is the same. It is a standard used in various cryptographic contexts. 'SHA-1' is considered as not very secure since being cryptographically broken. However, it is still widely used, for example by the 2-factor authentication standard that uses authenticator apps for the second form of authentication.

TextCharacterEncode( 'SHA-1', 'Hello world') → '7b502c3a1f48c8609ae212cdfb639dee39673f5e'

TextCharacterEncode( 'SHA-256', 'Hello world') → '64ec88ca00b268e5ba1a35678a1b5316d212f4f366b2477232534a8aeca37f3c'

Examples

TextLength(TextCharacterEncode( ['NFC', 'NFD'], 'á' )) → [1,2]

TextLength(TextCharacterEncode( ['NFC', 'NFD'], 'může' )) & rarr; [4,6]

Chr(0xfb01) → 'fi'

TextLength(TextCharacterEncode( ['None', 'NFC','NFD','NFKC','NFKD'], 'fix your résumé' )) → [14,14,16,15,17]

In the last example, note that the first character in «text» is the pre-composed ligature 'fi'. The 'NFKC' and 'NFKD' cases expanded this ligature into two characters.

See Also

• ReadTextFile, WriteTextFile
• TextLocale