TextCharacterEncode



Release:

4.6  •  5.0  •  5.1  •  5.2  •  5.3  •  5.4  •  6.0  •  6.1  •  6.2  •  6.3


(new to Analytica 5.0)

TextCharacterEncode( type, text )

Converts «text» into a special encoded or unencoded form according to «type». Possible values for «type» include these:

  • For encoding or decoding URLs: 'URL', 'IRI', 'URL%', '-URL'
  • For encoding XML or HTML: 'XML', '-XML'
  • For UTF-8 encodings: 'UTF-8', 'UTF-8+', '-UTF-8'
  • For Unicode normalized forms: 'NFC', 'NFD', 'NFKC', 'NFKD'.
  • Return «text» with no change: 'None'

Note that a convention is used that «type» options starting with a minus invert the respective encoding.

Encoding text for inclusion in a URL

«Type» options 'URL', 'IRI' and 'URL%' encode data for inclusion on a URL. The option '-URL' decodes URL data.

Data is often passed in the query string portion of a URL, such as "John Doe" in the following URL:

http://acme.com/somePage?name=John+Doe

Notice that the space has been converted to a '+' before inserting it in the URL. The special characters ""!*'();:@&=+$,/?#[]% have special meaning in a URL and must be converted into text that does not involve those characters. The «type» value 'URL' encodes data according to the RFC-3986 standard.

If you ever need to pass a URL as a data item in another URL, then all its special characters need to be encoded so they aren't interpreted as part of the outer URL.

This same encoding appears in other standards as well, including in the standard for submitting form data in HTTP, and in the JSON standard, among others.

When using TextCharacterEncode('URL', text), your text should only encode the value that will be placed after an equal sign in the query, but nothing more. For example, you should write:

'http://acme.com/somePage?name=' & TextCharacterEncode( 'URL', 'John Doe' )

and not

TextCharacterEncode( 'URL', 'http://acme.com/somePage?name=John Doe' )

since in the latter case the characters ? =:/, etc. will all be encoded, which you don't want.

A problem with the 'URL' encoding is that all characters except the letters, digits, and -._~ are percent encoded, making URLs extremely non-readable, especially for non-English sites. The 'IRI' option (International Resource Identifier) preserves all but the reserved characters ("!*'();:@&=+$,/?#[]%"), which generally still works correctly for URLs.

The standard URL encoding changes space to a plus character. The 'URL%' option uses percent encoding for space (%20) instead.

The type option '-URL' converts the URL-encoded text back into the original text. It works for any of the encodings 'URL', 'IRI' and 'URL%'.

Examples

TextCharacterEncode('URL', '(1+2) = 3') → "%281%2B2%29+%3D+3"
TextCharacterEncode('URL%', '(1+2) = 3') → "%281%2B2%29%20%3D%203"
TextCharacterEncode('-URL','%281%2B2%29+%3D+3') → "(1+2) = 3"
TextCharacterEncode('-URL','%281%2B2%29%20%3D%203') → "(1+2) = 3"
TextCharacterEncode('URL', 'test@中文.com') → "test%40%E4%B8%AD%E6%96%87.com"
TextCharacterEncode('IRI', 'test@中文.com') → "test%40中文.com"
Variable email := "John_Doe@yahoo.com"
Variable website := "http://acme.com?name=johnDoe&type=student"
Variable cityToFind = "San Francisco, CA"
Variable UrlToRead := "http://dataSource.com/query?email=" & TextCharacterEncode( 'URL', email ) & "&site=" & TextCharacterEncode('URL', website) & "&city=" & TextCharacterEncode('URL', cityToFind)
UrlToRead"http://dataSource.com/query?email=John_Doe%40yahoo.com&site=http%3A%2F%2Facme.com%3Fname%3DjohnDoe%26type%3Dstudent&city=San+Francisco%2C+CA"

Encoding text in XML or HTML

The option 'XML' for «type» encodes data for insertion in XML or HTML. Without this encoding, the XML or HTML parser will attempt to interpret special characters such as '<', '>', '&', quotes. Also, a few characters falling in control ranges (below ascii 32 or between ascii 128 and 159) will be automatically converted to entities as required be the standards.

The '-XML' does the inverse decoding.


Examples

TextCharacterEncode( 'XML', 'One < Two, Three & Four are "Bigger"' )
"One &lt; Two, &lt;b&gt;Three &amp; Four&lt;/b&gt; are &quot;Bigger&quot;"
TextCharacterEncode('-XML', One &lt; Two, &lt;b&gt;Three &amp; Four&lt;/b&gt; are &quot;Bigger&quot;' )
'One < Two, Three & Four are "Bigger"'

UTF-8 encoding

Set «type» to 'UTF-8' to obtain the UTF-8 encoding, or to '-UTF-8' to decode a UTF-8 encoding into the Unicode characters.

The option 'UTF-8+' prepends the UTF-8 Byte Order Mark (BOM). The '-UTF-8' decoding option always removes the BOM if it is present.

Examples

TextCharacterEncode('UTF-8', '확률 분포') → "확률 분포Œ"
Asc(SplitText(TextCharacterEncode('UTF-8', '확률 분포')))
[0xed, 0x99, 0x95, 0xeb, 0xa5, 0xa0, 0x20, 0xeb, 0xb6, 0x84, 0xed, 0x8f, 0xac]
TextCharacterEncode('-UTF-8', '확률 분포Œ') → "확률 분포"
TextCharacterEncode('UTF-8+', '확률 분포') → "ï»¿í™•ë¥ ë¶„í¬Œ" { '' is the BOM }

Unicode normalization

The options 'NFC', 'NFD', 'NFKC', and 'NFKD' for «type» convert text into canonical Unicode normalized forms.

The Unicode standard includes special combining characters that allow ' composite character to be constructed from one or more combining characters applied to a main character. As a result, there are often multiple ways to encode the same visible glyph, which is often the case with accented characters. For example, the accented 'a' character 'á' can be obtained with either of the following:

Chr(225) → 'á'
'a' & Chr(0x301) → 'á'

Although these display the same, the first has a text length of 1, the second a text length of 2. The character Chr(0x301) is the acute accent combining character, and can be applied to any character. In fact, it is even possible to apply multiple combining characters to the same glyph.

When you want to ensure that combined characters are used (such as the one character 'á'), set «type» to 'NFC', which stands for Normalized Form Combined. If the two character sequence 'a' & Chr(0x301) appears in «text», it will be replaced with the single pre-composed character Chr(225).

When you want to ensure that composite characters are split into individual combining character constituents, set «type» to 'NFD', which stands for Normal Form Decomposed. Hence, for example, the single character 'á' will be replaced with the two character sequence :'a' & Chr(0x301).

Unicode also includes digraph ligature characters], such as the ligature 'fi', which is a single character glyph that contains both 'f' and 'i'. The «type» options 'NFKC' and 'NFKD' expand ligatures and digraphs into their individual characters. None of the four canonical Unicode encodings re-combine character sequences into pre-composed ligatures.

Note: The four Unicode normalization encodings required Windows Vista or later. When running on XP, «text» is returned unchanged.

Examples

TextLength(TextCharacterEncode( ['NFC', 'NFD'], 'á' )) → [1,2]
TextLength(TextCharacterEncode( ['NFC', 'NFD'], 'může' )) & rarr; [4,6]
Chr(0xfb01) → 'fi'
TextLength(TextCharacterEncode( ['None', 'NFC','NFD','NFKC','NFKD'], 'fix your résumé' )) → [14,14,16,15,17]

In the last example, note that the first character in «text» is the pre-composed ligature 'fi'. The 'NFKC' and 'NFKD' cases expanded this ligature into two characters.

See Also

Comments


You are not allowed to post comments.