UTF-16

From Wikipedia, the free encyclopedia
  (Redirected from UTF16)
Jump to: navigation, search

UTF-16 (16-bit Unicode Transformation Format) is a character encoding capable of encoding all 1,112,064 possible characters in Unicode. The encoding is variable-length, as code points are encoded with one or two 16-bit code units. (also see Comparison of Unicode encodings for a comparison of UTF-8, -16 & -32)

UTF-16 developed from an earlier fixed-width 16-bit encoding known as UCS-2 (for 2-byte Universal Character Set) once it became clear that a fixed-width 2-byte encoding could not encode enough characters to be truly universal.

How this encoding works and why it came about[edit]

In the late 1980s work began on developing a uniform encoding for a "Universal Character Set" (UCS) that would replace earlier language-specific encodings with one coordinated system. The original idea was to expand the typical 256-character encodings requiring 1 byte per character with an encoding using 216 = 65,536 values requiring 2 bytes per character. The goal was to include all required characters from most of the world's languages, as well as symbols from technical domains such as science, mathematics, and music. Two groups worked on this in parallel, the IEEE and the Unicode Consortium, the latter representing mostly manufacturers of computing equipment. The two groups attempted to synchronize their character assignments, so that the developing encodings would be mutually compatible. The early 2-byte encoding was usually called "Unicode" or "UCS-2".

Early in this process, however, it became increasingly clear that 216 characters would not suffice, and IEEE introduced a larger 31-bit space with an encoding (UCS-4) that would require 4 bytes per character. This was resisted by the Unicode Consortium, both because 4 bytes per character wasted a lot of disk space and memory, and because some manufacturers were already heavily invested in 2-byte-per-character technology. The UTF-16 encoding scheme was developed as a compromise to resolve this impasse.

In UTF-16, characters are encoded with 2-byte values ("encoding units") in one of two ways, depending on the size of the codepoint, the numerical value representing the character. Some characters are encoded by one 2-byte encoding unit, and some by a pair of 2-byte encoding units.

In the first case, if the codepoint is less than 216 = 65,536 (so that it fits in 2 bytes), it is represented by that single 2-byte value, as in the earlier UCS-2 encoding. In this sense, UTF-16 extends UCS-2, since all the values that had already been assigned characters can be encoded this way. Characters with codepoints less than 216 are collectively known as the Basic Multilingual Plane (BMP), and include most of the high use characters in the major languages around the world.[1]

In the second case, for the new codepoints at 216 or higher, the excess over 216 is split into two halves (most significant and least significant) and the resulting two values are each encoded with a 2-byte encoding unit in a special encoding described below. Encoding units used in this way are called "surrogates" rather than codepoints; so each character above the BMP is encoded with a "surrogate pair" of two 2-byte encoding units.

To prevent confusion between the two cases, surrogates are limited to the range 0xD800..0xDFFF, since these values had not been assigned to characters earlier. Conversely, the Unicode standard now reserves these values for use as surrogates in UTF-16 encoding: they will never be assigned character values and are therefore not individually valid as codepoints. Characters from the BMP never come from this range, and characters beyond the BMP always use a pair of encoding units from this range. Thus there is no overlap between the two cases, and a scanning process can readily parse the encoding units into codepoints and surrogate pairs.

There is an upper limit on the codepoint values that can be encoded by surrogates, so a significant part of the Unicode compromise was to cut off Unicode space at this maximum value (0x10FFFF). This is far less than the 231−1 addressed by UCS-4, but it is likely to suffice for some time to come: this space falls naturally into 17 planes, each of size 216 (where BMP is plane 0); and as of 2015, Unicode character assignments have not begun to exhaust even plane 1.

A codepoint between 0x10000 and 0x10FFFF above the BMP is encoded as follows. 0x10000 is subtracted from the codepoint, leaving a value between 0 and 0xFFFFF (220−1). This is split into two 10-bit halves. The high-order half is added to 0xD800 (giving an encoding unit in the range 0xD800..0xDBFF), and the low-order half is added to 0xDC00 (for an encoding unit in the range 0xDC00..0xDFFF). The first encoding unit is the high surrogate, or leading surrogate; the second the low surrogate or trailing surrogate. The high surrogate followed by the low surrogate is the UTF-16 encoding of the codepoint.

Since the high surrogate and low surrogate come from different ranges, it is always easy to tell which is which. So even in a sequence of surrogate-encoded characters, it is easy to parse the surrogates into pairs, as the high surrogate always comes first.

Technical issues relating to the encoding scheme and comparison with UCS-2[edit]

The older UCS-2 (2-byte Universal Character Set) is a similar character encoding that was superseded by UTF-16 in version 2.0 of the Unicode standard in July 1996.[2] UCS-2 produces a fixed-length format by simply using the code point as the 16-bit code unit. UTF-16 expands the code space significantly by using surrogate pairs to encode code points above 0xFFFF, and produces the same result as UCS-2 for all code points in the range 0-0xFFFF that had been or ever will be assigned a character.

UTF-16 is specified in the latest versions of both the international standard ISO/IEC 10646 and the Unicode Standard, as well as in RFC 2781 published in 2000 by the IETF.[3][4][5]

Code points U+0000 to U+D7FF and U+E000 to U+FFFF[edit]

Both UTF-16 and UCS-2 encode code points in this range as single 16-bit code units that are numerically equal to the corresponding code points. The code points in the BMP are the only code points that can be represented in UCS-2. Within this plane, code points U+D800 to U+DFFF (see below) were never assigned character values in UCS-2, and in UTF-16 are reserved for high and low surrogates used to encode codepoint values greater than U+FFFF.

Code points U+010000 to U+10FFFF[edit]

Code points from the other planes (called Supplementary Planes) are encoded in as two 16-bit code units called surrogate pairs, by the following scheme:

UTF-16 decoder
High \ Low DC00 DC01    …    DFFF
D800 010000 010001 0103FF
D801 010400 010401 0107FF
  ⋮
DBFF 10FC00 10FC01 10FFFF
  • 0x010000 is subtracted from the code point, leaving a 20-bit number in the range 0..0x0FFFFF.
  • The top ten bits (a number in the range 0..0x03FF) are added to 0xD800 to give the first 16-bit code unit or high surrogate, which will be in the range 0xD800..0xDBFF.
  • The low ten bits (also in the range 0..0x03FF) are added to 0xDC00 to give the second 16-bit code unit or low surrogate, which will be in the range 0xDC00..0xDFFF.

(High and low surrogates are also known as "leading" and "trailing" surrogates, respectively, analogous to the leading and trailing bytes of UTF-8.[6] Note that "high" surrogates have lower code-point numbers than "low" surrogates.)

Since the ranges for the high surrogates, low surrogates, and valid BMP characters are disjoint, searches are simplified: it is not possible for part of one character to match a different part of another character. It also means that UTF-16 is self-synchronizing on 16-bit words: whether a code unit starts a character can be determined without examining earlier code units. UTF-8 shares these advantages, but many earlier multi-byte encoding schemes did not allow unambiguous searching and could only be synchronized by re-parsing from the start of the string. UTF-16 is not self-synchronizing if one byte is lost or if traversal starts at a random byte.

Because the most commonly used characters are all in the Basic Multilingual Plane, handling of surrogate pairs is often not thoroughly tested. This leads to persistent bugs and potential security holes, even in popular and well-reviewed application software (e.g. CVE-2008-2938, CVE-2012-2135).[7]

Code points U+D800 to U+DFFF[edit]

The Unicode standard permanently reserves these code point values for UTF-16 encoding of the high and low surrogates, and they will never be assigned a character, so there should be no reason to encode them. The official Unicode standard says that no UTF forms, including UTF-16, can encode these code points.

However UCS-2, UTF-8, and UTF-32 can encode these code points in trivial and obvious ways, and large amounts of software does so even though the standard states that such arrangements should be treated as encoding errors. It is possible to unambiguously encode them in UTF-16 by using a code unit equal to the code point, as long as no sequence of two code units can be interpreted as a legal surrogate pair (that is, as long as a high surrogate is never followed by a low surrogate). The majority of UTF-16 encoder and decoder implementations translate between encodings as though this were the case.[citation needed]

Byte order encoding schemes[edit]

UTF-16 and UCS-2 produce a sequence of 16-bit code units. Each unit thus takes two 8-bit bytes, and the order of the bytes may depend on the endianness (byte order) of the computer architecture.

To assist in recognizing the byte order of code units, UTF-16 allows a Byte Order Mark (BOM), a code unit with the value U+FEFF, to precede the first actual coded value.[8] (U+FEFF is the invisible zero-width non-breaking space/ZWNBSP character.)[9] If the endian architecture of the decoder matches that of the encoder, the decoder detects the 0xFEFF value, but an opposite-endian decoder interprets the BOM as the non-character value U+FFFE reserved for this purpose. This incorrect result provides a hint to perform byte-swapping for the remaining values. If the BOM is missing, RFC 2781 says that big-endian encoding should be assumed. (In practice, due to Windows using little-endian order by default, many applications also assume little-endian encoding by default.) If there is no BOM, one method of recognizing a UTF-16 encoding is searching for the space character (U+0020) which is very common in texts in most languages.

The standard also allows the byte order to be stated explicitly by specifying UTF-16BE or UTF-16LE as the encoding type. When the byte order is specified explicitly this way, a BOM is specifically not supposed to be prepended to the text, and a U+FEFF at the beginning should be handled as a ZWNBSP character. Many applications ignore the BOM code at the start of any Unicode encoding. Web browsers often use a BOM as a hint in determining the character encoding.[10]

For Internet protocols, IANA has approved "UTF-16", "UTF-16BE", and "UTF-16LE" as the names for these encodings. (The names are case insensitive.) The aliases UTF_16 or UTF16 may be meaningful in some programming languages or software applications, but they are not standard names in Internet protocols.

UCS-2 encoding is defined to be big-endian only. In practice most software defaults to little-endian,[citation needed] and handles a leading BOM to define the byte order just as in UTF-16. Although the similar designations UCS-2BE and UCS-2LE imitate the UTF-16 labels, they do not represent official encoding schemes.

Use in major operating systems and environments[edit]

UTF-16 is used for text in the OS API in Microsoft Windows 2000/XP/2003/Vista/7/8/CE.[11] Older Windows NT systems (prior to Windows 2000) only support UCS-2.[12] In Windows XP, no code point above U+FFFF is included in any font delivered with Windows for European languages.[13][14] Files and network data tend to be a mix of UTF-16, UTF-8, and legacy byte encodings.

IBM iSeries systems designate code page CCSID 13488 for UCS-2 character encoding, CCSID 1200 for UTF-16 encoding, and CCSID 1208 for UTF-8 encoding.[15]

UTF-16 is used by the Qualcomm BREW operating systems; the .NET environments; and the Qt cross-platform graphical widget toolkit.

Symbian OS used in Nokia S60 handsets and Sony Ericsson UIQ handsets uses UCS-2.

The Joliet file system, used in CD-ROM media, encodes file names using UCS-2BE (up to sixty-four Unicode characters per file name).

The Python language environment officially only uses UCS-2 internally since version 2.0, but the UTF-8 decoder to "Unicode" produces correct UTF-16. Since Python 2.2, "wide" builds of Unicode are supported which use UTF-32 instead;[16] these are primarily used on Linux. Python 3.3 no longer ever uses UTF-16, instead strings are stored in one of ASCII/Latin-1, UCS-2, or UTF-32, depending on which code points are in the string, with a UTF-8 version also included so that repeated conversions to UTF-8 are fast.[17]

Java originally used UCS-2, and added UTF-16 supplementary character support in J2SE 5.0.

In many languages quoted strings need a new syntax for quoting non-BMP characters, as the "\uXXXX" syntax explicitly limits itself to 4 hex digits. The most common (used by C#, D and several other languages) is to use an upper-case 'U' with 8 hex digits such as "\U0001D11E"[18] In Java 7 regular expressions and ICU and Perl, the syntax "\x{1D11E}" must be used. In many other cases (such as Java outside of regular expressions)[19] the only way to get non-BMP characters is to enter the surrogate halves individually, for example: "\uD834\uDD1E" for U+1D11E.

These implementations all return the number of 16-bit code units rather than the number of Unicode code points when the equivalent of strlen() is used on their strings, and indexing into a string returns the indexed code unit, not the indexed code point,[20][21][22] this leads some people to claim that UTF-16 is not supported. However the term "character" is defined and used in multiple ways within the Unicode terminology,[23] so an unambiguous count is not possible and there is no reason for strlen to attempt to return any such value. Most of the confusion is due to obsolete ASCII-era documentation using the term "character" when a fixed-size "byte" or "octet" was intended.[citation needed]

Examples[edit]

Consider the encoding of U+10437 (𐐷):

  • Subtract 0x10000 from 0x10437. The result is 0x00437, 0000 0000 0100 0011 0111.
  • Split this into the high 10-bit value and the low 10-bit value: 0000000001 and 0000110111.
  • Add 0xD800 to the high value to form the high surrogate: 0xD800 + 0x0001 = 0xD801.
  • Add 0xDC00 to the low value to form the low surrogate: 0xDC00 + 0x0037 = 0xDC37.

The following table summarizes this conversion, as well as others. The colors indicate how bits from the code point are distributed among the UTF-16 bytes. Additional bits added by the UTF-16 encoding process are shown in black.

Character Binary code point Binary UTF-16 UTF-16 hex
code units
UTF-16BE
hex bytes
UTF-16LE
hex bytes
$ U+0024 0000 0000 0010 0100 0000 0000 0010 0100 0024 00 24 24 00
U+20AC 0010 0000 1010 1100 0010 0000 1010 1100 20AC 20 AC AC 20
𐐷 U+10437 0001 0000 0100 0011 0111 1101 1000 0000 0001 1101 1100 0011 0111 D801 DC37 D8 01 DC 37 01 D8 37 DC
𤭢 U+24B62 0010 0100 1011 0110 0010 1101 1000 0101 0010 1101 1111 0110 0010 D852 DF62 D8 52 DF 62 52 D8 62 DF

Example code[edit]

C functions to convert Unicode code points from/to UTF-16 streams, assuming there are read and write functions that handle the 16-bit code units. The decoder requires the ability to "push back" a 16-bit code unit so the next read gets it (this is a popular but not the only method of dealing with unpaired surrogates):

void write_utf16(unsigned code_point)
 {
   if (code_point < 0x10000) {
     write_unsigned_short(code_point);
  } else if (code_point <= 0x10FFFF) {
    write_unsigned_short((code_point >> 10) + 0xD7C0); //Equiv. subtracting 0x010000
    write_unsigned_short((code_point & 0x3FF) + 0xDC00);
  } else {
    error("invalid code_point");
  }
}
 
unsigned read_code_point_from_utf16()
{
  unsigned code_unit = read_unsigned_short();
  if (code_unit >= 0xD800 && code_unit <= 0xDBFF) {
    unsigned code_unit_2 = read_unsigned_short();
    if (code_unit_2 >= 0xDC00 && code_unit_2 <= 0xDFFF)
      return (code_unit << 10) + code_unit_2 - 0x35FDC00;
    push_back(code_unit_2);
  }
  return code_unit;
}

See also[edit]

References[edit]

  1. ^ "Unicode FAQ". Unicode Consortium. Retrieved 16 Jan 2015. 
  2. ^ "Questions about encoding forms". Retrieved 12 November 2010. 
  3. ^ ISO/IEC 10646:2014 "Information technology — Universal Coded Character Set (UCS)" sections 9 and 10.
  4. ^ The Unicode Standard version 7.0 (2014) section 2.5.
  5. ^ RFC 2781, February 2000.
  6. ^ Allen, Julie D.; Anderson, Deborah; Becker, Joe et al., eds. (2014). "3.8 Surrogates". The Unicode Standard, Version 7.0—Core Specification. Mountain View: The Unicode Consortium. p. 118. Retrieved 3 November 2014. 
  7. ^ https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2008-2938 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2012-2135
  8. ^ UTF-8 encoding produces byte values strictly less than 0xFE, so either byte in the BOM sequence also identifies the encoding as UTF-16.
  9. ^ Use of U+FEFF as the character ZWNBSP instead of as a BOM has been deprecated in favor of U+2060 (WORD JOINER); see Byte Order Mark (BOM) FAQ at unicode.org. But if an application interprets an initial BOM as a character, the ZWNBSP character is invisible, so the impact is minimal.
  10. ^ goto.glocalnet.net Encoding test pages (load these files and check the encoding that the browser selects)
  11. ^ Unicode (Windows). Retrieved 08 March 2011 "These functions use UTF-16 (wide character) encoding (...) used for native Unicode encoding on Windows operating systems."
  12. ^ "Description of storing UTF-8 data in SQL Server". microsoft.com. December 7, 2005. Retrieved 2008-02-01. 
  13. ^ "Unicode". microsoft.com. Retrieved 2009-07-20. 
  14. ^ "Surrogates and Supplementary Characters". microsoft.com. Retrieved 2009-07-20. 
  15. ^ "Character conversion". IBM. Retrieved 2012-05-22. 
  16. ^ PEP 261
  17. ^ PEP 393
  18. ^ ECMA-334, section 9.4.1
  19. ^ Java Language Specification, Third Edition, section 3.3
  20. ^ Austin, Calvin (May 2004). "J2SE 5.0 in a Nutshell". Retrieved 2008-06-17. Supplementary Character Support ,
  21. ^ "Javadoc for java.lang.String.charAt(int)". 
  22. ^ "C Sharp Language Specification". microsoft.com. Retrieved 2009-07-08. 
  23. ^ http://unicode.org/glossary/#C

External links[edit]