Jump to content

Byte order mark

From Wikipedia, the free encyclopedia

This is an old revision of this page, as edited by Tom.Reding (talk | contribs) at 16:12, 9 March 2016 (→Usage: wl Word Joiner). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

The byte order mark (BOM) is a Unicode character, U+FEFF ZERO WIDTH NO-BREAK SPACE (BOM), whose appearance as a magic number at the start of a text stream can signal several things to a program consuming the text:[1]

  • What byte order, or endianness, the text stream is stored in;
  • The fact that the text stream is Unicode, to a high level of confidence;
  • Which of several Unicode encodings that text stream is encoded as.

BOM use is optional, and, if used, should appear at the start of the text stream.

Unicode can be encoded as 8-bit, 16-bit, or 32-bit integers. For the 16- and 32-bit representations, a computer receiving text from arbitrary sources needs to know which byte order the integers are encoded in. Because the BOM itself is encoded in the same scheme as the rest of the document, but has a known value, the consumer of the text can examine these first few bytes to determine the encoding. The BOM thus gives the producer of the text a way to describe the text stream's endianness to the consumer of the text without requiring some contract or metadata outside of the text stream itself.

Once the receiving computer has consumed the text stream, it is free to process the characters in its own native byte order and no longer needs the BOM. Hence the need for a BOM arises in the context of text interchange, rather than in text processing within a closed environment.

Usage

If the BOM character appears in the middle of a data stream, Unicode says it should be interpreted as a "zero-width non-breaking space" (inhibits line-breaking between word-glyphs). In Unicode 3.2, this usage is deprecated in favour of the "Word Joiner" character, U+2060.[1] This allows U+FEFF to be only used as a BOM.

UTF-8

The UTF-8 representation of the BOM is the byte sequence 0xEF,0xBB,0xBF. A text editor or web browser misinterpreting the text as ISO-8859-1 or CP1252 will display the characters  for this.

The Unicode Standard permits the BOM in UTF-8,[2] but does not require or recommend its use.[3] Byte order has no meaning in UTF-8,[4] so its only use in UTF-8 is to signal at the start that the text stream is encoded in UTF-8. The BOM may also appear when UTF-8 data is converted from other encodings that use a BOM. The standard also does not recommend removing a BOM when it is there, so that round-tripping between encodings does not lose information, and so that code that relies on it continues to work.[5][6] The IETF recommends that if a protocol either (a) always uses UTF-8, or (b) has some other way to indicate what encoding is being used, then it "SHOULD forbid use of U+FEFF as a signature."[7]

Not using a BOM allows text to be backwards-compatible with some software that is not Unicode-aware. Examples include programming languages that permit non-ASCII bytes in string literals and comments (and sometimes in variable names) but not elsewhere (such at the start of the file).

On the other hand, without a BOM, heuristic analysis is required to determine what character encoding a file is using. In general UTF-8 is easily detected and therefore analysis can efficiently distinguish whether or not a document is encoded in Unicode. Several libraries are freely available to ease the task, such as Mozilla Universal Charset Detector[8] and International Components for Unicode.[9] Besides detecting Unicode, these libraries can determine which legacy encoding a document was stored in if it is not Unicode, though the reliability might be lower.

Microsoft compilers[10] and interpreters, and many pieces of software on Microsoft Windows such as Notepad treat the BOM as a required magic number rather than use heuristics. These tools add a BOM when saving text as UTF-8, and cannot interpret UTF-8 unless the BOM is present, or the file contains only ASCII bytes.

Google Docs also adds a BOM when converting a document to a plain text file for download.

UTF-16

In UTF-16, a BOM (U+FEFF) may be placed as the first character of a file or character stream to indicate the endianness (byte order) of all the 16-bit code units of the file or stream. If an attempt is made to read this stream with the wrong endianess, the bytes will be swapped, thus delivering the character U+FFFE, which is defined by Unicode as a "non character" that should never appear in the text.

  • If the 16-bit units are represented in big-endian byte order, this BOM character will appear in the sequence of bytes as 0xFE followed by 0xFF. This sequence appears as the ISO-8859-1 characters þÿ in a text display that expects the text to be ISO-8859-1.
  • if the 16-bit units use little-endian order, the sequence of bytes will have 0xFF followed by 0xFE. This sequence appears as the ISO-8859-1 characters ÿþ in a text display that expects the text to be ISO-8859-1.

Programs expecting UTF-8 may show these or error indicators, depending on how they handle UTF-8 encoding errors. In all cases they will probably display the rest of the file as garbage (a UTF-16 text containing ASCII only will be fairly readable).

For the IANA registered charsets UTF-16BE and UTF-16LE, a byte order mark should not be used because the names of these character sets already determine the byte order. If encountered anywhere in such a text stream, U+FEFF is to be interpreted as a "zero width no-break space".

Clause D98 of conformance (section 3.10) of the Unicode standard states, "The UTF-16 encoding scheme may or may not begin with a BOM. However, when there is no BOM, and in the absence of a higher-level protocol, the byte order of the UTF-16 encoding scheme is big-endian." Whether or not a higher-level protocol is in force is open to interpretation. Files local to a computer for which the native byte ordering is little-endian, for example, might be argued to be encoded as UTF-16LE implicitly. Therefore, the presumption of big-endian is widely ignored. When those same files are accessible on the Internet, on the other hand, no such presumption can be made. Searching for 16-bit characters in the ASCII range or just the space character (U+0020) is a method of determining the UTF-16 byte order.

UTF-32

Although a BOM could be used with UTF-32, this encoding is rarely used for transmission. Otherwise the same rules as for UTF-16 are applicable.

Representations of byte order marks by encoding

This table illustrates how BOMs are represented as byte sequences and how they might appear in a text editor that is interpreting each byte as a legacy encoding (CP1252 and symbols for the C0 controls):

Encoding Representation (hexadecimal) Representation (decimal) Bytes as CP1252 characters
UTF-8[t 1] EF BB BF 239 187 191 
UTF-16 (BE) FE FF 254 255 þÿ
UTF-16 (LE) FF FE 255 254 ÿþ
UTF-32 (BE) 00 00 FE FF 0 0 254 255 ␀␀þÿ (␀ refers to the ASCII null character)
UTF-32 (LE) FF FE 00 00 255 254 0 0 ÿþ␀␀ (␀ refers to the ASCII null character)
UTF-7[t 1] 2B 2F 76 38
2B 2F 76 39
2B 2F 76 2B
2B 2F 76 2F[t 2]
2B 2F 76 38 2D[t 3] 		43 47 118 56
43 47 118 57
43 47 118 43
43 47 118 47
43 47 118 56 45 		+/v8
+/v9
+/v+
+/v/
+/v8-
UTF-1[t 1] F7 64 4C 247 100 76 ÷dL
UTF-EBCDIC[t 1] DD 73 66 73 221 115 102 115 Ýsfs
SCSU[t 1] 0E FE FF[t 4] 14 254 255 ␎þÿ (␎ represents the ASCII "shift out" character)
BOCU-1[t 1] FB EE 28 251 238 40 ûî(
GB-18030[t 1] 84 31 95 33 132 49 149 51 „1•3
  1. ^ a b c d e f g This is not literally a "byte order" mark, since the byte is also the code unit in these encodings and there is no byte order to resolve. The sequence can be used to indicate the encoding of the text which it is preceding, however.[4][11]
  2. ^ In UTF-7, the fourth byte of the BOM, before encoding as base64, is 001111xx in binary. The final two bits, xx, are not specifically part of the BOM, but contain the first two bits of the first encoded character following the BOM. All four possible byte combinations are shown in the table, as well as a fifth which is used for an empty string.
  3. ^ If no following character is encoded, 38 is used for the fourth byte and the following byte is 2D.
  4. ^ SCSU allows other encodings of U+FEFF, the shown form is the signature recommended in UTR #6.[12]

See also

References

  1. ^ a b Unicode FAQ: UTF-8, UTF-16, UTF-32 & BOM
  2. ^ "The Unicode Standard 5.0, Chapter 2:General Structure" (PDF). p. 36. Retrieved 2009-03-29. Table 2-4. The Seven Unicode Encoding Schemes
  3. ^ "The Unicode Standard 5.0, Chapter 2:General Structure" (PDF). p. 36. Retrieved 2008-11-30. Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature
  4. ^ a b "FAQ - UTF-8, UTF-16, UTF-32 & BOM: Can a UTF-8 data stream contain the BOM character (in UTF-8 form)? If yes, then can I still assume the remaining UTF-8 bytes are in big-endian order?". Retrieved 2009-01-04.
  5. ^ "Re: pre-HTML5 and the BOM from Asmus Freytag on 2012-07-13 (Unicode Mail List Archive)". Retrieved 2012-07-14.
  6. ^ Bug ID: JDK-6378911 UTF-8 decoder handling of byte-order mark has changed
  7. ^ Yergeau, Francois (2003). UTF-8, a transformation format of ISO 10646. IETF. doi:10.17487/RFC3629. RFC 3629. Retrieved May 15, 2014. {{citation}}: Unknown parameter |month= ignored (help)
  8. ^ [1]
  9. ^ http://site.icu-project.org
  10. ^ Alf P. Steinbach (2011). "Unicode part 1: Windows console i/o approaches". Retrieved 24 March 2012. However, since the C++ source code was encoded as UTF-8 without BOM (as is usual in Linux), the Visual C++ compiler erroneously assumed that the source code was encoded as Windows ANSI.
  11. ^ STD 63: UTF-8, a transformation of ISO 10646 Byte Order Mark (BOM)
  12. ^ UTR #6: Signature Byte Sequence for SCSU
Retrieved from "https://en.wikipedia.org/w/index.php?title=Byte_order_mark&oldid=709186052"