# Binary Ordered Compression for Unicode

(Redirected from BOCU-1)

Binary Ordered Compression for Unicode (BOCU) is a MIME compatible Unicode compression scheme. BOCU-1 combines the wide applicability of UTF-8 with the compactness of Standard Compression Scheme for Unicode (SCSU). This Unicode encoding is designed to be useful for compressing short strings, and maintains code point order. BOCU-1 is specified in a Unicode Technical Note.[1]

For comparison SCSU was adopted as standard Unicode compression scheme with a byte/code point ratio similar to language-specific code pages. SCSU has not been widely adopted, as it is not suitable for MIME “text” media types. For example, SCSU cannot be used directly in emails and similar protocols. SCSU requires a complicated encoder design for good performance. Usually, the zip, bzip2, and other industry standard algorithms compact larger amounts of Unicode text more efficiently.[2]

Both SCSU[3] and BOCU-1[4] are IANA registered charsets.

## Details

All numbers in this section are hexadecimal, and all ranges are inclusive.

Code points from `U+0000` to `U+0020` are encoded in BOCU-1 as the corresponding byte value. All other code points (that is, `U+0021` through `U+D7FF` and `U+E000` through `U+10FFFF`) are encoded as a difference between the code point and a normalized version of the most recently encoded code point that was not an ASCII space (`U+0020`). The initial state is `U+0040`. The normalization mapping is as follows:

Code range Normalized code point Notes
`U+3040` to `U+309F` `U+3070` Hiragana
`U+4E00` to `U+9FA5` `U+7711` Unihan
`U+AC00` to `U+D7A3` `U+C1D1` Hangul
`U+0020` encoder state kept as is Space
`U+hhhh00` to `U+hhhh7F`
(excluding ranges above)
`U+hhhh40` middle
of 128
`U+hhhh80` to `U+hhhhFF`
(excluding ranges above)
`U+hhhhC0` middle
of 128

The difference between the current code point and the normalized previous code point is encoded as follows:

Difference range Byte sequence range
(see below)
`-10FF9F` to `-2DD0D` `21` `F0` `58` `D9` to `21` `FF` `FF` `FF`
`-2DD0C` to `-2912` `22` `01` `01` to `24` `FF` `FF`
`-2911` to `-41` `25` `01` to `4F` `FF`
`-40` to `3F` `50` to `CF`
`40` to `2910` `D0` `01` to `FA` `FF`
`2911` to `2DD0B` `FB` `01` `01` to `FD` `FF` `FF`
`2DD0C` to `10FFBF` `FE` `01` `01` `01` to `FE` `19` `B4` `54`

Each byte range is lexicographically ordered with the following thirteen byte values excluded: `00 07 08 09 0A 0B 0C 0D 0E 0F 1A 1B 20`. For example, the byte sequence `FC 06 FF`, coding for a difference of `1156B`, is immediately followed by the byte sequence `FC 10 01`, coding for a difference of `1156C`.

Any ASCII input `U+0000` to `U+007F` excluding space `U+0020` resets the encoder to `U+0040`. Because the above-mentioned values cover line end code points `U+000D` and `U+000A` as is (`0D 0A`), the encoder is in a known state at the begin of each line. The corruption of a single byte therefore affects at most one line. For comparison, the corruption of a single byte in UTF-8 affects at most one code point, for SCSU it can affect the entire document.

BOCU-1 offers a similar robustness also for input texts without the above-mentioned values with the special reset code `0xFF`. When a decoder finds this octet it resets its state to `U+0040` as for a line end. The use of `0xFF` reset bytes is not recommended in the BOCU-1 specification, because it conflicts with other BOCU-1 design goals, notably the binary order.

The optional use of a signature `U+FEFF` at the begin of BOCU-1 encoded texts, i.e. the BOCU-1 byte sequence `FB EE 28`, changes the initial state `U+0040` to `U+FEC0`. In other words, the signature cannot simply be stripped as in most other Unicode encoding schemes. Adding a reset byte after the signature (`FB EE 28 FF`) could avoid this effect, but the BOCU-1 specification does not recommend this practice.

In theory UTF-1 and UTF-8 could encode the original UCS-4 set with 31 bits up to `7FFFFFFF`. BOCU-1 and UTF-16 can encode the modern Unicode set from `U+0000` to `U+10FFFF`. Excluding the thirteen protected code points encoded as single octets BOCU-1 can use ${\displaystyle 256-13=243}$ octets in multi-byte encodings. BOCU-1 needs at most four bytes consisting of a lead byte and one to three trail bytes. The trail bytes encode a remaining "modulo 243" (base 243) difference, the lead byte determines the number of trail bytes and an initial difference. Note that the reset byte `0xFF` is not protected and can occur as trail byte.

## Patent

The general BOCU algorithm is covered by United States Patent #6,737,994, which also mentions the specific BOCU-1 implementation.[5] IBM, which employed both of the inventors of BOCU-1 at the time it was created, states in the Unicode Technical Note that implementers of a "fully compliant version of BOCU-1" must contact IBM to request a royalty-free license.[6] BOCU-1 is the only Unicode compression scheme described on the Unicode Web site that is known to be encumbered with intellectual property restrictions.

By contrast, IBM also filed for a patent on UTF-EBCDIC, but it chose in that case to make the documentation and encoding scheme “freely available to anyone concerned towards making the transformation format as part of the UCS standards,” instead of requiring implementers to request a license.[7]

## In HTML

Supporting BOCU-1 in HTML documents is prohibited by the W3C[8][9] and WHATWG[10] HTML standards, as it would present a cross-site scripting vulnerability.[11]

## References

1. ^ Markus Scherer, Mark Davis (2006-02-04). "UTN #6: BOCU-1". Retrieved 2008-05-18.
2. ^ Ewell, Doug (2004-01-30). "UTN #14: A survey of Unicode compression" (PDF). Retrieved 2008-06-13.
3. ^ IANA registration record for SCSU
4. ^ IANA registration record for BOCU-1
5. ^ Davis; et al. (2004-05-18). "United States Patent #6,737,994, "Binary-ordered compression for unicode"". Retrieved 2008-11-16.
6. ^ Markus Scherer, Mark Davis (2006-02-04). "UTN #6: BOCU-1". Retrieved 2014-02-05.
7. ^ V.S. Umamaheswaran (2002-04-16). "UTR #16: UTF-EBCDIC". Retrieved 2008-11-16.
8. ^ "8.2.2.3. Character encodings". HTML 5.1 Standard. W3C.
9. ^ "8.2.2.3. Character encodings". HTML 5 Standard. W3C.
10. ^ "12.2.3.3 Character encodings". HTML Living Standard. WHATWG.
11. ^ "<meta> - HTML". MDN Web Docs. Mozilla.