GB 18030

From Wikipedia, the free encyclopedia
  (Redirected from GB18030)
Jump to: navigation, search

GB18030 is a Chinese government standard, described as Information technology — Chinese coded character set and defines the required language and character support necessary for software in China. GB18030 is the registered Internet name for the official character set of the People's Republic of China (PRC) superseding GB2312. As a Unicode Transformation Format (i.e. an encoding of all Unicode code points) compatible with legacy encodings including GB2312, CP936[a], and GBK 1.0, GB18030 supports both simplified and traditional Chinese characters.

In addition to the "GB18030 character encoding", this standard contains requirements about which scripts must be supported, font support, etc.

History[edit]

See also: GB 2312, GBK, and Code page 1386

The GB18030 character set is formally called "Chinese National Standard GB 18030-2005: Information technology — Chinese coded character set". GB abbreviates Guójiā Biāozhǔn (国家标准), which means national standard in Chinese. The standard was published by the China Standard Press, Beijing, November 8, 2005. Only a portion of the standard is mandatory. Since May 1, 2006, support for the mandatory subset is officially required for all software products sold in the PRC.

An older version of the standard, known as "Chinese National Standard GB 18030-2000: Information Technology — Chinese ideograms coded character set for information interchange — Extension for the basic set", was published on March 17, 2000. The encoding scheme remains the same in the new version, except that code points for the characters ḿ and U+E7C7 have been exchanged.[1]:534 More code points are now associated with characters due to update of Unicode, especially the appearance of CJK Unified Ideographs Extension B. Some characters used by ethnic minorities in China, such as Mongolian characters and Tibetan characters (GB 16959-1997 and GB/T 20542-2006), have been added as well, which accounts for the renaming of the standard.

Compared with its ancestors, GB 18030's mapping to Unicode has been modified for the 81 characters that were provisionally assigned a Unicode Private Use Area code point (U+E000–F8FF) in GBK 1.0 and that have later been encoded in Unicode.[2] This is specified in Appendix E of GB 18030.[3]:499[1]:534 There are 24 characters in GB 18030-2005 that are still mapped to Unicode PUA.[4]


Private use characters in GB-to-Unicode mappings
GB byte
sequence
Unicode code point (blue = private use)
GBK 1.0[5] GB 18030
-2005
Unicode 4.1
A6 D9[6]:108 U+E78D U+FE10
A6 DA U+E78E U+FE12
A6 DB U+E78F U+FE11
A6 DC U+E790 U+FE13
A6 DD U+E791 U+FE14
A6 DE U+E792 U+FE15
A6 DF U+E793 U+FE16
A6 EC U+E794 U+FE17
A6 ED U+E795 U+FE18
A8 BC U+E7C7 U+1E3F ḿ
A8 BF U+E7C8 U+01F9 ǹ
A9 89 U+E7E7 U+303E
A9 8A U+E7E8 U+2FF0
A9 8B U+E7E9 U+2FF1
A9 8C U+E7EA U+2FF2
A9 8D U+E7EB U+2FF3
A9 8E U+E7EC U+2FF4
A9 8F U+E7ED U+2FF5
A9 90 U+E7EE U+2FF6
A9 91 U+E7EF U+2FF7
A9 92 U+E7F0 U+2FF8
A9 93 U+E7F1 U+2FF9
A9 94[6]:173 U+E7F2 U+2FFA
A9 95 U+E7F3 U+2FFB
FE 50 U+E815 U+2E81
FE 51 U+E816 U+20087 𠂇
FE 52 U+E817 U+20089 𠂉
FE 53 U+E818 U+200CC 𠃌
FE 54 U+E819 U+2E84
FE 55 U+E81A U+3473
FE 56 U+E81B U+3447
FE 57 U+E81C U+2E88
FE 58 U+E81D U+2E8B
FE 59 U+E81E U+9FB4
FE 5A U+E81F U+359E
FE 5B U+E820 U+361A
FE 5C U+E821 U+360E
FE 5D U+E822 U+2E8C
FE 5E U+E823 U+2E97
FE 5F U+E824 U+396E
FE 60 U+E825 U+3918
FE 61 U+E826 U+9FB5
FE 62 U+E827 U+39CF
FE 63 U+E828 U+39DF
FE 64 U+E829 U+3A73
FE 65 U+E82A U+39D0
FE 66 U+E82B U+9FB6
FE 67 U+E82C U+9FB7
FE 68 U+E82D U+3B4E
FE 69 U+E82E U+3C6E
FE 6A U+E82F U+3CE0
FE 6B U+E830 U+2EA7
FE 6C U+E831 U+215D7 𡗗
FE 6D U+E832 U+9FB8
FE 6E U+E833 U+2EAA
FE 6F U+E834 U+4056
FE 70 U+E835 U+415F
FE 71 U+E836 U+2EAE
FE 72 U+E837 U+4337
FE 73 U+E838 U+2EB3
FE 74 U+E839 U+2EB6
FE 75 U+E83A U+2EB7
FE 76 U+E83B U+2298F 𢦏
FE 77 U+E83C U+43B1
FE 78 U+E83D U+43AC
FE 79 U+E83E U+2EBB
FE 7A U+E83F U+43DD
FE 7B U+E840 U+44D6
FE 7C U+E841 U+4661
FE 7D U+E842 U+464C
FE 7E U+E843 U+9FB9
FE 80 U+E844 U+4723
FE 81 U+E845 U+4729
FE 82 U+E846 U+477C
FE 83 U+E847 U+478D
FE 84 U+E848 U+2ECA
FE 85 U+E849 U+4947
FE 86 U+E84A U+497A
FE 87 U+E84B U+497D
FE 88 U+E84C U+4982
FE 89 U+E84D U+4983
FE 8A U+E84E U+4985
FE 8B U+E84F U+4986
FE 8C U+E850 U+499F
FE 8D U+E851 U+499B
FE 8E U+E852 U+49B7
FE 8F U+E853 U+49B6
FE 90 U+E854 U+9FBA
FE 91 U+E855 U+241FE 𤇾
FE 92 U+E856 U+4CA3
FE 93 U+E857 U+4C9F
FE 94 U+E858 U+4CA0
FE 95 U+E859 U+4CA1
FE 96 U+E85A U+4C77
FE 97 U+E85B U+4CA2
FE 98 U+E85C U+4D13
FE 99 U+E85D U+4D14
FE 9A U+E85E U+4D15
FE 9B U+E85F U+4D16
FE 9C U+E860 U+4D17
FE 9D U+E861 U+4D18
FE 9E U+E862 U+4D19
FE 9F U+E863 U+4DAE
FE A0 U+E864 U+9FBB

As a national standard[edit]

The mandatory part of GB 18030-2005 consists of 1 byte and 2 byte encoding, together with 4 byte encoding for CJK Unified Ideographs Extension A. The corresponding Unicode code points of this subset lie entirely in the BMP.

Most major computer companies had already standardised on some version of Unicode as the primary format for use in their binary formats and OS calls. However, they mostly had only supported code points in the BMP originally defined in Unicode 1.0, which supported only 65,536 codepoints and was often encoded in 16 bits as UCS-2.

In a move of historic significance for software supporting Unicode, the PRC decided to mandate support of certain code points outside the BMP. This means that software can no longer get away with treating characters as 16 bit fixed width entities (UCS-2). Therefore, they must either process the data in a variable width format (such as UTF-8 or UTF-16), which are the most common choices, or move to a larger fixed width format (such as UCS-4 or UTF-32). Microsoft made the change from UCS-2 to UTF-16 with Windows 2000.

Mapping [edit]

Part of the mapping data is from a lookup table (similarly to GBK). The rest is calculated algorithmically. Unfortunately it also inherits the bad aspects of the legacy standards on which it is based (most notably needing special code to safely find ASCII characters in a GB18030 sequence).

GB 18030 encoding[7]
GB 18030 code points[b] Unicode
byte 1 (MSB) byte 2 byte 3 byte 4
007F 128 0000007F
80 unassigned
81FE 40FE except 7F[c] 23940 0080FFFF except D800DFFF[d]
8184 3039 81FE 3039 39420
unassigned D800DFFF[e]
90E3 3039 81FE 3039 1048576 1000010FFFF
FF invalid
Total 1112064

The one- and two-byte code points are essentially GBK. The four byte scheme can be thought of as consisting of two units, each of two bytes. Each unit has a similar format to a GBK two byte character but with a range of values for the second byte of 0x30–0x39 (the ASCII codes for decimal digits). The first byte has the range 0x81 to 0xFE, as before. This means that a string search routine that is safe for GBK should also be reasonably safe for GB18030 (in much the same way that a basic byte-oriented search routine is reasonably safe for EUC).

This gives a total of 1,587,600 (126×10×126×10) possible 4 byte sequences, which is easily sufficient to cover Unicode's 1,111,998 (17×65536 − 2048 surrogates − 66 noncharacters) assigned and reserved code points. (Surrogates and noncharacters are considered designated but not assigned.)

Unfortunately, to further complicate matters there are no simple rules to translate between a 4 byte sequence and its corresponding code point. Instead, codes are allocated sequentially (with the first byte containing the most significant part and the last the least significant part) only to Unicode code points that are not mapped in any other manner. For example:

U+00DE (Þ) → 81 30 89 37
U+00DF (ß) → 81 30 89 38
U+00E0 (à) → A8 A4
U+00E1 (á) → A8 A2
U+00E2 (â) → 81 30 89 39
U+00E3 (ã) → 81 30 8A 30

An offset table is used in the WHATWG and W3C version of GB 18030 to efficiently translate code points.[8]

Windows support[edit]

Encoding[edit]

Windows 2000 can support the GB18030 encoding if GB18030 Support Package [1] is installed. Windows XP can support it natively. Microsoft SQL Server cannot (including SQL Server 2008) as it can use UCS-2 but not UTF-16 (except through the use of varbinary blobs). The open source PostgreSQL database supports GB18030 through its full support for UTF-8.

More specifically, supporting the GB18030 encoding on Windows means that Code Page 54936 is supported by MultiByteToWideChar and WideCharToMultiByte. Due to the backward compatibility of the mapping, many files in GB18030 can be actually opened successfully as the legacy Code Page 936, that is GBK, even if the Code Page 54936 is not supported. However, that is only true if the file in question contains only GBK characters. Loading will fail or cause corrupted result if the file contains characters that do not exist in GBK (see below for examples).

Glyphs[edit]

The GB18030 Support Package contains SimSun18030.ttc, a TrueType font collection file which combines two Chinese fonts, SimSun-18030 and NSimSun-18030.

The SimSun 18030 font includes all the characters in Unicode 2.1 plus new characters found in the Unicode CJK Unified Ideographs Extension A section, but despite its name, it does not contain glyphs for all GB 18030 characters. Note that all (about a million) Unicode code points up to U+10FFFF can be encoded as GB 18030, hence "a font that fully supports GB 18030" would mean a font that contains glyphs for all Unicode characters, not only for CJK ones. HAN NOM A and HAN NOM B [2] are free fonts, which include all the characters in the Extension A and the Extension B, more exhaustive than SimSun-18030, or even than Simsun (Founder Extended), but they don't support all code points defined in Unicode 5.0.0 either.

Notes[edit]

  1. ^ with the exception of the euro sign which is given a single byte code of 0x80 in Microsoft's later versions of GBK and a two byte code of A2 E3 in GB18030
  2. ^ Including the 66 Unicode noncharacters
  3. ^ For a finer division of this range see GBK#Encoding.
  4. ^ Some code points are encoded with two bytes (upper row), the others with four bytes (lower row).
  5. ^ These are surrogate code points; they have no meaning outside of UTF-16 encoding.

References[edit]

  1. ^ a b Standardization Administration of China (SAC) (2005-11-18). GB 18030-2005: Information Technology—Chinese coded character set. 
  2. ^ "Unicode FAQ on GB 18030". ICU Project. Retrieved 10 September 2016. 
  3. ^ Standardization Administration of China (SAC) (2000-03-17). GB 18030-2000: Information Technology—Chinese coded character set for information interchange — Extension for the basic set. 
  4. ^ Lunde, Ken. "L2/06-394 Update on GB 18030:2005". UTC Document Registry. Retrieved 2016-09-10. 
  5. ^ "Group:GBK外字". GlyphWiki. Retrieved 11 September 2016. 
  6. ^ a b Lunde, Ken (December 2008). CJKV Information Processing. O'Reilly Media, Inc. ISBN 978-0-596-51447-1. Retrieved 11 September 2016. 
  7. ^ Authoritative mapping table between GB18030-2000 and Unicode. ICU – International Components for Unicode. 2001-02-21. Accessed 2016-09-04.
  8. ^ "Encoding Standard # gb18030-index". W3C. Retrieved 2016-09-24. 

See also[edit]

External links[edit]