Character encodings in HTML
HTML (Hypertext Markup Language) has been in use since 1991, but HTML 4.0 (December 1997) was the first standardized version where international characters were given reasonably complete treatment. When an HTML document includes special characters outside the range of seven-bit ASCII two goals are worth considering: the information's integrity, and universal browser display.
Specifying the document's character encoding
There are several ways to specify which character encoding is used in the document. First, the web server can include the character encoding or "
charset" in the Hypertext Transfer Protocol (HTTP)
Content-Type header, which would typically look like this:
Content-Type: text/html; charset=ISO-8859-4
This method gives the HTTP server a convenient way to alter document's encoding according to content negotiation; certain HTTP server software can do it, for example Apache with the module mod_charset_lite.
For HTML it is possible to include this information inside the
head element near the top of the document:
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<?xml version="1.0" encoding="ISO-8859-1"?>
Note that as the character encoding can't be known until this declaration is parsed, there can be a problem knowing which encoding is used for the declaration itself. The main principle is that the declaration shall be encoded in pure ASCII, and therefore (if the declaration is inside the file) the encoding needs to be an ASCII extension. In order to allow encodings not backwards compatible with ASCII, browsers must be able to parse declarations in such encodings. Examples of such encodings are UTF-16BE and UTF-16LE.
As of HTML5 the recommended charset is UTF-8. An "encoding sniffing algorithm" is defined in the specification to determine the character encoding of the document based on multiple sources of input, including:
- Explicit user instruction
- An explicit meta tag within the first 1024 bytes of the document
- A Byte order mark within the first three bytes of the document
- The HTTP Content-Type or other transport layer information
- Analysis of the document bytes looking for specific sequences or ranges of byte values, and other tentative detection mechanisms.
For ASCII-compatible character encodings the consequence of choosing incorrectly is that characters outside the printable ASCII range (32 to 126) usually appear incorrectly. This presents few problems for English-speaking users, but other languages regularly—in some cases, always—require characters outside that range. In CJK environments where there are several different multi-byte encodings in use, auto-detection is also often employed. Finally, browsers usually permit the user to override incorrect charset label manually as well.
It is increasingly common for multilingual websites and websites in non-Western languages to use UTF-8, which allows use of the same encoding for all languages. UTF-16 or UTF-32, which can be used for all languages as well, are less widely used because they can be harder to handle in programming languages that assume a byte-oriented ASCII superset encoding, and they are less efficient for text with a high frequency of ASCII characters, which is usually the case for HTML documents.
Successful viewing of a page is not necessarily an indication that its encoding is specified correctly. If the page's creator and reader are both assuming some platform-specific character encoding, and the server does not send any identifying information, then the reader will nonetheless see the page as the creator intended, but other readers on different platforms or with different native languages will not see the page as intended.
In addition to native character encodings, characters can also be encoded as character references, which can be numeric character references (decimal or hexadecimal) or character entity references. Character entity references are also sometimes referred to as named entities, or HTML entities for HTML. HTML's usage of character references derives from SGML.
HTML character references
where nnnn is the code point in decimal form, and hhhh is the code point in hexadecimal form. The x must be lowercase in XML documents. The nnnn or hhhh may be any number of digits and may include leading zeros. The hhhh may mix uppercase and lowercase, though uppercase is the usual style.
Not all web browsers or email clients used by receivers of HTML documents, or text editors used by authors of HTML documents, will be able to render all HTML characters. Most modern software is able to display most or all of the characters for the user's language, and will draw a box or other clear indicator for characters they cannot render.
For codes from 0 to 127, the original 7-bit ASCII standard set, most of these characters can be used without a character reference. Codes from 160 to 255 can all be created using character entity names. Only a few higher-numbered codes can be created using entity names, but all can be created by decimal number character reference.
Character entity references can also have the format
&name; where name is a case-sensitive alphanumeric string. For example, "λ" can also be encoded as
λ in an HTML document. The character entity references
& are predefined in HTML and SGML, because
& are already used to delimit markup. This notably does not include XML's
' (') entity. For a list of all named HTML character entity references (about 250), see List of XML and HTML character entity references.
Unnecessary use of HTML character references may significantly reduce HTML readability. If the character encoding for a web page is chosen appropriately, then HTML character references are usually only required for markup delimiting characters as mentioned above, and for a few special characters (or none at all if a native Unicode encoding like UTF-8 is used). Incorrect HTML entity escaping may also open up security vulnerabilities for injection attacks such as cross-site scripting. If HTML attributes are left unquoted, certain characters, most importantly whitespace, such as space and tab, must be escaped using entities. Other languages related to HTML have their own methods of escaping characters.
- 0 to 31, except 9, 10, and 13 (C0 control characters)
- 127 (DEL character)
- 128 to 159 (x80 – x9F, C1 control characters)
- 55296 to 57343 (xD800 – xDFFF, the UTF-16 surrogate halves)
The Unicode standard also forbids:
- 65534 and 65535 (xFFFE – xFFFF), non-characters, related to xFEFF, the byte order mark.
These characters are not allowed by numeric character references. However, references to characters 128–159 are commonly interpreted by lenient web browsers as if they were references to the characters assigned to bytes 128–159 (decimal) in the Windows-1252 character encoding. This is in violation of HTML and SGML standards, and the characters are already assigned to higher code points, so HTML documents should always use the higher code points. For example the trademark sign (™) should represented with
™ and not with
The characters 9 (tab), 10 (linefeed), and 13 (carriage return) are allowed in HTML documents, but, along with 32 (space) are all considered "whitespace". The "form feed" control character, which would be at 12, is not allowed in HTML documents, but is also mentioned as being one of the "white space" characters – perhaps an oversight in the specifications. In HTML, most consecutive occurrences of white space characters, except in a
<pre> block, are interpreted as comprising a single "word separator" for rendering purposes. A word separator is typically rendered a single en-width space in European languages, but not in all the others.
XML character references
Unlike traditional HTML with its large range of character entity references, in XML there are only five predefined character entity references. These are used to escape characters that are markup sensitive in certain contexts:
&→ & (ampersand, U+0026)
<→ < (less-than sign, U+003C)
>→ > (greater-than sign, U+003E)
"→ " (quotation mark, U+0022)
'→ ' (apostrophe, U+0027)
All other character entity references have to be defined before they can be used. For example, use of
é (which gives é, Latin lower-case E with acute accent, U+00E9 in Unicode) in an XML document will generate an error unless the entity has already been defined. XML also requires that the
x in hexadecimal numeric references be in lowercase: for example
ਛ rather than
ਛ. XHTML, which is an XML application, supports the HTML entity set, along with XML's predefined entities.
- Charset sniffing – used by many browsers when character encoding metadata is not available
- Unicode and HTML
- Language code
- List of XML and HTML character entity references
- "Content-Type", Hypertext Transfer Protocol (HTTP/1.1): Semantics and Content, IETF, June 2014, retrieved 2014-07-30
- Apache Module mod_charset_lite
- "Specifying the document's character encoding", HTML5, World Wide Web Consortium, 17 June 2014, retrieved 2014-07-30
- Bray, T.; Paoli, J.; Sperberg-McQueen, C.; Maler, E.; Yergeau, F. (26 November 2008), "Prolog and Document Type Declaration", XML, W3C, retrieved 8 March 2010
- HTML5 prescan a byte stream to determine its encoding
- "SGML Declaration of HTML 4". HTML 4.01 Specification. World Wide Web Consortium (W3C). 24 December 1999. Retrieved 2014-09-06.
- "Text - White space". HTML 4.01 Specification. World Wide Web Consortium (W3C). 24 December 1999. Retrieved 2014-09-06.
- Bray, T.; Paoli, J.; Sperberg-McQueen, C.; Maler, E.; Yergeau, F. (26 November 2008), "Character and Entity References", XML, W3C, retrieved 8 March 2010