GSM 03.38

In Mobile telephony GSM 03.38 is a character set used in the Short Message Service of GSM based cell phones. It is defined in GSM recommendation 03.38. Messages sent via this encoding can be encoded in the default GSM 7-bit alphabet, the 8-bit data alphabet, and the 16-bit UTF-16 alphabet. Support of the GSM 7-bit alphabet is mandatory for GSM handsets and network elements, but characters in languages such as Arabic, Chinese, Korean, Japanese or Cyrillic alphabet languages must be encoded using the 16-bit UTF-16 character encoding or an extended national language shift table.

GSM 7 bit default alphabet and extension table of 3GPP TS 23.038 / GSM 03.38

The standard encoding for GSM messages is the 7 bit default alphabet as defined in the 23.038 recommendation. Seven-bit characters must be encoded into octets following one of three packing modes: SMS, CBS or USSD.

Using this encoding, it is possible to send up to 160 characters (140 octets) in one SMS message in the GSM network.

3GPP TS 23.038 / GSM 03.38
	x0	x1	x2	x3	x4	x5	x6	x7	x8	x9	xA	xB	xC	xD	xE	xF
0x	@	£	$	¥	è	é	ù	ì	ò	Ç	LF	Ø	ø	CR	Å	å
1x	Δ	_	Φ	Γ	Λ	Ω	Π	Ψ	Σ	Θ	Ξ	ESC	Æ	æ	ß	É
2x	SP	!	"	#	¤	%	&	'	(	)	*	+	,	-	.	/
3x	0	1	2	3	4	5	6	7	8	9	:	;	<	=	>	?
4x	¡	A	B	C	D	E	F	G	H	I	J	K	L	M	N	O
5x	P	Q	R	S	T	U	V	W	X	Y	Z	Ä	Ö	Ñ	Ü	§
6x	¿	a	b	c	d	e	f	g	h	i	j	k	l	m	n	o
7x	p	q	r	s	t	u	v	w	x	y	z	ä	ö	ñ	ü	à
1B 0x											FF
1B 1x					^							ESC2
1B 2x									{	}						\
1B 3x													[	~	]
1B 4x	|
1B 5x
1B 6x						€
1B 7x

Note that the second part of the table is only accessible if the GSM device supports the 7-bit extension mechanism, using the ESC character prefix. Otherwise, the ESC code itself is interpreted as a space, and the following character will be treated as if there was no leading ESC code. Most of the high part of the table is not used in the default character set, but the GSM standard defines some language code indicators that allows the system to identify national variants of this part, to support more characters than those displayed in the above table.

In a standard GSM text message, all characters are encoded using 7-bit code units, packed together to fill all bits of octets. So, for example, a 160-octet envelope, with no other language indicator but only the standard class prefix, can transport up to (160*8)/7=182.8, that is 182 characters (but note that the ESC code counts for one of them, if characters in the high part of the table are used).

Longer messages may be sent, but will require a continuation prefix and a sequence number on subsequent SMS messages (these prefix bytes and sequence number are counted within the maximum length of the 160-octet payload of the envelope format.

When there are 1 to 6 spare bits in the last octet of a message, these bits are set to zero (these bits do not count as a character but only as a filler). When there are 7 spare bits in the last octet of a message, these bits are set to the 7-bit code of the CR control (also used as a padding filler) instead of being set to zero (where they would be confused with the 7-bit code of an '@' character.

This 7-bit encoding allows the transport of texts encoded in the Basic Latin subset of ASCII, as well as some characters of the ISO Latin 1 character set. It also allows the encoding of texts written in the Greek script, but only capitals; for such use in Greek, the Latin capital letters that look like the Greek letters are reused with the same code, so that the above character set is complete only for modern monotonic Greek restricted to capital letters. A complete support for the Greek alphabet (including small letters) requires a national version of the shifted 7-bit table (using the ESC code for each national character encoded in this shifted table), or an unspecified proprietary 8-bit encoding, or the use of the UCS2 encoding (see below).

Note that the special code marked ESC2 in the table above has also been assigned (and encoded as 0x1B,0x1B) to allow using another alternate 7-bit shift table. But this mechanism has never been used and the UCS2 encoding has been preferred.

GSM 8 bit data encoding

8 bit data encoding mode treats the information as raw data. According to the standard, the alphabet for this encoding is user specific.

UCS-2 Encoding

This encoding allows use of a greater range of characters and languages. UCS-2 can represent the most commonly used latin and eastern characters at the cost of a greater space expense.

A single SMS GSM message using this encoding can have at most 70 characters (140 octets).

Note that on many GSM smartphones, there's no specific preselection of the UCS-2 encoding. The default is to use the 7-bit encoding above, until one enters a character that is not present in the GSM 7-bit table (for example the lowercase c with cedilla 'ç'). In that case, the whole message gets reencoded using the UCS-2 encoding, and the maximum length of the message sent in only 1 SMS is immediately reduced to 70 code units, instead of 160.

To avoid unexpected costs for senders that have a subscription for a limited pack of sent SMS, smartphones should display the number of character used and the maximum number of characters in the composed SMS. When a message does exceeds this maximum, the message will be sent as multiple successive SMS containing parts of the message (each one containing a sequence number, which also uses a few leading characters in each part); these parts will be reassembled later by the recipient.

Some GSM smartphones will alert the user about the number of SMS messages needed to send the message, when it requires more than one.

National Language Shift Tables

Since release 8 of the GSM 23.038 standard, additional characters sets can be accessed through the use of a National Language Shift Table.

These tables allow the use of different character sets according to the language the text is going to be written. The choice of table for a given message is selected in the User Data Header section of an SMS message and can be specified for the whole text (a Locking shift table) or a single character (Single shift table).

Using a shift table, a message can still use 7-bit encoding for the characters, but a different set can be chosen to correctly show accented and language specific characters. This allows up to 155 characters, encoded in 136 octets (140 octets, minus the 4-octets of User Data Header required to indicate the use of a shift table and the language code).

Initially, shift tables for the following languages were specified: Spanish, Portuguese, Turkish, and 10 languages used in India written with a Brahmic script (Bengali, Gujarati, Hindi, Kannada, Malayalam, Oriya, Punjabi, Tamil, Telugu and Urdu).

Until recently, there was still no defined national language shift table for French, Greek, Russian, Bulgarian, Arabic, Hebrew and most Central European languages that need a better coverage than the default 7-bit standard character set and its default 7-bit extension character set: if ever any character is composed that cannot be represented in those default GSM 7-bit sets, the message will be automatically reencoded using UCS-2, with the effect of dividing by more than two the maximum length in characters of messages that can be sent at the price of a single SMS (when a message is split in multiple parts, a few other octets are needed in the User Data Header to indicate the sequence number of each part).

But a revision of GSM 03.38 (in specification document CR 007, version 4.2.0 of september 2001) has added support for more languages using a 7-bit national shift table : English (extended), German, Dutch, Swedish, Danish, Finnish, Norwegian, French, Italian, Hungarian, Polish, Czech, Icelandic, Greek, Russian, Hebrew and Arabic, in addition to the previous languages. Unfortunately, many smartphones (and national operators) still don't support these extensions.

There's also no language shift table for Japanese written in basic kanas, or for Korean written in Hangul jamos, or for Chinese written in the Han script. This is often not a problem in Japan, because it uses other standards than GSM and WAP for messaging.

External links

• GSM 03.38 to Unicode - the GSM 03.38 to Unicode mapping data file from unicode.org.
• Text to GSM 03.38 in C# - Text to GSM 03.38 mapping in the C# programming language.
• JCharset - Java Charset package includes GSM 03.38 support - JCharset - Java Charset package includes GSM 03.38 support