Character coding system is one of the fundamental elements of the software and information processing. Without proper handling of character codes, your software is far from realization of internationalization. Thus the author begins this document with the story on character codes.
In this chapter, basic concepts such as coded character set and encoding are introduced. These terms will be needed to read this document and other documents on internationalization and character codes including Unicode.
At first I begin this chapter by defining a few very important word.
As many people point out, there is a confusion on terminology, since words are
used in various different ways. The author does not want to add a new
terminology to a confusing ocean of various terminologies. Otherwise,
terminology of RFC
2130 will be adopted in this document, besides one exception of a
word 'character set'.
RFC 2050 and means a
character set where all characters have unique numbers by some method. There
are many national and international standards for CCS. Many national standards
for CCS adopt the way of coding so that they obey some of international
standards such as ISO 646 or ISO 2022. ASCII, BS 4730, JISX 0201 Roman, and so
on are examples of ISO-646 variants. All ISO-646 variants, ISO 8859-*, JISX
0208, JISX 0212, KSX 1001, GB 2312, CNS 11643, CCCII, TIS 620, TCVN 5712, and
so on are examples of ISO 2022-compliant CCS. VISCII and Big5 are examples of
non-ISO 2022-compliant CCS. UCS-2 and UCS-4 (ISO 10646) are also examples of
CCS.
RFC 2050 to call
methods to construct an encoding using one or more CCS. This is important when
two or more CCS are used to construct an encoding. ISO 2022 is a method to
construct an encoding from one or more ISO 2022-compliant CCS. ISO 2022 is
very complex system and subsets of ISO 2022 are usually used such as EUC-JP
(ASCII and JISX 0208), ISO-2022-KR (ASCII and KSX 1001), and so on. CES is not
important for encodings with only one 8bit CCS. UTF series (UTF-8, UTF-16LE,
UTF-16BE, and so on) can be regarded as CES whose CCS is Unicode or ISO 10646.
Some other words are usually used related to character codes.
Character code is a widely-used word to mean encoding. This is an primitive and crude word to call the way a computer handles characters with assigning numbers. For example, character code can call encoding and can call coded character set. Thus this word can be used only in the case when both of them can be regard in the same category. This word should be avoided in serious discussions. This document will not use this word hereafter.
Codeset is a word to call encoding or character encoding scheme. [3]
charset is also a well-used word. This word is used very widely, for example, in MIME (like Content-Type: text/plain, charset=iso8859-1), in XLFD (X Logical Font Description) font name (CharSetResigtry and CharSetEncoding fields), and so on. Note that charset in MIME is encoding, while charset in XLFD font name is coded character set. This is very confusing. In this document, charset and character set are used in XLFD meaning, since I think character set should mean a set of characters, not encoding.
Ken Lunde's "CJKV Information Processing" uses a word encoding method. He says that ISO-2022, EUC, Big5, and Shift-JIS are examples of encoding methods. It seems that his encoding method is CES in this document. However, we should notice that Big5 and Shift-JIS are encodings while ISO-2022 and EUC are not. [4]
Character Encoding
Model, Unicode Technical Report #17 (hereafter, "the
Report") suggests five-level model.
TES is also suggested in RFC 2130. Some
examples of TES are: base64, uuencode, BinHex,
quoted-printable, gzip, and so on. TES means a transform of
encoded data which may (or may not) include textual data. Thus, TES is not a
part of character encoding. However, TES is important in the Internet data
exchange.
When using a computer, we rarely have a chance to face with ACR. Though it is true that CJK people have their national standard of ACR (for example, standard for ideograms which can be used for personal names) and some of us may need to handle these ACR with computers (for example, citizen registration system), this is too heavy theme for this document. This is because there are no standardized or encouraged methods to handle these ACR. You may have to build the whole system for such purposes. Good luck!
CCS in "the Report" is same as what I wrote in this document. It has concrete examples: ASCII, ISO 8859-{1,2,...,15}, JISX 0201, JISX 0208, JISX 0212, KSX 1001, KSX 1002, GB 2312, Big5, CNS 11643, TIS 620, VISCII, TCVN 5712, UCS2, UCS4, and so on. Some of them are national standards, some are international standards, and others are de-facto standards.
CEF and CES in "the Report" correspond to CES in this document. This document will not distinguish these two, since I think there are no inconvenience. An encoding with a significant CEF doesn't have a significant CES (in "the Report" meaning), and vice versa. Then why should we have to distinguish these two? The only exception is UTF-16 series. In UTF-16 series, UTF-16 is a CEF and UTF-16BE is a CES. This is the only case where we need distinction between CEF and CES.
Now, CES is a concrete concept with concrete examples: ASCII, ISO 8859-{1,2,...,15}, EUC-JP, EUC-KR, ISO 2022-JP, ISO 2022-JP-1, ISO 2022-JP-2, ISO 2022-CN, ISO 2022-CN-EXT, ISO 2022-KR, ISO 2022, VISCII, UTF-7, UTF-8, UTF-16LE, UTF-16BE, and so on. Now they are encodings themselves.
The most important concept in this section is distinction between coded character set and encoding. Coded character set is a component of encoding. Text data are described in encoding, not coded character set.
To construct an encoding with two or more CCS, CES has to supply a method to avoid collision between these CCS. There are two ways to do that. One is to make all characters in the all CCS have unique code points. The other is to allow characters from different CCS to have the same code point and to have a code such as escape sequence to switch SHIFT STATE, that is, to select one character set.
An encoding with shift states is called STATEFUL and one without shift states is called STATELESS.
Examples of stateful encodings are: ISO 2022-JP, ISO 2022-KR, ISO 2022-INT-1, ISO 2022-INT-2, and so on.
For example, in ISO 2022-JP, two bytes of 0x24 0x2c may mean a Japanese Hiragana character 'GA' or two ASCII character of '$' and ',' according to the shift state.
Encodings are classified into multibyte ones and the others, according to the relationship between number of characters and number of bytes in the encoding.
In non-multibyte encoding, one character is always expressed by one byte. On the other hand, one character may expressed in one or more bytes in multibyte encoding. Note that the number is not fixed even in a single encoding.
Examples of multibyte encodings are: EUC-JP, EUC-KR, ISO 2022-JP, Shift-JIS, Big5, UHC, UTF-8, and so on. Note that all of UTF-* are multibyte.
Examples of non-multibyte encodings are: ISO 8859-1, ISO 8859-2, TIS 620, VISCII, and so on.
Note that even in non-multibyte encoding, number of characters and number of bytes may differ if the encoding is stateful.
Ken Lunde's "CJKV Information Processing" [5] classifies encoding methods into the following three categories:
Modal corresponds to stateful in this document. Other two are stateless, where non-modal is multibyte and fixed-length is non-multibyte. However, I think stateful - stateless and multibyte - non-multibyte are independent concept. [6]
One ASCII character is always expressed by one byte and occupies one column on console or X terminal emulators (fixed font for X). One must not make such an assumption for I18N programming and have to clearly distinguish number of bytes, characters, and columns.
Speaking of relationship between characters and bytes, in multibyte encodings, two or more bytes may be needed to express one character. In stateful encodings, escape sequences are not related to any characters.
Number of columns is not defined in any standards. However, it is usual that CJK ideograms, Japanese Hiragana and Katakana, and Korean Hangul occupy two columns in console or X terminal emulators. Note that 'Full-width forms' in UCS-2 and UCS-4 coded character set will occupy two columns and 'Half-width forms' will occupy one column. Combining characters used for Thai and so on can be regarded as zero-column characters. Though there are no standards, you can use wcwidth() and wcswidth() for this purpose. See Number of Columns, Section 7.1.2 for detail.
Introduction to i18n
14 February 2003kubota@debian.org