8.3 Wide and Multibyte Characters

The familiar char type is sometimes called a narrow character, as opposed to wchar_t, which is a wide character. The key difference between a narrow and wide character is that a wide character can represent any single character in any character set that an implementation supports. A narrow character, on the other hand, might be too small to represent all characters, so multiple narrow char objects can make up a single, logical character called a multibyte character.

Beyond some minimal requirements for the character sets (see Chapter 1), the C++ standard is purposely open-ended and imposes few restrictions on an implementation. Some basic behavioral requirements are that conversion from a narrow character to a wide character must produce an equivalent character. Converting back to a narrow character must restore the original character. The open nature of the standard gives the compiler and library vendor wide latitude. For example, a compiler for Japanese customers might support a variety of Japanese Industrial Standard ( JIS) character sets, but not any European character sets. Another vendor might support multiple ISO 8859 character sets for Western and Eastern Europe, but not any Asian multibyte character sets. Although the standard defines universal characters in terms of the Unicode (ISO/IEC 10646) standard, it does not require any support for Unicode character sets.

This section discusses some of the broad issues in dealing with wide and multibyte characters, but the details of specific characters and character sets are implementation-defined.

8.3.1 Wide Characters

A program that must deal with international character sets might work entirely with wide characters. Although wide characters usually require more memory than narrow characters, they are usually easier to use. Searching for substrings in a wide string is easy because you never have the problem of matching partial characters (which can happen with multibyte characters).

A common implementation of wchar_t is to use Unicode UTF-32 encoding, which means each wide character is 32 bits and represents a single Unicode character. Suppose you want to declare a wide string that contains the Greek letter pi (). You can specify the string with a universal name (see Chapter 1):

wchar_t wpi[] = "\u03c0";

Using UTF-32, the string would contain L"\x03c0". With a different wchar_t implementation, the wpi string would contain different values.

The standard wstring class supports wide strings, and all the I/O streams support wide characters (e.g., wistream, wostream).

8.3.2 Multibyte Characters

A multibyte character represents a single character as a series of one or more bytes, or narrow characters. Because a single character might occupy multiple bytes, working with multibyte strings is more difficult than working with wide strings. For example, if you search a multibyte string for the character '\x20', when you find a match, you must test whether the matching character is actually part of a multibyte character and is therefore not actually a match for the single character you want to find.

Consider the problem of comparing multibyte strings. Suppose you need to sort the strings in ascending order. If one string starts with the character '\xA1' and other starts with '\xB2', it seems that the first is smaller than the second and therefore should come before the second. On the other hand, these characters might be the first of multibyte character sequences, so the strings cannot be compared until you have analyzed the strings for multibyte character sequences.

Multibyte character sets abound, and a particular C++ compiler and library might support only one or just a few. Some multibyte character sets specifically support a particular language, such as the Chinese Big5 character set. The UTF-8 character set supports all Unicode characters using one to six narrow characters.

For example, consider how an implementation might encode the Greek letter pi (), which has a Unicode value of 0x03C0:

char    pi[]  = "\u03c0";

If the implementation's narrow character set is ISO 8859-7 (8-bit Greek), the encoding is 0xF0, so pi[] contains "\xf0". If the narrow character set is UTF-8 (8-bit Unicode), the representation is a multibyte character, and pi[] would contain "\xe0\x8f\x80". Many character sets do not have any encoding for , in which case the contents of pi[] might be "?", or some other implementation-defined marker for unknown characters.

8.3.3 Shift State

You can convert a multibyte character sequence to a wide character and back using the functions in <cwchar>. When performing such conversions, the library might need to keep track of state information during the conversion. This is known as the shift state and is stored in an mbstate_t object (also defined in <cwchar>).

For example, the Japanese Industrial Standard (JIS) encodes single-byte characters and double-byte characters. A 3-byte character sequence shifts from single- to double-byte mode, and another sequence shifts back. The shift state keeps track of the current mode. The initial shift state is single-byte. Thus, the multibyte string "\x1B$B&P\x1B(B" represents one wide character, namely, the Greek letter pi (). The first three characters switch to double-byte mode. The next two characters encode the character, and the final three characters restore single-byte mode.

Shift states are especially important when performing I/O. By definition, file I/O uses multibyte characters. That is, a file is treated as a sequence of narrow characters. When reading a wide-character stream, the narrow characters are converted to wide characters, and when writing a wide stream, wide characters are converted back to multibyte characters. Seeking to a new position in a file might seek to a position that falls in the middle of a multibyte sequence. Therefore, a file position is required to keep track of a shift state in addition to a byte position in the file. See <ios> in Chapter 13.