mirror of
git://git.gnupg.org/gnupg.git
synced 2024-10-31 20:08:43 +01:00
211 lines
16 KiB
Plaintext
211 lines
16 KiB
Plaintext
UTF-8 decoder capability and stress test |
|
||
---------------------------------------- |
|
||
|
|
||
Markus Kuhn <mkuhn@acm.org> - 1999-11-07 |
|
||
|
|
||
This test text examines, how UTF-8 decoders handle various types of |
|
||
corrupted or otherwise interesting UTF-8 sequences. |
|
||
|
|
||
According to ISO 10646-1, sections R.7 and 2.3c, a device receiving |
|
||
UTF-8 shall interpret a "malformed sequence in the same way that it |
|
||
interprets a character that is outside the adopted subset". This means |
|
||
usually that the malformed UTF-8 sequence is replaced by a replacement |
|
||
character (U+FFFD), which looks a bit like an inverted question mark, |
|
||
or a similar symbol. It might be a good idea to visually distinguish a |
|
||
malformed UTF-8 sequence from a correctly encoded Unicode character |
|
||
that is just not available in the current font but otherwise fully |
|
||
legal. For both cases, a clearly recognisable symbol should be used. |
|
||
Just ignoring malformed sequences or unavailable characters will make |
|
||
debugging more difficult and can lead to user confusion. |
|
||
|
|
||
Test sequences: |
|
||
|
|
||
Check, whether a malformed UTF-8 sequence is (1) represented at all, |
|
||
(2) represented by exactly one replacement character (or equivalent |
|
||
signal), and (3) the following quotation mark after an illegal UTF-8 |
|
||
sequence is correctly displayed, i.e. proper resynchronization takes |
|
||
place. This file says "THE END" in the last line, so if you don't see |
|
||
that, your decoder crashed somehow before, which is also not nice. |
|
||
|
|
||
All lines in this file are exactly 79 characters long (plus the line |
|
||
feed). In addition, all lines end with "|", except for the two test |
|
||
lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls |
|
||
U+0000 and U+007F. If you display this file with a fixed-width font, |
|
||
these "|" characters should all line up in column 79 (right margin). |
|
||
This allows you to test quickly, whether your UTF-8 decoder finds the |
|
||
correct number of characters in every line. |
|
||
|
|
||
Here come the tests: |
|
||
|
|
||
1 Some correct UTF-8 text |
|
||
|
|
||
You should see the Greek word 'kosme': "κόσμε" |
|
||
|
|
||
2 Minimum and maximum valid values for every UTF-8 sequence length |
|
||
|
|
||
You should see a correctly encoded character each time (which is not |
|
||
necessarily an existing character). |
|
||
|
|
||
2.1 First possible sequence of a certain length |
|
||
|
|
||
2.1.1 1 byte (U-00000000): " "
|
||
2.1.2 2 bytes (U-00000080): "" |
|
||
2.1.3 3 bytes (U-00000800): "ࠀ" |
|
||
2.1.4 4 bytes (U-00010000): "𐀀" |
|
||
2.1.5 5 bytes (U-00200000): "<22><><EFBFBD><EFBFBD><EFBFBD>" |
|
||
2.1.6 6 bytes (U-04000000): "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
||
|
|
||
2.2 Last possible sequence of a certain length |
|
||
|
|
||
2.2.1 1 byte (U-0000007f): ""
|
||
2.2.2 2 bytes (U-000007ff): "߿" |
|
||
2.2.3 3 bytes (U-0000ffff): "" |
|
||
2.2.4 4 bytes (U-001fffff): "<22><><EFBFBD><EFBFBD>" |
|
||
2.2.5 5 bytes (U-03ffffff): "<22><><EFBFBD><EFBFBD><EFBFBD>" |
|
||
2.2.6 6 bytes (U-7fffffff): "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
||
|
|
||
3 Malformed sequences |
|
||
|
|
||
3.1 Unexpected continuation bytes |
|
||
|
|
||
Each unexpected continuation byte should be separately signalled as a |
|
||
malformed sequence of its own. |
|
||
|
|
||
3.1.1 First continuation byte 0x80: "<22>" |
|
||
3.1.2 Last continuation byte 0xbf: "<22>" |
|
||
|
|
||
3.1.3 2 continuation bytes: "<22><>" |
|
||
3.1.4 3 continuation bytes: "<22><><EFBFBD>" |
|
||
3.1.5 4 continuation bytes: "<22><><EFBFBD><EFBFBD>" |
|
||
3.1.6 5 continuation bytes: "<22><><EFBFBD><EFBFBD><EFBFBD>" |
|
||
3.1.7 6 continuation bytes: "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
||
3.1.8 7 continuation bytes: "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
||
|
|
||
3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf): |
|
||
|
|
||
"<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> |
|
||
<20><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> |
|
||
<20><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> |
|
||
<20><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
||
|
|
||
3.2 Lonely start characters |
|
||
|
|
||
3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf), |
|
||
each followed by a space character: |
|
||
|
|
||
"<22> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> |
|
||
<20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> " |
|
||
|
|
||
3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef), |
|
||
each followed by a space character: |
|
||
|
|
||
"<22> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> " |
|
||
|
|
||
3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7), |
|
||
each followed by a space character: |
|
||
|
|
||
"<22> <20> <20> <20> <20> <20> <20> <20> " |
|
||
|
|
||
3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb), |
|
||
each followed by a space character: |
|
||
|
|
||
"<22> <20> <20> <20> " |
|
||
|
|
||
3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd), |
|
||
each followed by a space character: |
|
||
|
|
||
"<22> <20> " |
|
||
|
|
||
3.3 Sequences with last continuation byte missing |
|
||
|
|
||
All bytes of an incomplete sequence should be signalled as a single |
|
||
malformed sequence, i.e., you should see only a single replacement |
|
||
characters in each of the next 10 tests. (Characters as in section 2) |
|
||
|
|
||
3.3.1 2-byte sequence with last byte missing (U+0000): "<22>" |
|
||
3.3.2 3-byte sequence with last byte missing (U+0000): "<22><>" |
|
||
3.3.3 4-byte sequence with last byte missing (U+0000): "<22><><EFBFBD>" |
|
||
3.3.4 5-byte sequence with last byte missing (U+0000): "<22><><EFBFBD><EFBFBD>" |
|
||
3.3.5 6-byte sequence with last byte missing (U+0000): "<22><><EFBFBD><EFBFBD><EFBFBD>" |
|
||
3.3.6 2-byte sequence with last byte missing (U-000007ff): "<22>" |
|
||
3.3.7 3-byte sequence with last byte missing (U-0000ffff): "<22><>" |
|
||
3.3.8 4-byte sequence with last byte missing (U-001fffff): "<22><><EFBFBD>" |
|
||
3.3.9 5-byte sequence with last byte missing (U-03ffffff): "<22><><EFBFBD><EFBFBD>" |
|
||
3.3.10 6-byte sequence with last byte missing (U-7fffffff): "<22><><EFBFBD><EFBFBD><EFBFBD>" |
|
||
|
|
||
3.4 Concatenation of incomplete sequences |
|
||
|
|
||
All the 10 sequences of 3.3 concatenated, you should see 10 malformed |
|
||
sequences being signalled: |
|
||
|
|
||
"<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
||
|
|
||
3.5 Impossible bytes |
|
||
|
|
||
The following two bytes cannot appear in a correct UTF-8 string |
|
||
|
|
||
3.5.1 fe = "<22>" |
|
||
3.5.2 ff = "<22>" |
|
||
3.5.3 fe fe ff ff = "<22><><EFBFBD><EFBFBD>" |
|
||
|
|
||
4 Overlong sequences |
|
||
|
|
||
The following sequences are not malformed according to the letter of |
|
||
the Unicode 2.0 standard. However, they are longer then necessary and |
|
||
a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8 |
|
||
decoder" should reject them just like malformed sequences for two |
|
||
reasons: (1) It helps to debug applications if overlong sequences are |
|
||
not treated as valid representations of characters, because this helps |
|
||
to spot problems more quickly. (2) Overlong sequences provide |
|
||
alternative representations of characters, that could maliciously be |
|
||
used to bypass filters that check only for ASCII characters. For |
|
||
instance, a 2-byte encoded line feed (LF) would not be caught by a |
|
||
line counter that counts only 0x0a bytes, but it would still be |
|
||
processed as a line feed by an unsafe UTF-8 decoder later in the |
|
||
pipeline. From a security point of view, ASCII compatibility of UTF-8 |
|
||
sequences means also, that ASCII characters are *only* allowed to be |
|
||
represented by ASCII bytes in the range 0x00-0x7f. To ensure this |
|
||
aspect of ASCII compatibility, use only "safe UTF-8 decoders" that |
|
||
reject overlong UTF-8 sequences for which a shorter encoding exists. |
|
||
|
|
||
4.1 Examples of an overlong ASCII character |
|
||
|
|
||
With a safe UTF-8 decoder, all of the following five overlong |
|
||
representations of the ASCII character slash ("/") should be rejected |
|
||
like a malformed UTF-8 sequence, for instance by substituting it with |
|
||
a replacement character. If you see a slash below, you do not have a |
|
||
safe UTF-8 decoder! |
|
||
|
|
||
4.1.1 U+002f = c0 af = "<22><>" |
|
||
4.1.2 U+002f = e0 80 af = "<22><><EFBFBD>" |
|
||
4.1.3 U+002f = f0 80 80 af = "<22><><EFBFBD><EFBFBD>" |
|
||
4.1.4 U+002f = f8 80 80 80 af = "<22><><EFBFBD><EFBFBD><EFBFBD>" |
|
||
4.1.5 U+002f = fc 80 80 80 80 af = "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
||
|
|
||
4.2 Maximum overlong sequences |
|
||
|
|
||
Below you see the highest Unicode value that is still resulting in an |
|
||
overlong sequence if represented with the given number of bytes. This |
|
||
is a boundary test for safe UTF-8 decoders. All five characters should |
|
||
be rejected like malformed UTF-8 sequences. |
|
||
|
|
||
4.2.1 U-0000007f = c1 bf = "<22><>" |
|
||
4.2.2 U-000007ff = e0 9f bf = "<22><><EFBFBD>" |
|
||
4.2.3 U-0000ffff = f0 8f bf bf = "<22><><EFBFBD><EFBFBD>" |
|
||
4.2.4 U-001fffff = f8 87 bf bf bf = "<22><><EFBFBD><EFBFBD><EFBFBD>" |
|
||
4.2.5 U-03ffffff = fc 83 bf bf bf bf = "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
||
|
|
||
4.3 Overlong representation of the NUL character |
|
||
|
|
||
The following five sequences should also be rejected like malformed |
|
||
UTF-8 sequences and should not be treated like the ASCII NUL |
|
||
character. |
|
||
|
|
||
4.3.1 U+0000 = c0 80 = "<22><>" |
|
||
4.3.2 U+0000 = e0 80 80 = "<22><><EFBFBD>" |
|
||
4.3.3 U+0000 = f0 80 80 80 = "<22><><EFBFBD><EFBFBD>" |
|
||
4.3.4 U+0000 = f8 80 80 80 80 = "<22><><EFBFBD><EFBFBD><EFBFBD>" |
|
||
4.3.5 U+0000 = fc 80 80 80 80 80 = "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
||
|
|
||
THE END |
|