1
0
mirror of git://git.gnupg.org/gnupg.git synced 2024-12-25 10:49:57 +01:00
gnupg/contrib/UTF-8-test.txt
2002-06-29 13:31:13 +00:00

211 lines
16 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

UTF-8 decoder capability and stress test |
---------------------------------------- |
|
Markus Kuhn <mkuhn@acm.org> - 1999-11-07 |
|
This test text examines, how UTF-8 decoders handle various types of |
corrupted or otherwise interesting UTF-8 sequences. |
|
According to ISO 10646-1, sections R.7 and 2.3c, a device receiving |
UTF-8 shall interpret a "malformed sequence in the same way that it |
interprets a character that is outside the adopted subset". This means |
usually that the malformed UTF-8 sequence is replaced by a replacement |
character (U+FFFD), which looks a bit like an inverted question mark, |
or a similar symbol. It might be a good idea to visually distinguish a |
malformed UTF-8 sequence from a correctly encoded Unicode character |
that is just not available in the current font but otherwise fully |
legal. For both cases, a clearly recognisable symbol should be used. |
Just ignoring malformed sequences or unavailable characters will make |
debugging more difficult and can lead to user confusion. |
|
Test sequences: |
|
Check, whether a malformed UTF-8 sequence is (1) represented at all, |
(2) represented by exactly one replacement character (or equivalent |
signal), and (3) the following quotation mark after an illegal UTF-8 |
sequence is correctly displayed, i.e. proper resynchronization takes |
place. This file says "THE END" in the last line, so if you don't see |
that, your decoder crashed somehow before, which is also not nice. |
|
All lines in this file are exactly 79 characters long (plus the line |
feed). In addition, all lines end with "|", except for the two test |
lines 2.1.1 and 2.2.1, which contain non-printable ASCII controls |
U+0000 and U+007F. If you display this file with a fixed-width font, |
these "|" characters should all line up in column 79 (right margin). |
This allows you to test quickly, whether your UTF-8 decoder finds the |
correct number of characters in every line. |
|
Here come the tests: |
|
1 Some correct UTF-8 text |
|
You should see the Greek word 'kosme': "κόσμε" |
|
2 Minimum and maximum valid values for every UTF-8 sequence length |
|
You should see a correctly encoded character each time (which is not |
necessarily an existing character). |
|
2.1 First possible sequence of a certain length |
|
2.1.1 1 byte (U-00000000): ""
2.1.2 2 bytes (U-00000080): "€" |
2.1.3 3 bytes (U-00000800): "ࠀ" |
2.1.4 4 bytes (U-00010000): "𐀀" |
2.1.5 5 bytes (U-00200000): "<22><><EFBFBD><EFBFBD><EFBFBD>" |
2.1.6 6 bytes (U-04000000): "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
2.2 Last possible sequence of a certain length |
|
2.2.1 1 byte (U-0000007f): ""
2.2.2 2 bytes (U-000007ff): "߿" |
2.2.3 3 bytes (U-0000ffff): "￿" |
2.2.4 4 bytes (U-001fffff): "<22><><EFBFBD><EFBFBD>" |
2.2.5 5 bytes (U-03ffffff): "<22><><EFBFBD><EFBFBD><EFBFBD>" |
2.2.6 6 bytes (U-7fffffff): "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
3 Malformed sequences |
|
3.1 Unexpected continuation bytes |
|
Each unexpected continuation byte should be separately signalled as a |
malformed sequence of its own. |
|
3.1.1 First continuation byte 0x80: "<22>" |
3.1.2 Last continuation byte 0xbf: "<22>" |
|
3.1.3 2 continuation bytes: "<22><>" |
3.1.4 3 continuation bytes: "<22><><EFBFBD>" |
3.1.5 4 continuation bytes: "<22><><EFBFBD><EFBFBD>" |
3.1.6 5 continuation bytes: "<22><><EFBFBD><EFBFBD><EFBFBD>" |
3.1.7 6 continuation bytes: "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
3.1.8 7 continuation bytes: "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf): |
|
"<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> |
<20><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> |
<20><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD> |
<20><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
3.2 Lonely start characters |
|
3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf), |
each followed by a space character: |
|
"<22> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> |
<20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> " |
|
3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef), |
each followed by a space character: |
|
"<22> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> <20> " |
|
3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7), |
each followed by a space character: |
|
"<22> <20> <20> <20> <20> <20> <20> <20> " |
|
3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb), |
each followed by a space character: |
|
"<22> <20> <20> <20> " |
|
3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd), |
each followed by a space character: |
|
"<22> <20> " |
|
3.3 Sequences with last continuation byte missing |
|
All bytes of an incomplete sequence should be signalled as a single |
malformed sequence, i.e., you should see only a single replacement |
characters in each of the next 10 tests. (Characters as in section 2) |
|
3.3.1 2-byte sequence with last byte missing (U+0000): "<22>" |
3.3.2 3-byte sequence with last byte missing (U+0000): "<22><>" |
3.3.3 4-byte sequence with last byte missing (U+0000): "<22><><EFBFBD>" |
3.3.4 5-byte sequence with last byte missing (U+0000): "<22><><EFBFBD><EFBFBD>" |
3.3.5 6-byte sequence with last byte missing (U+0000): "<22><><EFBFBD><EFBFBD><EFBFBD>" |
3.3.6 2-byte sequence with last byte missing (U-000007ff): "<22>" |
3.3.7 3-byte sequence with last byte missing (U-0000ffff): "<22><>" |
3.3.8 4-byte sequence with last byte missing (U-001fffff): "<22><><EFBFBD>" |
3.3.9 5-byte sequence with last byte missing (U-03ffffff): "<22><><EFBFBD><EFBFBD>" |
3.3.10 6-byte sequence with last byte missing (U-7fffffff): "<22><><EFBFBD><EFBFBD><EFBFBD>" |
|
3.4 Concatenation of incomplete sequences |
|
All the 10 sequences of 3.3 concatenated, you should see 10 malformed |
sequences being signalled: |
|
"<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
3.5 Impossible bytes |
|
The following two bytes cannot appear in a correct UTF-8 string |
|
3.5.1 fe = "<22>" |
3.5.2 ff = "<22>" |
3.5.3 fe fe ff ff = "<22><><EFBFBD><EFBFBD>" |
|
4 Overlong sequences |
|
The following sequences are not malformed according to the letter of |
the Unicode 2.0 standard. However, they are longer then necessary and |
a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8 |
decoder" should reject them just like malformed sequences for two |
reasons: (1) It helps to debug applications if overlong sequences are |
not treated as valid representations of characters, because this helps |
to spot problems more quickly. (2) Overlong sequences provide |
alternative representations of characters, that could maliciously be |
used to bypass filters that check only for ASCII characters. For |
instance, a 2-byte encoded line feed (LF) would not be caught by a |
line counter that counts only 0x0a bytes, but it would still be |
processed as a line feed by an unsafe UTF-8 decoder later in the |
pipeline. From a security point of view, ASCII compatibility of UTF-8 |
sequences means also, that ASCII characters are *only* allowed to be |
represented by ASCII bytes in the range 0x00-0x7f. To ensure this |
aspect of ASCII compatibility, use only "safe UTF-8 decoders" that |
reject overlong UTF-8 sequences for which a shorter encoding exists. |
|
4.1 Examples of an overlong ASCII character |
|
With a safe UTF-8 decoder, all of the following five overlong |
representations of the ASCII character slash ("/") should be rejected |
like a malformed UTF-8 sequence, for instance by substituting it with |
a replacement character. If you see a slash below, you do not have a |
safe UTF-8 decoder! |
|
4.1.1 U+002f = c0 af = "<22><>" |
4.1.2 U+002f = e0 80 af = "<22><><EFBFBD>" |
4.1.3 U+002f = f0 80 80 af = "<22><><EFBFBD><EFBFBD>" |
4.1.4 U+002f = f8 80 80 80 af = "<22><><EFBFBD><EFBFBD><EFBFBD>" |
4.1.5 U+002f = fc 80 80 80 80 af = "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
4.2 Maximum overlong sequences |
|
Below you see the highest Unicode value that is still resulting in an |
overlong sequence if represented with the given number of bytes. This |
is a boundary test for safe UTF-8 decoders. All five characters should |
be rejected like malformed UTF-8 sequences. |
|
4.2.1 U-0000007f = c1 bf = "<22><>" |
4.2.2 U-000007ff = e0 9f bf = "<22><><EFBFBD>" |
4.2.3 U-0000ffff = f0 8f bf bf = "<22><><EFBFBD><EFBFBD>" |
4.2.4 U-001fffff = f8 87 bf bf bf = "<22><><EFBFBD><EFBFBD><EFBFBD>" |
4.2.5 U-03ffffff = fc 83 bf bf bf bf = "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
4.3 Overlong representation of the NUL character |
|
The following five sequences should also be rejected like malformed |
UTF-8 sequences and should not be treated like the ASCII NUL |
character. |
|
4.3.1 U+0000 = c0 80 = "<22><>" |
4.3.2 U+0000 = e0 80 80 = "<22><><EFBFBD>" |
4.3.3 U+0000 = f0 80 80 80 = "<22><><EFBFBD><EFBFBD>" |
4.3.4 U+0000 = f8 80 80 80 80 = "<22><><EFBFBD><EFBFBD><EFBFBD>" |
4.3.5 U+0000 = fc 80 80 80 80 80 = "<22><><EFBFBD><EFBFBD><EFBFBD><EFBFBD>" |
|
THE END |