Text 24295, 98 rader
Skriven 2012-01-04 15:01:57 av Peter Krefting (2:203/0.222)
  Kommentar till text 23986 av FidoNews Robot (2:2/2.0)
Ärende: A plea for UTF-8 in Fidonet (was: FidoNews 28:52 [02/05]: General Artic
===============================================================================
Den 2011-12-26 03:34:25 skrev FidoNews Robot <0@2.2.2>:

>                  A PLEA FOR UTF-8 IN FIDONET  Part 1
>                  By Michiel van der Vlist. 2:280/5555

Some notes (I've been working on Unicode and legacy encodings professionally
for the better part of the last decade):

> The "A" is "ASCII" stands for "American". So it is no surprise that as
> far as the letters go, once again it only covers the 26 letters found in
> American English. ASCII is much richer that all of its predecessors, it
> has many punctuation and special characters, 32 - now mostly obsolete -
> control codes and as a new feature, the distinction between upper and
> lower case.

Initially, it also *did* have some limited support for characters from other
languages -- by using the backspace control code, you could produce some
diacritics. For instance, an 'ö' could be produced by emitting "o", backspace,
'"'. This did work for paper-based terminals, but the first screen-based
terminals never really supported this, so the functionality was mostly lost
there. This is why ASCII was changed to include '^' and '_' (for underlining),
instead of the up-arrow and left-arrow of the original version (compare the
Commodore PET character set, which was based on the earlier version of ASCII).

> Anyway, at the end of the DOS era, there were dozens of code pages,
> covering the needs for hundreds of languages. One could write in
> German, Swedish, Russian and Greek without problems. Well, one could
> not write in Greek and Russian in the same article because on e could
> not change code pages in mid stream. But who wanted that?

Your history is missing out a bit on the MBCSes (multi-byte character sets)
used for Chinese, Japanese and Korean here.

> Enter Unicode.

Unicode does predate the web, IIRC, though.

> Unicode introduces the concept of The Universal Character Set. It is not
> a static entity, it is still growing. Presently there are over a million
> characters defined. While in the code page concept, character set and
> character encoding scheme are one and the same, in Unicode they are
> decoupled. There is ONE charceter set: the Universal Character Set.

You're mixing things a bit. Universal Character Set (UCS) is ISO-10646, not
Unicode. The two (ISO and the Unicode consortium) do work together to
coordinate their character sets, though, making sure they are always
compatible, so it is easy enough to mistake one for the other. The surrounding
documentation is different between the two, however.

> There are several encoding schemes that all have their merits.
>
> First there is UTF-7. Designed for stone age transport layers that are 7
> bits only.

UTF-7 is an encoding of Unicode that is not specified by Unicode itself. It was
devised to work around problems with seven-bit email links, but is not
officially "sanctioned" by Unicode (and it is a horrible encoding to work with,
trust me on that).

> Next there is UTF-8. This is an 8 byte multibyte encoding that takes one
> to six bytes to encode a character.

The five and six byte forms are not used in modern UTF-8, as Unicode has been
defined as ending at U+10FFFF. So UTF-8 is at most four bytes per characters.

> Next there is UTF-16. Not suitable for byte onrientated transport media
> that use NULL as a special character, but is is used internally by
> Windows from XP and up.

This is what originally was the encoding of Unicode. It is used (in its unnamed
form, or just called "Unicode") in all versions of Windows NT, from NT 3.1 and
up, all up to Windows 7. I don't remember which version that first supported
the UTF-16 features (i.e., the surrogate pairs), but I do believe that Windows
NT 4 had some support for that already. UTF-16 is two or four bytes per
character.

> And finally there is UTF-32.

Which is a nice internal representation if you're not concerned with memory.
Each Unicode character fits nicely in a 32-bit data unit. Of course, this
wastes several bits per character, as values over 0x10FFFF are not used, but it
makes working on the characters a breeze.

> The obvious choice for FidoNet is UTF-8.

Indeed.


Fortunately interest for ISO 2022, which is a lot older, has dwindled after
Unicode appeared. ISO 2022 can be used to mix characters from any character
sets, using a stateful encoding. But writing support for that in software is
not fun (trust me, I know this from experience).


\\// Peter

--- Opera Mail/11.60 (Linux)
 * Origin: Softwolves Software @ Oslo, Norway (2:203/0.222)