Text 24213, 207 rader
Skriven 2012-01-02 01:47:53 av FidoNews Robot (2:2/2.0)
Ärende: FidoNews 29:01 [02/05]: General Articles
================================================
=================================================================
                        GENERAL ARTICLES
=================================================================

                        A PLEA FOR UTF-8 IN FIODONET   Part 2
                        By Michiel van der Vlist, 2:280/5555

Last week I discussed the various ways to have a computer deal with
more than just the ASCII character set. Now let us look at some of the
technical details regarding what the WWW has adopted as the
preferential encoding scheme and what I think FidoNet should evolve
into as well: UTF-8.

As mentioned last week, Unicode is based on The Universal Character
Set. Characters in the Universal Character Set are identified by their
Code Point. This is a simple scalar unsigned integer value. The usual
way of representing it is in the form U+wxyz, where wxyz is a four,
five or six digit hexadecimal value. So the code point for the Roman
letter 'A' is U+0041. The code point for the Cyrillic capital 'A' is
U+0410, and the code point for the Euro sign is U+20AC.

Table 1 tells us how to transform the value of the code point into a
variable length byte stream of one to four bytes. In theory the
transformation covers up to six bytes, encoding a 31 bit integer, but
for now it stops at four bytes, coding for a 21 bit integer.

     Table 1.  UTF-7 bit distibution

----------------------------------------------------------------------
Scalar value               | 1st Byte | 2nd Byte | 3rd Byte | 4th Byte
----------------------------------------------------------------------
00000000 0xxxxxxx          | 0xxxxxxx |          |          |
----------------------------------------------------------------------
00000yyy yyxxxxxx          | 110yyyyy | 10xxxxxx |          |
----------------------------------------------------------------------
zzzzyyyy yyxxxxxx          | 1110zzzz | 10yyyyyy | 10xxxxxx |
----------------------------------------------------------------------
000uuuuu zzzzyyyy yyxxxxxx | 11110uuu | 10uuzzzz | 10yyyyyy | 10xxxxxx
----------------------------------------------------------------------

As you can see, the transformation algorithm is relatively
straightforward. Code points U+0000 through u+007f are represented by
a single byte. The first 128 characters of the Universal Character Set
coincide with the ASCII set and the coding is the same.

Code points U+0080 to U+07FF are coded into two bytes, Code points
U+0800 - U+FFFF into three bytes and above that into four bytes.

For the multibyte sequences the number of consecutive '1's of the most
significant bits in the first byte tells us the length of the
sequence. The first byte always has the two most significant bits up.
Subsequent bytes have the two most significant bits set to '10'. This
makes it relatively easy keep track of the number of characters that
are represented by a byte stream and also easy to resync when
synchronisation is lost due to temporary loss of bits.

For example when one needs to know the length of a UTF-8 string, not
in terms of the number of bytes, but in terms of the number of
characters that will appear on the screen, it suffices to just count
the bytes with the most significant bit set to 0 plus the bytes that
have the two most significant bits set to 1.

int Str8len(unsigned char *s)
{
        int i=0, j=0;
        unsigned char c;

        while (s[j] && (j<=MAXINT))
        {
               c=s[j++];
               if ((c & 0x80)==0) || (c & 0xC0)==0xC0)) i++;
        }
        return (i);
}

This may be useful for screen wrapping in a message editor.

So what else would we need to make FidoNet work with UTF-8? As far as
the transport layer is concerned nothing really. FidoNet is fully 8
bit transparent except for the NULL as the terminating character for
strings. There is no conflict as in UTF-8 the NULL character has the
exact same meaning as in ASCII. Oh wait, there is this tiny little
snag: the archaic soft return. In their infinite wisdom, the founding
fathers decided that the character 0x8D had special meaning; that of
soft return. Probably a remnant from the Wordstar days. In hindsight
totally superfluous and a conflict with many code page schemes that
treat it as a printable character. It also conflicts with UTF-8. 0x8D
is a valid byte in a well formed UTF-8 string. Fortunately most bronze
age software allows configuring 0x8D as a printable character instead
of soft return, so this should no longer be a problem. Be sure however
to configure your tosser to not strip soft returns.

So the transport layer - mailers and tossers - are not a problem. But
that is where it ends. AFAIK there is no dedicated FidoNet message
reader that can properly handle UTF-8.

There is a solution: JAMNNTP. A programme by Jon Billings that
converts a JAM message base to something that can be read by a
standard News Reader. Thunderbird, InterNet Explorer, Firefox, you
name it. All News readers, except some from the previous millenium
support UTF-8, so there you go...

Now about FidoNet readers. It is not the OS or the console that is the
problem. Not from Windows NT/XP on anyway. The Window XP console
supports unicode and UTF-8 encoding. It even has a pseudo code page
for it. Code page 65001.

Be sure however to set the font for the cmd window to "Lucida
Console". None of what comes next will work otherwise.

Consider this simple C programme:

=== Cut ======= Begin HWT8.C  ====


#include <stdio.h>

int main(void)
{
   printf("Hello World\n");
   printf("Hall%c%c Bj%c%crn\n",0xC3,0xA5,0xC3,0xB6);
   printf("%c%c%c%c%c%c%c%c%c%c%c%c %c%c%c%c%c%c%c%c%c%c%c%c\n",
       0xD0,0x9F,0xD1,0x80,0xD0,0xB8,0xD0,0xB2,0xD0,0xB5,0xD1,0x82,
       0xD1,0x82,0xD1,0x80,0xD0,0xBE,0xD0,0xB9,0xD0,0xBA,0xD0,0xB0);
   printf("Saluton %c%c%c\n",0xE2,0x82,0xAC);
   return(0);
}

=== Cut ===

Compile it and run it in a cmd window. Or if you do not have a
compiler, download http://www.vlist.eu/downloads/hwt8.bin and rename
it to hwt8.exe.

You will see the line "Hello world" followed by some lines of garbage.
It is a DOS programma, so this works in any DOS or Windows command
window.

But in XP or up we can do more. First do a:

chcp 65001

 ... and run it again. You may se some garbage on the screen the first
time, I haven't yet figured out why, but if you run it a second time,
you will see four lines of text, the first one being our famous "Hello
world". The next three lines will show something in Swedish, something
in Russian en something in Esperanto. And lo and behold: Cyrillic and
Swedish characters on the same screen. Plus a Euro sign as a bonus.

You can also divert the output of hwt8 to a text file with the
redirect command:

hwt8 >hwt8.utf

and then display it with the type command:

type hwt8.utf

So the limitation is not in the OS or the cmd console. They can handle
utf-8.

Unfortuntely the trick of changing the code page to 65001 does not
work with any of the FidoNet readers/editors that I have tried. The
InterMail editor and Msged just go in a flat spin when run under cp
65001. Golded runs without problems, but for some reason its output
does not go into utf-8 mode. It keeps displaying a message containing
utf-8 as if the local code page was set to 850. Even if one disables
all character translation. The odd thing however is that if one writes
the message containing garbage to a file with the Alt-W command, and
subsequently leaves golded and  displays the file with the type
command, then one gets the proper characters on the screen. So the
problem is in how Golded handles its output.

Well, enough for this week. Obviously some more reseacrh in the inner
workings of Golded and possibly other readers is required to get this
to work. This may be the subject of some future article. Or not. Time
will tell and no promises.


References.

Jukka Korpela, A tutorial on character code issues.
http://www.cs.tut.fi/~jkorpela/chars.html

Roman Czyborra, czyborra.com
http://czyborra.com/

Roman Czyborra, Der Globalzeichensatz Unicode im Betriebssystem Unix
http://www.unicodecharacter.com/

Tom Jennings, An annotated history of some character codes or ASCII:
American Standard Code for Information Infiltration
http://wps.com/projects/codes/index.html

The Unicode Consortium, Official unicode website.
http://www.unicode.org/



© Michiel van der Vlist, all rights reserved.
Permission to publish in the FIDONEWS file echo and the FIDONEWS
discussion echo as originating from 2:2/2

-----------------------------------------------------------------

--- Azure/NewsPrep 3.0
 * Origin: Home of the Fidonews (2:2/2.0)