Text 21144, 103 rader
Skriven 2015-02-16 01:18:33 av FidoNews Robot (2:2/2.0)
Ärende: FidoNews 32:07 [02/07]: General Articles
================================================
=================================================================
                        GENERAL ARTICLES
=================================================================

                         UTF-8 In The Nodelist
                         By Michiel van der Vlist, 2:280/5555

This weekend an initiative was launched in Z2 to break the "ASCII only
barrier" for the nodelist. Ever since the first Fidonet nodelist was
issued in 1984, the character set for the nodelist has been the ASCII
set. The relevant FTSC document, FTS-5000.005.005 says this:

  The nodelist is a flat text file containing any number of lines,
  using only the ASCII (7 bit) character set.

In the early days of Fidonet, this was acceptable when Fidonet was
limited to Northern America and there was no other choice anyway
because at the time that was the only standardised character set that
all Fidonet systems had in common.

But already in 1989 when Fidonet had spread all over the world the
ASCII only barrier provoked attempts to break it. In the R22 segment
of nodelist.197 of 1989, we find lines like this:

,60,Night_Service,Espoo_Finland,Jali_H{kkil{,358-0-5471235,1200,V22,XA
,98,Vallu_Opus,Valkeala_Finland,Kimmo_Kotim{ki,358-51-863364,1200,V22
,304,Kummised{n_kyytipoika,Keitele_Finland,Urpo_Heikkil{,358-78-52663,
,308,Kiveri|_BBS,Lahti_Finland,Jukka_Sorjonen,358-18-182476,1200,CT3
,86,Dataline,Sipoo_Finland,Mikael_L|nnroth,358-0-234141,2400,V22,XB

What we see here is an early attempt to accomodate names written with
"funny" characters. At the time MakeNl did not allow 8 bit characters,
they were replaced by question marks, so if a different character set
was used, it had to be a seven bit set. The character set used is
ISO 646-FI. In this set the seldom used ASCII characters { \ } { | and
} are replaced by the upper and lower case a diaresis, o diaresis and
a ring. (ä ö å)

My nodelist archive is incomplete, so I was unable to look beyond 2004
In 2004 this was still in use in the Finnish part of the nodelist. It
illustrates that there is a demand for the ability to write the
sysop's name in his own language in his own alphabet.

Using national characters sets however is impractical to use an
understatement. The above method is not suitable for more general use.
If all regions or nets started using their own encoding, it would
become quit messy.

The latest version of MakeNl allows the use of non-ASCII characters by
using the option "Allow8bit 1". Using the 8th bit gives much more
freedom than substituting not much used characters by national
characters.

An eight bit character set however is not good enough either. Even
with the nodelist shrunk to a percentage of what it once was, there
is no single eight bit set that covers all of Fidonet. Net even "most
of Fidonet. Fortunately most modern operating systems now use
unicode internally, so dealing with UTF-8 encoding is not all that
hard.

In December a test was performed by adding a season's greeting to the
prolog of the daily nodelist as distributed in Z2. The encoding was
UTF-8 and it covered all the umlauts and accents as used in Western
Europe in addition to characters with macrons as used in Eastern
Europe as well as Cyrilics as used in Russia, Ukrania and Bulgaria
and more. This test was a success in that it broke nothing. Most
likely the majority of Fidonet sysops never saw it.

As a follow up ZC2 is now starting to distribute a daily nodelist in
UTF-8 format. It is not a replacement of the existing weekly nodelist
and nodelist diff, nor does it replace the daily nodelist in ASCII.
It is an additional service with a seperate distribution. The name
of the file is DAILYUTF.ddd where ddd is the day of the year.

Presently there are contributions in UTF-8 from R28 and R56.

The following paragraph has been coordinated with the ZC2:

RC's wishing to participate in the project are invited to do so. They
should:

1) Send a message to the ZC2 to inform him. The ZC2 needs to adjust
his configuration to process UTF-8 formatted segments from an RC.

2) Submit segments named UTFRrr.ddd  where rr is the region number and
ddd is the day of the year of the coming or present Friday or
UTFRrr.Cdd where C is the compression and dd are the last two digits
of the day of the year of the coming or present Friday.

3) The character encoding for the segment must be UTF-8.

4) Particpating in the project does not relieve RC's of the
oubligation to submit REGIONrr segments in ASCII. They must continue
to submit those segments in addition to the segments in UTF-8.


Next week: How to create and edit UTF-8 text files.


-----------------------------------------------------------------

--- Azure/NewsPrep 3.0
 * Origin: Home of the Fidonews (2:2/2.0)