Text 5296, 84 rader
Skriven 2012-02-14 20:07:50 av Andre Grueneberg (2:2411/525.0)
   Kommentar till text 5295 av Stas Degteff (2:5080/102.1)
Ärende: binkp checksum protected file transfer
==============================================
Hi Stas

Stas Degteff schrieb:

> AG> It does not guarantee that the data you put on the link is
> AG> the one you read from disk or you write to disk.
> This is not link problem. Please don't mix link entities and local
> entities. 

I never said that link layer issues are the only reason for having a checksum
in binkp.

> AG> Why would we have a checksum in TIC files or in archives?
> In the 7-level model OSI: binkp -  session layer, TIC/archive -
> presentation layer. Please don't mix these levels.

That's an interesting interpretation. Some people would say binkp is
application layer (at least according to the 4 layer TCP/IP layer model).
Anyway, OSI model is theory and hardly fits cleanly withs any real-life
protocol stack out there.

If we remained within your picture you'd have a reliable link, but an
unreliable session layer. Your session layer does not ensure that the data on
the way to the presentation layer (is this the FTS-0001 PKT being passed
between the tossers) is unaltered. :)

> >> It means that CRC32 on a file detects less errors than CRC16 on a
> >> TCP packet. You offer a monkey job :)
> AG> That's why I suggested to use hash functions (see my original mail).
> AG> And you forget about the argument of hardware/software issues between
> AG> disk and NIC.
> Between disk block and NIC transmitter data path like (I may forget
> some details):
> 1) disk block -> disk memory -> disk controller transmitter -> storage
> controller reciever -> storage controller memory -> main memory (one
> step: read data from disk), on each "->" data integrity is checked.

Hmmm ... I am not aware of really wide use of main memory ECC. Yes, it's
available in server processors, but not on usual desktop systems. (See
http://www.intel.com/support/processors/sb/CS-029913.htm#9) ... not speaking of
the usual PCs being equipped with ECC DIMMs.

Then we are passing through a few layers of software which stitches together a
few packets, e.g. file system fragmentation ... potentially with a multitude of
(unprotected) memory access.

If there were no potential issues in this path, why would Oracle/Sun be so
stupid and promote ZFS to overcome such issues? See
<http://en.wikipedia.org/wiki/ZFS#Data_Integrity> [Incl. citations therein]

> 2) prepare data to send (program work), make TCP packets and responses
> to waiting for transport layer, prepare network-layer data)

... without checksum so far (as reality breaks layer boundaries) ...

> 3) main memory -> NIC buffer -> NIC transmitter (step three: send
> network-layer data)

Again no ECC on the path between main memory and PCIe controller. Finally the
NIC calculates the TCP checksum.

> If you speaks about "issues between disk and NIC", You should describe
> all real actions of the processes in the hardware.

Well ... as you already see, you missed a few important steps. Any such effort
will be rather longish. All I need is one point of potential failure in the
line between getting data from disk and the place where the TCP checksum is
calculated.

With modern NICs the TCP checksum is only calculated in hardware (I know,
breaches OSI). In the usual system memory is not ECC protected. I.e. any data
in RAM is not to be trusted. That's all I need. 

> If disk/controller/NIC/cable is bad and hardware don't detects errors,
> then this hardware plase on a dustbin!

Come on ... until I realize, this one important Arcmail package is crap ...

Anyway ... all the argumentation is of not much use. What are your objections
to the extension of binkp (and eventually binkd)?

CU Andre               E-Mail: andre@grueneberg.de 
--- timEd/Linux 1.11.b5
 * Origin: Testing timed/Linux (2:2411/525)