Text 10, 120 rader
Skriven 2004-08-02 04:07:02 av Roy J. Tellason (1:270/615.0)
      Kommentar till text 9 av Mike Tripp (1:382/61)
Ärende: file size issues?
=========================
Mike Tripp wrote in a message to Roy J. Tellason:

 MT> Hello Roy!

 MT> 01 Aug 04 04:07, Roy J. Tellason wrote to Mike Tripp:

 RT> I'm seeing the same numbers in both cases there because I have no
 RT> settings in the area.

 MT> Lack of settings alone won't keep the numbers static.  If you 
 MT> manually kill one message and SQPACK again, Old should be larger
 MT> than New. If you manually edit one existing message to remove a few
 MT> lines from the message, Old should be larger than New.

Sure.

 RT> Right.  But to get to that point I had to *move* the remaining
 RT> messages I wanted to keep from one area to another.

 MT> Gotcha...and obviously the stats you're posting are for the new
 MT> version of the area and not the original hosed version.

Right.  That one's gone.  Old habit,  getting rid of stuff like that,  from
when disk space was a scarcer resource.

 MT> It is the hosed version that should've been shrinking as messages 
 MT> were moved out of it.

No,  the hosed version wouldn't have been shrinking as sqpack wouldn't touch
it...   Too bad my "pack.log" gets rewritten every day,  or I'd pull a line out
of that one to show it.

 MT> The new version was only growing to swallow the new messages being 
 MT> appended, so there's no reason for them to be occupying more than 
 MT> the actual number of bytes required to do so.

Yup.  And the way things looked it wasn't ever gonna get any smaller.  I'm
looking at something like 38M currently on that drive,  not so much that I need
to worry about an 8M file eating too much space but still...

 RT> All with today's date,  and a timestamp within the past hour when I
 RT> was reading it last.  What's that sqi file used for anyhow?

 MT> Those that've been into the source up to their elbows can probably
 MT> find plenty of holes with my analogy, but if you understand the
 MT> basics of how DOS manages files and space allocated on a FAT hard
 MT> drive: the .SQI is the FAT table for the files (messages) within
 MT> Squish's hard drive (.SQD).

Ok.

 MT> The .SQI basically has the pointers to where the header for each
 MT> message starts and how many blocks it occupies.  If you delete a
 MT> message, Squish just updates the .SQI file that references the
 MT> deleted data to remove the pointer to its header and updates it's
 MT> usage map to show that space as available for writing new data
 MT> again.  The size of the .SQD is not necessarily representative of
 MT> the actual number of bytes required by the currently retained
 MT> messages.  Squish will expand the .SQD if more space is needed to
 MT> accomodate a new retention peak, but it will not shrink it just
 MT> because fewer are required today than yesterday.

Makes sense.  This also explains some why TimED's undelete function seems to
grab *all* deleted messages,  I guess that was just simpler to code,  or
something.

 MT> Just as bad things can happen to a FAT if operations fail or are
 MT> interrupted by a system lockup or power failure, the same is true
 MT> for Squish and .SQI/.SQD manipulations.  SQFIX does for the
 MT> Squishbase what CHKDSK and/or Disk Doctor does for the integrity of
 MT> the FAT.  It looks to make sure that a message header actually
 MT> starts where each .SQI entry says it does and looks for space that
 MT> the .SQI says should be in use by a retained message but actually
 MT> isn't (akin to lost clusters and cross-linked files) and attempts
 MT> to either fix or remove .SQI entries that don't seem to match
 MT> reality found in the .SQD.

 MT> SQPACK is analagous to DEFRAG.  If you have specified parameters,
 MT> it first marks the messages that fail those parameters as deleted
 MT> and then "compresses" the .SQD data to the beginning of the .SQD
 MT> file and shrinks the .SQD file down to the actual size required to
 MT> store the actual bytes associated with those messages that are
 MT> still retained, rather than the size being some former worst-case
 MT> maximum that was required to store messages that met the SQSET
 MT> parameters at some point in the past since you last SQPACKed.

It also seems to write the results out to a new file,  so for example if a
single message area is larger than the available space on disk,  it will
*never* get packed down.  I have one such area at present,  the .sqd file is
something around 66 megs.  This was from an internet mailing list I was
carrying for a while,  back when I had that capability on the system,  and I
figured that I'd just delete stuff as I read 'em.  I somehow haven't found the
time to mess with that one lately,  and if I don't eventually I may end up just
deleting it.

 MT> So if you use no purge parameters, never manually delete/edit a
 MT> message in an area, and never have any problem that causes the .SQI
 MT> to lose synch with the actual contents of the .SQD, it will appear
 MT> that SQPACK never does anything...just as running DEFRAG wouldn't
 MT> accomplish anything for your hard drive if you only appended new
 MT> files one at a time and never deleted or modified an existing file
 MT> since your last DEFRAG.  If, however, you make any of those changes
 MT> or SQFIX frees a pointer to an existing message because of a data
 MT> integrity issue, it is possible/probable that there will be some
 MT> delta between the old/new bytes when you SQPACK next, even if you
 MT> haven't assigned specific purge criteria to the area.

There are only a few areas like that here,  most are set for specified time
frames,  typically 60 days.

And there do seem to be times when random errors creep in,  I don't know if
it's memory glitches,  power surges,  or what,  though for the most part things
are pretty quiet around here in the small hours when maintenance is done. 
Every so often I'll scan my pack.log file for "err" and find some,  like that
one mentioned above,  but mostly it won't find it,  and I'll figure things are
okay at that point as there's plenty of areas and they all get packed down each
night.

--- 
 * Origin: TANSTAAFL BBS 717-838-8539 (1:270/615)