[Pan-users] Re: Disk dowload slowdown

[Duncan]

fred <address@hidden> posted
address@hidden, excerpted below, on  Thu, 12 Oct 2006
12:33:42 -0500:

> I have noticed that if I am running something disk intensive(working 
> with 5gb files) while pan is running,  pan will stop/slow down while 
> waiting for the disk (from 600 KiB/s down to 200 KiB/s). With a dual 
> processor system I leave pan downloading while I am working, so the slow 
> down is not an uncommon event. Is there an easy way to have pan lazy 
> write (or something)?

It depends on the nature of your system.  If you are using the correct
chipset drivers for your hardware and have DMA enabled on the hard drives,
you shouldn't see much of this.  If you are using the
generic/compatibility/fallback drivers and/or have DMA disabled, disk
access is not only far more CPU intensive, but also often stops whatever
else is going on until the I/O is finished.  Note that if you have
multiple drivers loaded, as I all too often see even where not all drivers
match hardware on the system (one guy I saw recently had drivers for I
think it was eight different kinds of SATA chipsets loaded, plus a couple
PATA chipsets, all three kinds of USB plus firewire, several SCSI and
firmware RAID drivers... the USB-mass-storage drivers, etc, it's a bit
of a wonder the kernel could access the hardware at all without getting
confused!), it's quite possible the /wrong/ driver will grab the hardware
first and activate it in compatibility/no-DMA mode, so the /right/ driver
can't access it.

That's the most likely problem.

Even without that, apparently it's still possible to have problems
depending on what filesystems you use.  I use reiserfs here, to fairly
good effect also on a dual CPU (Opteron 242s) system, but a chance remark I
read the other day leads me to believe it's not so efficient in at least
one way -- it (apparently) still uses the legacy Big-Kernel-Lock which is
global, rather than the finer tuned locking on most of the rest of the
kernel including ext3.  The result would be serializing all file system
access globally even when the hardware and drivers are otherwise capable of
parallelizing operations.  I can't for sure vouch for this but it's quite
possible, given that Hans Reiser has vociferously fought any updates to
the reiserfs code but the simplest bugfixes, including routine updates to
keep it instep with the rest of the kernel.  His point has been that
reiserfs is the stable version -- all feature additions and changes should
be to the new/experimental version -- reiser4, but the kernel hackers have
tended to see it otherwise, as fighting routine updates that were already
happening in the rest of the kernel.  Anyway, the squabble has meant the
delay of reiser4, with all those updates and more, mainlining, as the
kernel folks have been rather stricter on the standards it must meet than
normal given that from their viewpoint, maintainership of the reiserfs
(reiser3) code was just dumped in their lap and even then changes
resisted.  Hans Reiser's recent legal issues (if you've been under a
rock the last few weeks, he's now under arrest, but hasn't yet been
formally charged, for murder of his wife, one hopes he's not guilty of
course but while there's limited evidence at this point, it doesn't look
good) certainly haven't helped.

Fortunately for me even if the reiserfs big-kernel-lock thing is the case
(and I don't know for sure, I've only read it in a chance comment that
may be wrong), BKL problems get worse with number of CPUs and aren't
terribly bad yet at only two, and I have both 8 gigs of memory (thus a huge
cache) and a 4-way SATA kernel-RAID array (RAID-1/mirrored for /boot,
RAID-6 so two-way plus two-way parity for the main system, RAID-0
four-way striped for speed but without redundancy for swap, caches and
other temp stuff), speeding physical disk thruput regardless.

Finally, there's also I/O scheduling and elevator mechanisms that can be
tweaked in the kernel, if desired.  If you are using the the CFQ
(Completely Fair Queueing) process scheduler, it has had I/O priorities
since 2.6.13 at least.  If these remain untweaked, they'll roughly
correlate to the CPU scheduling process priorities.  Preemption can also
be configured.  However, I've not gotten too far into that here, running
preemptable BKL, but only voluntary preemption (the middle option), and
I'm running the simple deadline I/O scheduler, not CFQ or anticipatory
(the algorithm of which isn't optimized for RAID).

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman