Re: [Pan-users] Problems downloading binaries

[Duncan]

Orlok Nosferatu posted on Tue, 12 Apr 2011 07:31:39 +0000 as excerpted:

> Hi,
> 
> In reply to the issues raised by Ron and Duncan some points:
> 1. Ron: Maybe you've got a permissions problem?
> I would not know how to check that.Inside Pan? Looking at Pan's
> Preferences I can only see things like 'Preferred Applications' -
> 'Use Gnome Preferences' (Web and Mail) and Text - Gedit.

Permissions would normally refer to the place you're downloading to, 
ensuring pan can save there.  You said knode had no problem, but if by 
chance it's running as a different user or saving to a different 
location...

Quota's would be in the same vein, and then there's SELinux policies, if 
you're running something like fedora that has them (I don't believe ubuntu 
does, tho), and they're active for pan.

> Duncan:
> 2. It might also be a full partition Although my partition is alive (it
> expands and extracts on the go)
> I still have 9GB, and seldom less then 4.

Expands and /contracts/?  Anyway, doesn't appear that should be an issue.

> 3. If you visit a pictures group, pan will normally display image
> attachments directly.  Does it continue to do so?
> The last image that was displayed was the one with date stamp 'Mar 12
> 2:41 PM'. After that I only get replies to posts.

There's some data missing in your reply, I think, as I can't quite make 
sense of it as-is.

If pan can download and display images, tho, you know the decoder is 
working in at least basic mode.

If you can find a multi-segment image post, you can test segment-combining 
as well.  Some of the larger images should certainly be multi-segment, as 
would be mp3 files (but you can't test them in-pan).

> 4. Can you still download to cache?
> I'm not sure on what you mean with this question.
> But I have increased the cache size to 20 MB in preferences.xml

"Downloading" binaries is normally a two-step process, actually 
downloading the raw articles into the cache, then combining and decoding 
them into a file that is then actually saved somewhere.

Download to cache means doing the first step only, downloading the raw 
articles and saving them in the cache, but not actually combining and 
decoding the attachments to save them.

The idea behind the question is to try to isolate the problem to either 
the first or second step.

The pan function for download to cache is in the articles menu, or the 
context-menu for an article in the header pane, cache article.  It does 
the first step only, downloading the raw article to cache, so the little 
disk icon shows up (if you have that column shown in the header pane), 
meaning it's available locally without actually downloading it again, but 
doesn't do the second step, decoding and saving of the attachment to disk.

With the messages already cached locally, saving (when it works) will be 
MUCH faster than the download part, a bit slower than copying a file from 
one place to another on the disk, but MUCH faster than waiting for the 
download, since the file will already be local, so really, you ARE just 
loading the data from disk to the CPU, decoding it, then saving it 
elsewhere, instead of having to wait for the slow download.

It is for this reason that when I do binaries, I use a huge cache (I have 
a 12-gig partition dedicated to that purpose, and use it all), download to 
cache, go do something else, and come back when it's done, after which I 
can sort thru and save the files MUCH faster than I could if I were 
waiting for them to download each time.

But that requires a MUCH larger cache than pan's default 10 MB cache, 
since the raw message files are upto about 35% larger than the binary 
attachments they carry, depending on the coding method used and how much 
message overhead there is.  (Old-style UUE and MIME/Base64 encoding have a 
33% overhead plus message headers and any text message, while newer yEnc 
style encoding is only 2-5% overhead, IDR the exact numbers.)

So my 12 gig cache will only contain between about 8 and 11 gigs of actual 
attachment payload, due to the encoding and header overhead, and depending 
on how much is yenc vs older encoding.

And pan's 10 MB cache would fill up  and start expiring old messages to 
make room for newer ones, before I'd even looked at them!  Actually, I had 
this problem when I first started using pan, since I had used download to 
cache and look at later with my previous news downloader as well.  Stuff 
was disappearing before I even had a chance to deal with it!  Then I 
figured out the cache was too small, increased it to several gigs, and the 
problem disappeared!

So what I was saying here was split the task into the download to cache 
bit, and the save bit, and see which bit fails.  But in ordered to do so, 
you'll need a bigger cache, at least 35% larger then the total final saved 
size of all the binaries you intend to download to cache, then save 
later.  A 20 MB cache will let you download to cache perhaps 12 MB of UUE 
or Base64 encoded files, perhaps 18 MB of yEnc encodes, without running 
out of room and starting to expire the messages.  (That's playing it safe 
and allowing a bit of extra for header overhead as well, and text 
messages.)

> Looking at the preferences, what does '<flag name='show-all-headers'
> value='false'/>' mean? Is this the answer to my problem?

No.  That's simply pan's record of whether you have the show-all-headers 
toggled on or off (false means off, the preference is under view, body, 
show all headers) for the body pane.  With it on, you'll see the subject, 
author, date, etc, plus a bunch of other headers not normally shown 
(references, organization, x-trace, etc, provided they're in the post to 
display, of course), all shown in the body pane.  With it off, the normal 
mode, those won't be shown in the body pane text area, only a few in the 
title area above it, and of course in the header pane.  Pan's normal hotkey 
to toggle that function is "h", so hitting that a few times while viewing 
a post should show you what it does to the body pane.  False is the way 
you'll probably want it, normally, unless you're checking something out in 
someone's headers.  (I sometimes use it to see where they posted from, or 
what they posted with, for instance.)

> 5 If you have access to nzb files, will the posts from them still
> download?
> I have no problem downloading by nzb file. That's why my partition is
> alive ;-P

If you can still download using pan via nzb file, pan can't be /too/ 
screwed up, as that indicates it can both download to cache, and decode 
and save the attachments.

Wait a minute...  pan's own tasks file is an nzb file, tasks.nzb in pan's 
data dir.  If nzb's are working, pan should be fine, but maybe it's own 
tasks.nzb got corrupted!

With pan shutdown, try deleting that file.  Then open pan again and see if 
that fixed the problem.  You'll lose any partially completed tasks that 
pan had saved, but if it solves your problem...

> 6 I'd strongly suspect that an incompatible or buggy library update is
> causing the problem.  Do you perhaps have a record of what system
> updates you might have done at the time, that could have triggered the
> problem?
> I do not keep a record of when what update was installed. But I usually
> install the minute I see updates are ready to be installed. And, FWIW -
> libgmime2.4, installed version 2.4.14-1+nmu1 (libgmime2.4-cil, -cil-dev,
> -2, and -dev)
> - libgmime-2.0.2a and -2.0.2-dev, installed version 2.2.22.5 A quick
> search did not give me the installation date.

FWIW, newer pan at least (0.134), uses gmime 2.4, not the older 2.0/2.2 
version.  So that's the one you're looking for.  I'm not sure which one 
0.133 used.

And FWIW, I have gmime 2.4.24 installed here.  That's somewhat beyond your 
2.4.14 version you list, but pan /should/ work with either, especially if 
compiled against it.

Depending on how your install works there, you may well be able to check 
the dates on the actual file, probably located in /usr/lib (or possibly 
/usr/lib64 on a 64-bit install, the file will be named something like 
libgmime-2.4.so.2.4.14, probably with a symlink libgmime-2.4.so in the 
same dir, pointing at the specifically named version), to see when it was 
installed.  Either the creation or modification dates will likely be the 
time installed.  See if either one of them appears to correspond to the 
time you lost downloading ability.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman