Re: [Pan-users] pan cmd line things

[Duncan]

markballard posted on Mon, 02 Sep 2013 15:47:14 -0400 as excerpted:

> I'm using linux (amd64/gentoo) and pan r0.139.

Greetings, fellow gentooer! =:^)

(~amd64 here, with layman and several overlays, and running a number of 
masked for testing prereleases and live-9999 ebuilds, including kde 4.11 
live-branch 4.11.49.9999 from the kde overlay, and pan-9999, my own ebuild 
based upon the (masked-as-live) pan-9999 ebuild available in the gentoo 
tree.)

> a couple things about cmd line/"--no-gui --verbose" for nzb files:
> 
> a)  I have primary and one fallback set in servers.xml but if I put an
> invalid port in for primary before starting pan (to test if cmd line
> mode handles multiple servers like gui apparently does) pan seems to
> ignore there's a backup and just sits there not d/l anything.

The standard way to temporarily disable a server is to set its 
connections to 0.  That disables it, without removing it entirely.

In theory, what pan does (at least in GUI mode, barring bugs) with an 
invalid port but non-zero connections set, is attempt connection to that 
server, but those tasks will remain in the task list, as it can't contact 
the server to execute them.  It SHOULD fallback to the other server, but 
the attempted connection to the first server will remain in the task list 
until pan is restarted, or until the task is manually removed.

Of course in no-gui mode, there's no GUI task list editing...

Meanwhile, there aren't a lot of users actually using no-gui mode, and 
last I knew, some bits of it were actually broken and their would-be 
entries in the pan --help output were commented out.  I remember reading 
the git-whatchanged entry for that commit when it happened.  But that was 
basically simply commenting out the --help entries for functionality that 
had been broken for awhile, and I don't think that has made it to an 
actual release yet, so I believe the broken no-gui functionality still 
appears in the help output for the latest release, which I believe still 
is the 0.139 version you say you're using.

If you happen to be a coder[1], I'm sure patches would be welcomed to fix 
the broken functionality, and even if not, having someone that's actually 
testing no-gui mode is definitely welcomed, since apparently that 
functionality had been broken for some time and nobody had actually 
caught and reported it.

But I'm not sure whether the no-gui functionality you were using was 
actually part of the known-broken bit that has been commented out, or 
not.  You'd have to unmask and try the -9999 version to be sure, and bug 
report based on it, since I know SOME of the no-gui breakage is already 
known and the options thus removed from the --help output, but I haven't 
followed it closely enough, not using that functionality myself, to be 
sure whether your bits are there, or not.

> b)  ctl-q shows in gui as how to close pan but I can't figure out how to
> properly close pan when using --no-gui ("q" nor ctl-q stop it, I have to
> ctl-c).

Yes.  ctrl-q is a very common X-based-app "quit" keyboard shortcut -- I 
believe the standard one for gtk/gnome apps.

But being primarily an X-app standard keyboard shortcut, it's not for non-
gui mode, which is actually designed to be run "headless", as part of a 
script invoked via cron job or perhaps from a context menu entry as an 
action associated with say the *.nzb filetype.

For that, no "quit" keyboard shortcut should be needed.  If you're 
testing the command at the commandline prior to scripting it and it 
hangs, that's what ctrl-c is for.  Otherwise, you'd use the standard kill/
killall/pkill/etc commands to send an appropriate signal, just as you 
would with any other CLI command that goes haywire.

> c)  I'm not finding a way to have pan not redownload part of a rar set
> that already exists.  for example I start an nzb d/l, have to stop pan,
> it has its tasks.nzb but it lists everything from the original nzb.  so
> to avoid "_copy" when I restart I have to edit those articles out of
> tasks.nzb (I could open queue and remove the existing ones there but in
> --no-gui that's not available; plus imm it's kind of clunky anyway
> having to manually tell pan in effect something was already d/l so don't
> d/l again).

Well, that's actually due to two factors.  The first is as already (sort-
of) stated, that pan's no-gui mode is designed to be invoked on-demand 
for a particular task, which it completes and then exits.  The assumption 
is that no-gui mode won't be terminated in the middle of a task and then 
restarted... people would use the GUI for that sort of thing.

The second has to do with how pan works with its message cache.  By 
default, the cache is only 10 MB, which is clearly smaller than many 
binaries these days... or even over a decade ago.

Story time! =:^)

I remember shortly after the turn of the century when I first started 
using pan after switching from MS and Outlook Express (I switched to 
Linux when MS did eXPrivacy, as that crossed a line I simply wasn't going 
to cross), and an ad-ware binary downloader of some sort I've long 
forgotten the name of now, how frustrated I was at how pan worked, 
because I expected to be able to download binaries to cache and then go 
thru them and save them off to permanent storage after they were all in 
local cache already.

As I had been accustomed to doing on the MS side, I setup pan to cache a 
bunch of binaries then went off to do something else (sleep or work, 
generally), expecting to come back to find them all downloaded and ready 
for me to go thru and save off locally.  But when I came back only about 
10% of the messages I'd told it to download were shown as cached!

So after some grumbling, particularly about the stuff that was no longer 
available as it had expired in the mean time, I set it up to do again, 
and WATCHED it this time.  Then I watched in amazement as pan proceeded 
to delete the messages it had JUST downloaded, without my even getting a 
chance to read them!

Of course what was actually happening was that pan, still set to a 10 MB 
cache, was deleting the oldest messages in ordered to make way for the 
new ones, all the while keeping the cache under 10 MB.

Needless to say, when I figured that out, I quickly upped the cache to 
the then maximum allowed 1 GB, and my first pan bug report (and possibly 
my first Linux bug report actually filed) was a request to up the cap to 
the 4 GB or so that was at that time the size of the dedicated partition 
I was using for pan cache.

I remember Charles Kerr (pan lead dev back then) deciding to up the cap 
to the then huge (considering the size of disks back then) size of 20 
gig, tho he kept the default 10 MB size.  But I was happy, as with the 
appropriate cache size set, I could then use the full partition I had 
dedicated for pan cache, and messages no longer disappeared from cache 
before I even had a chance to read them. =:^)  I don't believe there 
actually is a maximum cache size cap at all, today.  If you have the 
space, I /think/ a setting of 100 Tebibytes would work in terms of cache 
size, tho that's untested and my cache remains only in the tens of gigs 
today.

(Additionally, it's worth noting that pan currently stores its entire 
threading tree in memory, reconstructing that tree every time it starts, 
making either the read-in-time (minutes for a several gig cache on 
spinning rust, here, with a cold disk cache, tho switching to SSD lowered 
that to less than minute again) or the size of the tree in memory (a 
bottleneck people running 32-bit pan run into frequently these days, 
since AFAIK 32-bit apps are limited to 2 GiB memory usage with the 
default kernel settings, while that can be raised to 4 GiB with the 
appropriate kernel config, even that's too small on some groups) being 
the practical cap there, likely to be reached WELL before a 100 TiB pan 
message cache would fill up!)

As with your use of no-gui mode today, the problem back then was one of 
assumptions.  As a user, my assumption was that what I downloaded to 
cache would stay there until I deleted it.  But pan had been created with 
rather different assumptions about cache, assumptions that are still in 
the defaults today, altho thankfully, those defaults can be changed to 
better fit the way *THIS* particular user works. =:^)  

Pan's assumption (or more correctly, that of its devs) back then (and 
still by default) was that users would save files off directly instead of 
saving them to cache first, then browsing the local cache from within 
pan, as I was used to doing on the MS side.

But the good thing was that with a little tweak to the cache size and 
ultimately to the max cache size, my assumptions could be met as well.

Anyway, how that relates to your issue is...

If messages happen to still be cached, pan won't redownload them, since a 
cached file for that message-id already exists (pan uses the message-id 
as the filename, which works since they're supposed to be globally 
unique, making it simple to see whether a particular message is already 
cached or not).

Which means if you set your cache size high enough, having a task attempt 
to run over again, with at least part of the component messages already 
cached, will at least not actually redownload those messages.

But you'd still have the _copyN problem, since that's separate from 
whether the messages are actually cached or not.

Again, it's a matter of differing assumptions.  You're obviously making 
different assumptions about how no-gui mode is supposed to work, than the 
pan devs did.

But as with my case all those years ago, the tweaks necessary to support 
the differing assumptions should be pretty small.

The bottleneck there is actual developer time to implement the necessary 
patches.  Heinrich has been the developer that has done the most work 
with pan recently, including the commit that commented out the broken 
bits of no-gui mode from the help output.  But he hasn't had near the 
time recently to work on pan that he had for the six months to a year 
that he was so active, so not that much is happening ATM.

If you happen to be a coder, and DO have some time, that would be 
WONDERFUL, since then we'd actually have a coder working on pan that 
actually uses no-gui mode enough to give it the testing and support it 
has been lacking all these years, and pan could become a major gui-less 
news client as well, as befits a news client which has always had the 
goal of being the Pimp-Ass-Newsclient (thus the name PAN, which until the 
rewrite some years ago, was most properly always in ALL CAPS as it was 
and remains an acronym, tho of course the pimp-ass part is not so 
acceptable today, so "pan" it has become, with pretty much only the folks 
who have been around for awhile or who have read my remarks about it 
here, knowing to what "pan" actually refers).

> or maybe the trouble is I can't find a way to stop pan gracefully in cmd
> line mode before entire nzb d/l has finished?  ie if I could it would
> remove from tasks.nzb/queue what's already been d/l?

Assumptions... and lack of people actually using no-gui mode, again...

> otoh wrt nzb imm it would still be handy if pan removed anything in the
> queue/tasks,nzb when it sees the filename is already there.

As mentioned, at least it does that with cache, not re-downloading 
messages already cached.  But you won't see it in action much with the 
default 10 MB cache size. =:^(

Also, I'd guess it'd need to be made a commandline option.  Because in 
some cases filenames are very common (for instance, grabNNNN.jpg as a 
default name for stills grabbed from a video), and people might actually 
want the _copyN files where they're *NOT* duplicates.

Alternatively, make it download and save to a _copyN file by default, 
then do a size and perhaps checksum comparison, and delete the _copyN 
file if they're identical.  That's safe enough to be default behavior 
(particularly if checksum-checked), since I can't imagine people actually 
wanting the _copyN files if they really are dups -- in that case if they 
wanted a copy they'd do a local copy.

Then the commandline option could be used to turn on the rather less safe 
always overwrite mode, for people who were more concerned about disk 
space usage and a whole slew of _copyN files even if NOT identical, than 
they were about potentially losing an earlier non-duplicate download 
simply because it had the same name.

> d)  a minor thing is there's no "time/files remaining" kind of
> notification using "--no-gui --verbose".

As I said, no-gui mode is designed to be scripted and run headless, or 
for trouble-shooting, possibly from the CLI shell, and thus wouldn't 
normally need progress output.  However, were progress output available, 
--verbose would indeed be the logical method of enabling it.

Again, no-gui mode has really been quite neglected as few use it.  So 
someone actually using it to report bugs is definitely welcomed, but 
unless you're a coder that can provide patches as well, it might take 
awhile to get anything more than punt-and-comment-out-the-corresponding-
help-output fixes.

But if you happen to be a coder that can actually provide the patches 
too, well then, you're fixes should be even *MORE* welcomed! =:^)

In this specific case, I definitely see no reason why a reasonable no-gui 
mode progress output patch, with that output activated by the --verbose 
switch, wouldn't be welcomed.  (Of course the usual caveats about coding 
style and etc apply, but I guess coders should take that as a given.)

---
[1] Coder:  Unfortunately I'm not a coder, tho I know enough about it to 
sort of follow some patches and to occasionally do my own, as well as to 
follow the gist of many developer discussions.  I sometimes describe that 
as a "sysadmin level" of understanding, but it's basically the same level 
of understanding a typical gentooer might have, tho I've been on gentoo 
for a decade early next year as well, and I know my own understanding has 
continued to grow in that time, so I suppose I've a bit more experience 
and depth of understanding there too than the typical gentooer.

Meanwhile, my claim to any sort of authority here is simply due to the 
fact that I've been active on-list for over a decade now (and I believe 
it's fair to say, the MOST active in reasonable replies, for much of that 
time), and as such I function as sort of an institutional memory for both 
users and real developers, as I generally reply, as here, not only direct 
answers, but with more detailed explanations of why it works that way, as 
well.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman