Re: [Pan-users] Pan GUI config file edits

[Duncan]

Cipher Cipher posted on Sat, 16 Mar 2013 13:01:32 -0400 as excerpted:

> In our last exciting issue (Vol 122 I7) Duncan offered:
> 
>> However, there are a few pan settings that don't appear in the GUI for
>> various reasons, but are honored if they're changed in the config
>> files.
>> It's not necessary to patch and rebuild pan to change these, only to
>> edit the appropriate config file or set the appropriate environmental
>> variable.  I've posted the list here a number of times, so you can
>> check the list archives (maybe on gmane =:^) if you're interested, or
>> ask and I'll repost, as I believe it has been awhile.
> 
> OK, I'm asking.  If/When you get a chance this would be nice.
> As always, thanks for your help here...  :-)


1. Environmental variable: PAN_HOME: defaults to ~/.pan2/ if unset

Pan uses the directory set in this variable in the environment pan 
inherits when it starts as its data/config dir.  The default if the 
variable is unset is ~/.pan2/.

Note that this also allows you to setup multiple independent pan 
configurations.  To do this, setup a wrapper script for each independent 
pan "instance" you want to run, and set PAN_HOME appropriately for each 
one.  As an example, here I run three different pan instances, a text-
group instance (that's the one I use for this list and others, viewed as 
groups using pan itself, via gmane.org's list2news server), a binary 
group instance, and a test group instance, for browsing groups I don't 
want to subscribe or that someone mentions a bad post in or whatever, 
thus allowing me to clean up afterward without screwing up my normal 
subscribed groups in the other instances.  However, someone could as 
easily set one up for the kids, another for their TV show groups, another 
for their MP3 groups, another for their "adult" groups (not creating a 
menu item for that wrapper so they have to start it from the command 
line, say), etc.

Since each data dir is separate, I can set cache sizes, etc, separately.  
But I use the same scorefile and hotkey listing for each, using symlinks 
from each data dir to the "global" file.

If you'd like me to post my wrapper scripts, again, let me know, but this 
post's going to be long enough as it is, so I'll skip it here.


2. servers.xml: connection-limit key: per-server number of connections.

The GUI does have a setting for this, but the Good Netkeeping Seal of 
Approval (GNKSA, see the link at the pan homepage), which pan scores 100% 
on, allows a compliant client to configure a maximum of four connections 
per server, because back when GNKSA was devised, trying to make more than 
four connections to a server was considered abuse.

However, these days most servers set and enforce their own maximum 
allowed connections policy, and for paying subscribers, the number of 
allowed connections is now very commonly in the double-digits, 20, 30, 50 
connections.  Of course most folks reach their maximum pipe bandwidth 
well before they reach their maximum allowed connections, and it's often 
convenient to be able to connect with another client (or another instance 
of pan) running on the same or a different machine, so 20+ connections 
may be pushing it, but it's still reasonably likely a person may want to 
set more than four, say 8 or 10 or 12.

Thus pan has a compromise.  GNKSA says a compliant client can't allow 
more than four connections per server to be configured.  It does NOT say 
that a compliant client can't honor a higher setting if the user sets it 
manually.  Thus, that's how pan works: it checks the connections per 
server value against the GNKSA four connections cap only when setting it, 
NOT when loading the setting from an existing config.  So if your server 
allows 50 connections and you want to try it, edit servers.xml 
appropriately, and do so.

Of course to avoid problems, only edit pan's config files manually when 
that pan instance isn't running.  =:^)

Bonus hint:  Setting a server to zero connections (I believe this is 
possible in the GUI as well) disables it. Someone who often ran out of 
quota on one of his servers got tired of pan erroring out on that server 
all the time when he was over quota, and requested a way to disable it 
until the next month's quota kicked in, without entirely deleting it.  
Setting zero connections now does the trick. =:^)

(As another use of zero connections, I have my cache for my text group 
instance set to several gigs, with no-expire set on the servers, and thus 
have been able to keep several years worth of text-group archives.  When 
my ISP quit the news servers it used to provide a couple years ago, I 
simply set that server to zero connections, and have thus been able to 
keep an archive of all the ISP-specific groups I used to subscribe to, 
even tho the server hasn't existed for over two years, now. =:^)


3. servers.xml: expire-articles-n-days-old: header expiration

Again, the pan GUI allows some configuration here, but the settings there 
have deliberately been kept a bit simpler.  Editing the setting directly 
in the config file gives you more flexibility.  In the config file 
expiration is an arbitrary number of days, so if you want 10 days, set 10 
days.  You can't do that in the GUI.

(It's worth noting that if you archive many years' worth of posts for 
groups as I do for text groups, pan's load time increases DRAMATICALLY as 
it has to sort thru them all every time.  On a "spinning rust" aka "not 
SSD" drive, the cold-system-cache load time can be several minutes. If 
you periodically copy the retained cache to a new directory, preferably 
on a freshly formatted partition perhaps as part of your backup and 
restore-test routine, it'll have the effect of defragging the files for 
messages cached since the last time it was done, thus reducing the effect 
to some degree, but it can still take minutes.  Pan simply wasn't 
designed to function as a news archiver with a multi-gigabyte cache, the 
use to which I'm putting it.  There's a reason the default cache is 10 MB 
and the default expiration isn't "non-expiring".  I'm thinking of 
creating yet another pan instance, "archive", that would hold say 
everything from 2012 and earlier, and deleting it from my operating 
instance to speed things up, but I've not done so yet.  Meanwhile, I've 
had my text-instance pan set to load with my X/kde session for years, and 
basically never quit the text instance except to reboot or to edit a 
config file or something and immediately restart it, and I've 16 gigs of 
RAM on my main machine these days, so once I've warmed the cache after 
reboot, the files basically stay in memory the whole time until the next 
reboot, and pan will restart nearly instantly since everything's cached 
in RAM.)
4. servers.xml: rank: server priority

Same flexibility theme.  In the GUI there's only primary and backup 
priority levels.  In the config file, you can have 10 or 100 different 
levels if you have servers enough to fill them and want to.


5. servers.xml: newsrc: newsrc filename

If like me you occasionally hand-edit your newsrc files, but have several 
servers and get tired of trying to remember which one "newsrc-1" (or 
whatever the default was, I changed these a long time ago...) is, change 
the name here and rename the files to match!  FWIW I use newsrc.servername 
style names here, but servername.newsrc would fit the usual 
name.extension model better.

IIRC another user reported that it's also possible to put a full path 
here if you like, tho I've not tried that.  He used that to put the file 
on an nfs mount, where he could access it from several different 
machines, thus allowing him to keep track of read messages regardless of 
which machine he had read them on.

(If you try this, however, do be aware that while it will allow pan to 
track read messages between machines, without a few other files as well, 
the headers will only show up on the machine that actually fetched them.  
If you delete read messages anyway, that shouldn't be a problem, but if 
you keep them around, not having the headers synced as well may be a 
problem.)


6. preferences.xml: cache-size-megs: the cache size, default 10 MB

If your pan is new enough, this setting is available on the behavior tab 
in preferences, but that's a relatively new setting in the GUI.  Back 
when Charles was pan's lead developer, he drank enough of the gnome 
koolade to believe that pan's configuration was already way too complex 
and repeatedly tried to dumb it down, getting rid of settings he didn't 
think users needed, occasionally returning them if enough people 
complained.  (FWIW, this is the primary reason for the GUI option 
simplification above, as well.  But I'm a kde guy and a gentoo user and 
THRIVE on settings, but am equally not afraid to edit the config files, 
or even apply an occasional custom patch if necessary, thus my tracking 
this list of non-GUI options. =;^)  This option was one of those removed 
with the 0.90 rewrite into C++ and for many years after that, it was 
available only to those willing to edit their config files directly.

But Charles lost interest in news some years ago, and with that, pan as 
well, and Heinrich's now the most active pan developer, having committed 
most of the recent new features.  Apparently he finds the cache size 
setting as useful a config option as I do, and if anything, he goes 
overboard in the OTHER direction with pan settings now (never thought I'd 
say THAT, and I DID qualify it with "if anything", but there it is...), 
so the option's now available in the GUI in recent enough pan.

But for those running "stale^H^Hble" distros with ancient pan...


7. preferences.xml: header-pane-*-column-width: header pane columm widths

These can actually be changed in the pan GUI by dragging the dividers 
between the columns.  However, there's a bug in pan that for some of us 
occasionally triggers a "score column hogs the whole pane" syndrome.  I 
thought I was the only one seeing it for awhile, but then someone else 
posted about it when it hit them too.  That thread is a few months old 
now.  My best guess is that it's some kind of race condition where 
certain gtk resources aren't loaded in time when pan starts up, so the 
columns they'd normally fill get automatically zero-width-ed.

The first couple times it happened I screwed around in the GUI trying to 
fix it, but once the columns are zero width, the action and state columns 
(icons only, thus supporting my resource race condition above) are 
INCREDIBLY difficult to get to expand again, and I ended up turning them 
off in prefs, then back on, at which point they appeared at the right 
instead of the left, and... it was all just way too fiddly and 
frustrating to want to do it repeatedly!

So after thinking about it and verifying that the settings were indeed 
storied in preferences.xml, the next time it happened, I quit pan and 
restored that file from backup.

That worked better, but after it happened a couple more times over a few 
more weeks, I got tired of doing that manually as well, so I automated a 
solution!

Remember the wrapper scripts I mentioned in #1 above?  Well, since I have 
a kwin window rule to always force my main pan window to the same 
horizontal size (horizontally maximized to full HD 1920 px width), and 
the header pane is full width across the pan window, header pane width is 
always consistent.  I was able to take advantage of that to setup some 
column-width patches to be applied to preferences.xml in the wrapper 
script.  Now, any time they zero out, I simply quit and restart pan, via 
the wrapper script, and if any of these entries are zeroed out, the patch 
applies and resets them to normal width before pan is restarted by the 
wrapper.  Now if that bug appears, a simple pan restart fixes it! =:^)

Depending on your needs, particularly if you want pixel-precise 
positioning, it may be easier to set some of the other sizes/positions 
found here with direct file edits as well.


8. Score: scorefile

Pan's GUI works reasonably well for creating scores, but it's horrible at 
keeping the scorefile well organized an efficient to load.  Additionally, 
people already familiar with standard regex (regular expressions) may 
well find directly editing the scorefile easier for more complex scoring, 
in any case.

As with a number of other features where pan borrowed existing file 
formats, pan uses the base slrn scorefile format.  The slrn scorefile 
documentation can be found here:

http://www.slrn.org/docs/score.txt

Do be aware, however, that pan's format isn't quite as advanced as 
documented there.  AFAIK, pan doesn't handle either {} grouping or 
include directives, for instance.

Perhaps the most important point of difference is that pan's scorefile 
processing is always case insensitive, so there's no need to do
[Ss][Ee][Xx] type regex, for instance.

FWIW, xnews also uses a very similar (but not identical) scorefile 
format.  That's where the case-insensitive idea came from, I believe, but 
pan doesn't incorporate most of the other xnews differences.  Here's the 
link for that documentation, tho, in case you find it interesting:

http://xnews.remarqs.net/scoring.txt

Also, I'm almost positive it worked at one point, but someone posted a 
few weeks ago, and I tested and confirmed the bug here, that pan no 
longer appears to work with AND conditions (single colon following 
Score), only OR (normally double colon).  Presently, either single or 
double colon both appear to function as OR.

Documented by slrn but worth emphasizing, % as the first (non-blank-
space) character of a line indicates a comment.  Thus, one of the first 
things you'll notice upon opening pan's scorefile in an editor is HOW 
EXTREMELY VERBOSE pan tends to be with its comments -- all those BOS/EOS 
lines are just that, comments that are 100% safe to delete without 
changing the functionality at all.  FWIW, generally the only pan-
generated comments I keep are the %Score created... lines, and those only 
for /expiring/ scores, since I find it useful to know when an expiring 
score was created as well as when it expires, and thus its effective 
duration.  However, I add my own comments as I find useful.

Meanwhile, as I said, pan's GUI scorefile entry ends up being horribly 
inefficient for processing, and it never deletes expired scores, either, 
it simply ignores them.  Take a look at the first few lines of the pan 
event log from when pan starts up, for instance.  Here, I have hundreds 
of individual conditions, which had I added them from pan without further 
editing would be hundreds of rules and sections, but pan reports only:

Read 8 scoring rules in 3 sections from ...

(FWIW, there's also a couple expired scores mentioned, but I don't have 
any expiring but not yet expired entries, and I deliberately keep a 
couple expired ones around to remind me how they look in case I want to 
add one manually at some point.  I've deleted all the other expired 
entries.)

As the documentation explains, a "section" begins with a group line, 
enclosed in [].  A "scoring rule" is delimited by a Score: line.  Thus, 
as hinted by pan's event log, the most efficient way to construct a 
scorefile has as few group lines aka sections as possible, with as many 
different scores as necessary under each group/section, but with each 
score appearing only once per section, with all the trigger conditions 
for that score appearing under it.  By contrast, pan's GUI adds a new 
section each time a new score is added, with just that single score and 
the single set of conditions added in the same scoring dialog.  Once you 
reach about a  hundred individual scoring conditions, that's inefficient 
for both pan and any human trying to make sense out of things.

Below is an (edited for posting) example from my scorefile, to show what 
a nicely organized one can look like.  Note the explanatory comments at 
the top, the section separator comments as well, and the scoreline 
comments (as specifically suggested in the documentation).  Additionally, 
note the final double-delimiter line below which anything I added from 
the PAN GUI would appear, until I actually edited the scorefile again to 
move those items elsewhere.  Finally, note that the majority of the  
conditions (only a small fraction of which I've shown) are listed under 
permanent unexpiring scoring rules and don't really need further 
comment.  The exceptions, which DO have a bit more comment in the form of 
the retained created date comment from pan, are the expiring (and now 
long expired) scores, but even those are placed in the appropriate 
preexisting group/section.

Cox is my ISP, it used the cox.* hierarchy, which as I mentioned above, I 
still have archived altho they stopped their news service a couple years 
ago.  As can be implied from the example, I used pan's score categories 
and the convenient fact that pan lets you configure colors for them to 
color-coding based on author within these groups.  Ignored was ignored (I 
generally deleted these, if I'd had pan's relatively new action feature 
back then I would have set them to delete automatically), and negative-
scored was negative-scored (would have been auto-marked-read), but other 
than that, I read, and saved, everything (so I would have auto-cached 
everything scored zero and above, had actions been available back then), 
so the positive score-zones were available for color-coding use, on the 
cox.* hierarchy especially.

!!! WARNING: some of the conditions are a bit... explicit!



% PAN scorefile
% Very close to SLRN's format at
% http://www.slrn.org/docs/score.txt
% but with case insensitivity (not other differences) from
% xnews at http://xnews.remarqs.net/scoring.txt

% [newsgroup.*] wildcard (not regex) format (~ negates).
% header lines regex. (~ negates).
% Score conditions, single : and, double :: or.
% Expires: immed. below score if present.
% Leading % indicates comment
% Leading whitespace and blank lines ignored.
% Regex and newsgroup matches case insensitive with
% keyword:, sensitive with keyword=.
% Newsgroup change delimits section,
% Score delimits "rule", multiple rules per section allowed.
% Comment after score becomes rule "name".

% Score levels: <=-9999 kill, -9998 to -1 low,
%               0, 1 - 4999 med, 5000 - 9998 high, >=9999 watch

%#########################################################################
%#########################################################################

[alt.*]
Score:: =-9999 %Alt kill
        From: sex coed
        From: NudeGirls
        From: voyeur only
        From: amateur
        From: SEXmag

        Subject: adult movies
        Subject: dupped
        Subject: ^\([-0-9/]*\)
        Subject: Use critical pack from Microsoft Corporation
        Subject: R/-\\PE
        Subject: R/-\|PE

%#########################################################################
%#########################################################################

[cox.*]
Score:: =9999 %Cox Watched (Cox employee)
        From: <address@hidden>$
        From: ^Jay Munsterman <address@hidden>$
        From: ^Steven Flynn <address@hidden>$
        From: CoxTech1
        From: ^David Knight <address@hidden>$
Score:: 100 %Cox Med
        From: C.M. Brannon
        From: ^Conrad J\. Sabatier <address@hidden>$
        From: ^Jim Rusling <address@hidden>$
Score:: 5000 %Cox Hi
        From: ^Lenroc <address@hidden>$
        From: ^Mag® 2º.. <address@hidden>
Score:: =-9999 %Cox Kill (repeat-kill)
        From: ^"John Smith" <address@hidden>$
        From: John Shocked
        From: You Got Punked\, Bitch
        From: address@hidden>

%#########################################################################

Score:: =-9999
        %Score created by Pan on Thu Apr 26 01:59:31 2007
        Expires: 10/29/2007
        From: ^"snake plisken" <address@hidden>$

        %Score created by Pan on Wed Jul 25 10:58:05 2007
        Expires: 1/27/2008
        From: ^common_ address@hidden

%#########################################################################
%#########################################################################


9. pan.hotkeys and accels.txt: hotkey config files

Until the relatively recent introduction of pan.hotkeys and the related 
tab in preferences, the accels.txt hotkey dump was the only way to assign 
certain hotkeys.  (The hover over a menu entry and hit the desired hotkey 
method doesn't/didn't work if that hotkey is assigned as a menu accel, as 
many alt-key modified keys are.   Frustratingly, these are/were often the 
most logical hotkey, too!)  The trouble was, pan did an entirely 
arbitrary-order dump of its hotkeys to the file at every exit (reading it 
at every startup), so one couldn't reorder it to something more human-
ordered in order to make it easier to actually find the function you were 
looking for.

What I did instead, and what people running stale pan versions without 
the new pan.hotkeys file and hotkeys tab in prefs may still need to do if 
they want to edit hotkeys today, is to copy the disordered accels.txt 
file elsewhere, then spend the hour or whatever laboriously sorting it 
(looking back with what I know now, I could have piped it to sort, 
but...).  Once it's sorted and saved, the desired changes can be made 
(and saved), and then the human-ordered file can be copied back over the 
pan-dumped file.  Assuming pan is not running at the time, when it next 
starts it'll load it, and when it exits it'll dump a disordered mess to 
it again, but that disordered mess will still have the customizations, 
and the user will still have the human-ordered copy they actually edited 
still stored elsewhere, to make further edits to if deemed necessary.

For the old accels.txt file, a leading ; indicated a comment, and all 
entries remaining at their defaults were commented in the disordered dump 
pan did at exit.

I did a fairly extensive hotkey remapping here, making the scheme easier 
to remember (all group functions were <modifiers>g, all thread functions 
were <modifiers>t, all article functions <modifiers>a, etc).  However, 
I've not kept up with it since the recent pan changes, and some of those 
hotkeys no longer work, so I should go back thru and redo it one of these 
days, probably using the pan.hotkeys file and/or the preferences tab, 
this time.  Someday...


10. newsrc and groups/* files, newsgroups.xov, group-preferences.xml

It can sometimes be worthwhile to edit these manually.  Newsrc files are 
how pan tracks the read state for messages, and over time, sequence gaps 
can add up.  Additionally, it's worth noting that pan keeps the record 
for all groups you've visited as well as for cross-posted messages in the 
groups they're cross-posted to, not just subscribed groups, so if you 
often browse unsubscribed groups or have unsubscribed from a number of 
groups you were subscribed to at one point, the article numbers read 
information is still available in this file.  That information might be a 
bit sensitive for some people, and in any case, having it sticking 
around, outdated, for groups you're no longer interested in, isn't very 
useful and just increases the size of the file.

There's a few other files that track related information -- the files in 
the groups subdir cache subject and author headers (among other data), 
for instance, and newsgroups.xov tracks total/unread counts as well as 
per-server highwater for visited groups.  Group-preferences.xml, 
meanwhile, sets a group entry for every group you visit, whether you 
change a preference for it or not.

It's partly because cleaning up these files is such a hassle that I have 
a separate pan test instance, where I can just blow away these files 
without having to worry about losing track of the info for groups 
(particularly for my archived groups from servers that aren't even there 
any more!) I'm actually subscribed to in my text and binary instances.


11. tasks.nzb: pan's saved tasks

There have been a couple of reported cases where this file got corrupted, 
and pan would crash/segfault when it tried to start.  Editing or removing 
the file allowed pan to startup normally.



That covers most of the files in PAN_HOME, actually.  Let's see, not 
covered yet are downloads.stats, which (other than an explanatory 
comment) contains a simple 64-bit integer that's the number of bytes 
downloaded since stats reset (for the download meter, a fairly new 
feature stale pan users likely won't have seen yet), newsgroups.dsc, the 
description list for newsgroups, as provided by the servers, 
newsgroups.ynm, posting allowed (y), read-only (n), or moderated (m), and 
posting.xml, posting prefs, but that corresponds pretty directly to 
what's in the GUI.

In terms of directories, there's article-cache, by default 10 MB but 
settable as discussed above, article-drafts, for draft messages, 
downloaded-attachments, AFAIK a legacy dir (possibly from an aborted git-
version-only experiment ?? the dates on mine are aug 23 2011 mtime and 
may 22 2012 ctime, likely when I last upgraded disks or restored to newly 
mkfsed partition from backup) that might not appear in new installations 
(?? I always keep a big cache download to cache first, then save from 
there, so maybe the dir is used for attachments when directly downloaded 
and saved ??), encode-cache, tmpdir used when encoding attachments for 
binposting, the groups subdir mentioned above, and ssl_certs, containing 
the stored server certs for use with the still relatively new secure 
connection support.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman