Re: [Pan-users] Kill files

[Duncan]

Dieter Britz posted on Mon, 24 Apr 2017 12:00:15 +0200 as excerpted:

> People talk about setting up a kill file for posters to news groups that
> annoy others, by off topic postings etc. Is it possible to do that with
> pan?

This repeats the same idea as the replies by HH, DG and Pedro in the 
other subthread, but with a bit more explanation of what pan's actually 
doing and why, and why it's like binary-choice killfiling (killfiled or 
not) but better. =:^)

First, let's understand the difference between a fine-grained scoring 
mechanism like pan has, where if desired the effects of many scoring 
rules can be applied together to arrive at a final score for a post, 
which then can be used to apply some action (like simply hiding the post, 
or marking it read, or deleting it, or on the other end, hilighting it 
with various colors depending on how high it scores, or automatically 
downloading the post to cache, or saving its attachments), vs a hard 
binary or trinary filter mechanism, which will act immediately on the 
first filter that applies to either kill (generally hide and mark-read, 
sometimes delete, depending on the implementation) or not, possibly (the 
trinary case) with the addition of a watch flag (and perhaps auto-
download depending on implementation) if the post isn't killed.

So in pan, a score of -9999 is defined as ignored.  That's what binary 
filters would filter out, also known as killing, thus the term killfile.

And a score of +9999 is defined as watched.

Meanwhile, FWIW, there's a number of other preset score category levels 
as well.  These can be seen under the view menu, header pane.  Here's the 
full listing, lowest to highest:

-9999 (or lower): Ignored

Either multiple scoring rules applied to result in the message being 
ignored, *OR* a single scoring rule set ignored/-9999 and stopped further 
processing of further scoring.

By default pan doesn't display these messages, but doesn't take any other 
action (marking them read, deleting them, etc).

-9998 to -1: Low

The result of one or more scoring rules lowered the message score into 
negative territory, but not enough to make it ignored.

0: Default

Of course 0 is the default score, if no scoring rules apply, or if the 
scoring rules exactly balance each other out.

1 to 4999: Medium

The result of one or more scoring rules was a moderate scoring boost, to 
less than 5000/high, however.

There's an option to display these in a different color, but I don't 
believe it's on by default.  (FWIW I've been running pan since 2002, a 
decade and a half now, and long ago forgot what the defaults were for 
many of the options I've customized.)

5000 to 9998: High

The result of one or more scoring rules was a higher scoring boost, more 
than 4999, but less than 9999.

Again, there's an option to display these in a different color, but I 
don't believe it's on by default.

9999 (or higher): Watched

Either multiple scoring rules resulted in a score at or above 9999, *OR* 
a single scoring rule set it to watched/9999 and stopped further scoring 
rule processing.

Pan should display these in a different color, by default I believe.  
There are options (off by default) that allow auto-downloading or the 
like.


As you should already see, scoring allows a far richer and more nuanced 
setup than arbitrary binary kill/show or trinary kill/show/watch 
filters.  But by using the watched/ignored options only, which basically 
set +9999/-9999 respectively and stop further score processing, you can 
have a simpler binary or trinary setup if you wish.

It's up to you. =:^)

Meanwhile, as I already mentioned, there are choices under view, header 
pane, to match (or not) each of these scoring categories separately.  
Again under view, header pane, pan can then be set to display either 
explicitly matched posts, matched posts and their subthreads, or matched 
posts and their entire threads, as desired.

It's up to you. =:^)

And in the preferences dialog (edit menu, preferences), on the colors 
tab, you can set the colors for each scoring category.

It's up to you. =:^)

(Tho do note that these days, pan only shows those colors in the score 
column, not the entire line as it used to do.  So you have to have the 
score column in your listing or you won't see the colors.  I preferred it 
coloring the entire line, but oh, well, I'm a user, not a dev... and 
unfortunately, that's NOT a user available option.  As I'm writing this, 
however, I'm wondering just how hard it might be to find that and patch 
it to whole line, tho.  I /am/ an advanced enough user that even tho I 
don't claim to be a dev, I can /sometimes/ work out patches on my own, 
and as I run gentoo, I normally build everything from sources and can and 
often do apply my own patches or those I've picked up from others to 
various packages, including pan.  So I'll have to look into patching 
this...)


OK, so you can set whether the various score categories are displayed or 
not, and if displayed, you can set the color per category, but what about 
more practical score-based actions?  In particular, for those who track 
things via marked-read, and who don't have pan's preference to 
automatically mark everything in the group read when they fetch headers 
or leave a group, not displaying ignored posts AND not having them 
automatically marked read is frustrating, because then they hang around, 
still marked unread!

Of course if you've been paying attention, you already know the answer, 
as I mentioned it above.

It is (of course) up to you! =:^)

(Noticing the trend yet? =:^)

Preferences dialog, actions tab.

One possible setup might be:

Delete articles scoring at:     -9999 or less   (ignored)

This would auto-delete ignored articles.

Mark articles read scoring at:  -9998 to -1     (low/negative)

This would auto-mark-read negative/low-scoring articles, but wouldn't 
delete them.  The idea here is to let you hide them by default (by 
showing only unread), but still keep them around in case you see a reply 
and you want to see the message it's replying to.

(I /believe/ it'll mark anything read UNDER the named category as well, 
so it would mark ignored articles read too, if they're not deleted with 
the earlier option, above.  But I'm not actually sure on this bit.)

Alternatively, if you don't delete ignored articles, you can simply mark 
them read, and still show negative/low-scoring articles that aren't 
entirely ignored.

Cache articles scoring at:      1 to 4999       (medium)

Of course you can set this to high/5000-9998 or watched/9999 instead, if 
that fits your needs better.

The idea is that if an article is sufficiently highly scored, you want it 
cached for you so it's already there when you would otherwise have to 
download it to cache.

Do be aware that pan's cache size is pretty small, 10 MB by default, and 
especially if you're doing binaries and using this setting, you'll 
probably want a larger cache.  That's set in preferences, on the behavior 
tab.

(Again, I /believe/ it'll do the same with the higher categories, high 
and watched, too, but I've not actually tested it to be sure.)

Download attachments of articles scoring at:    Disabled

If you're doing binaries, you might want to set this instead of the cache 
option.

Generally, people download binaries using one of two strategies.

Here, I prefer to have pan's cache set way big, and download messages to 
cache first, so they're local.  Then when they're already cached so I 
won't be waiting for the download, I can go thru and sort out what I 
really want, saving it where I want it, and deleting what I don't really 
want.  This works best for (relatively) small binaries that you will 
download many hundreds or thousands of, like still images or audio clips 
mostly under 10 minutes in length, with the occasional longer audio clip 
or short video.  It also requires a much larger cache setting (on the 
order of gigabytes, for me), or pan will start deleting previously 
downloaded to cache but still unread messages, to make room for the 
newest still downloading to cache messages.

For that binaries strategy or for text messages, the auto-download-to-
cache action exists.  Just be aware of the cache size requirements and 
adjust it accordingly.

The other strategy, which is obviously pan's default given the very small 
10 MB default cache size, is to have pan download and save off the 
binaries immediately, without caring at all about the messages they're 
attached to.  Because the attachments are saved immediately and the 
messages they were attached to don't matter, those messages can be 
deleted from cache as soon as the attachment is saved, so this requires a 
far smaller cache and pan's default 10 MB cache suffices.

This works best for very large binaries, typically half-hour or longer 
videos like TV series episodes or feature-length movies.  It works best 
if you don't care about the messages containing the attachments at all 
(no discussion of the series, etc), since unless you increase the size of 
the cache anyway, they'll be deleted effectively immediately after the 
attachment processing is completed.

It is for this binaries strategy that the auto-download-(and-save)-
attachments action exists.  Obviously this isn't going to work too well 
if your interest is primarily text groups (and people post binaries there 
too, and the messages score high enough for the action to trigger), 
because you'll end up with a bunch of random binaries that happened to be 
attached to watched or whatever level scoring messages saved off to 
wherever you have pan saving them.


OK, but what about the scoring itself?

First of all, the watch (thread) and ignore (thread or author) entries on 
the articles menu are the GUI method to create scoring rules that set the 
+/-9999 score and abort further score processing.

Next, there's the edit article's watch/ignore/score and add a scoring 
rule entries, again on the articles menu.  These bring up a dialog, 
either directly (for add) or indirectly (for edit, using the add button 
there), that lets you setup a more detailed scoring rule.  This is more 
flexible than the arbitrary watch/ignore options above, allowing you to 
match various options and if matched either set a specific score and 
abort further scoring as the above watch/ignore options do, or 
alternatively, to simply add/subtract whatever score and continue 
processing further scoring rules.  You can also set an expiry for the 
rule, if desired, or make it permanent.

It's this last option, to add/subtract some score value and continue 
processing more scoring rules, that's where the real flexibility comes 
in.  You can match on multiple subject keywords in multiple rules, adding 
or subtracting based on the match, then add/subtract based on author, 
then do some more based on references (effectively thread, only sometimes 
message-ids are deleted from the header and it won't match the thread any 
longer), then subtract points if it's cross-posted/spammed to too many 
groups, and add or subtract more points based on size in bytes or line 
count.

As long as no match sets an arbitrary score and stops further processing, 
all these matches will result in a final score that combines the effects 
and the relative scoring weight of all the others, and pan uses that 
final score to decide what scoring category the message belongs in, and 
thus whether to show it and how, as well as what automated actions to 
apply.

See how much richer a good scoring system is, compared to arbitrary 
binary/trinary-choice filtering on just ONE match-factor?

Of course if that's too complex for you, just use the watch/ignore and be 
done with it.

It's up to you. =:^)


Meanwhile, as the others suggested, the real advanced stuff is reserved 
for those who choose to directly edit the scorefile itself.  They posted 
the link to the format description.

http://www.slrn.org/docs/score.txt

But, keep in mind that the link above is for a different news client, 
slrn, which shares a general scorefile format with pan.  Unfortunately, 
however, pan's score-processing code isn't quite as advanced as slrn's, 
so some of the more complex stuff described there doesn't work in pan.  
Pan hasn't implemented the include statement, for instance, so don't try 
to use it.  The {} grouping logic isn't implemented either, AFAIK.

And, pan hasn't implemented the score keyword's single-colon AND logic, 
so single or double colon doesn't matter, it's always interpreted as OR 
(double-colon).  This is unfortunate, but the effect can be partially 
counteracted by simply creating multiple conditions, each of which gives 
partial points.  So instead of an AND score with five conditions to meet 
and a +1000 value, you can use pan's OR scoring on each of the five 
conditions, with a +200 value on each.  The total if all match will still 
be +1000, but of course the effect might be less anticipated if only some 
conditions match and that interacts with another would-be compound with 
only some conditions matching.

Another difference is that pan's scoring matches are always case 
insensitive.  So don't worry about John vs. JOHN vs. john vs. JoHN, the 
same regex will match them all without any fancy regex footwork.


Some additional scorefile format notes:

* Unfortunately for some, understanding regular expressions is really 
necessary to take full advantage of scoring, particularly when editing 
the scorefile itself, but it's worth it, and pan's GUI does allow simple 
scoring even if you don't know regex.

It's up to you. =:^)

* The note in section 1.1 recommending that one stick to the overview 
headers (typically subject/from/date/message-id/references/bytes/lines 
and often xref), but allowing others, most definitely applies.  
Unfortunately it's a technical limitation of the protocol, not something 
pan (or slrn or any other news client) can do anything about.

The thing is that pan can score headers in the overview without 
downloading the full message (or full headers).  For the most part, 
that's the headers needed to display the message in the headers pane, 
author, subject, date, etc, plus message-id and references for threading 
and tracking across multiple servers, etc.  But for the more exotic 
headers, pan won't get them, and thus can't score them, until the article 
is downloaded to cache.

So if you have an abuser that keeps nym-shifting and otherwise 
deliberately changing everything in the headers he has access to, in 
ordered to try to avoid killfiling, but who always posts thru a provider 
that adds an xtrace header with a consistent value you can score on, you 
*CAN* score on it, but you'll have to download the messages to cache 
first.

Take it from someone who was in the position of trying to killfile a 
poster like that at one point, before pan could score such non-overview 
headers, being able to ignore-score it, but only after downloading to 
cache, sucks, but it definitely sucks less than having to actually show 
the message in ordered to see who it is and block it!


* Note that while you can set an expiry on the score in the pan GUI, and 
at that point pan will indeed quit applying that score, it won't actually 
remove it from the scorefile.  The only way to actually remove the score 
from the scorefile is to manually edit it.

Unfortunately, this does mean that if you actively add expiring scoring 
rules and never manually remove them, eventually your scorefile will be 
cluttered with perhaps hundreds or thousands of expired rules and they'll 
begin to affect score-file loading performance as pan still has to 
process them at least far enough to see they're expired, and then how far 
to ignore until the beginning of the next possibly still valid rule.

So you'll probably want to either clear out the scorefile and start new 
occasionally, or manually edit it to at least clean out the expired rules 
from time to time, or simply don't use expiring scores, just living with 
it unless it's worth a permanent rule.

* Yes, an initial % on a line *DOES* mean it's a comment.

By implication, most of the lines pan adds when you add a score via the 
GUI are comments and don't matter for the actual scoring at all.  They're 
only there to aid human readers.

Of course that means you can edit or delete them as you wish, without 
affecting actual operation.

Here, I tend to delete pretty much all of pan's added comments, with the 
exception of the date added comments for expiring scores, since that way 
I can see how long I had set the expiry.

* If you do heavy scoring with lots of rules, using pan's GUI to set them 
up isn't particularly efficient for machine processing.  The example in 
the linked documentation is somewhat more efficient, but it's too short 
to really get the point across.  If you're planning to do a lot of manual 
scorefile editing or simply want to make your scorefile more efficient, 
either check past scoring threads for this list/group (the list is 
available as a newsgroup on news.gmane.org) where I've posted a longer 
example from my scorefile, or ask for such an example.

* Similarly, if you're not good with regular expressions and need some 
help designing a score that's more complex than you can easily do with 
the pan GUI, or if something's just not working as you expected it to, 
with scoring or something else, ask for help.  We've dealt with a number 
of such queries over the years. =:^)


OK, so hope that's of help.  Some people just want an answer to plug in 
without understanding it.  Others want to understand what's going on, so 
next time they want to do something similar but not identical, they can 
figure out how to do it themselves.  I'm certainly in this latter group, 
and my posts tend to go to the extreme in explaining things.  That 
frustrates the first group, but I've stacks of thanks from people who 
preferred the better understanding my explanatory if extremely verbose 
style gave them, and sometimes I get new insights or ideas (like possibly 
patching the score coloring to the whole line instead of just the score 
column, above) as I'm writing things down, and it's the combination of 
both of those that's my motivation to keep posting as I do. =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman