Re: [Pan-users] Astraweb Is Sending Bad Headers! -was: Articles Pan Can't Read

[Heinrich Mueller]

On Thu, 11 Aug 2011 13:12:20 -0700, Mark S Bilk wrote:

> When a newsreader requests article headers from a news server with an
> OVER (formerly XOVER) command, the server sends the headers of each
> article combined into a single line, with the contents of the various
> headers separated by tabs, in this order (quoting from Section 8.3.2 of
> RFC 3977 http://tools.ietf.org/html/rfc3977#section-8.3):
> 
>       "0" or article number Subject header content From header content
>       Date header content Message-ID header content References header
>       content :bytes metadata item :lines metadata item
> 
>    For all fields, the value is processed by first removing all CRLF
>    pairs (that is, undoing any folding and removing the terminating
>    CRLF) and then replacing each TAB with a single space.
> 
> Since tabs separate the various header content fields of the line the
> server sends to the newsreader, obviously those fields must not contain
> any tabs.  Otherwise the newsreader won't be able to count the tabs in
> order to distinguish and separate the fields, and thus reconstruct the
> headers.  That's why RFC 3977 says that tabs in any of the header
> contents must be replaced by spaces before those contents are combined
> and sent to the newsreader.
> 
> Unfortunately Astraweb doesn't replace those tabs with spaces.
> 
> Tabs occur most frequently in the References header, when the article is
> two or more followup levels below the original post. The Pan newsreader
> is unable to retrieve, or sometimes even see,
> articles from Astraweb whose References header is folded using tabs. 
> Most newsreaders don't post that way, but some do.
> 
> That's why Pan couldn't download from Astraweb three of the five test
> articles that I posted.  And why, from Astraweb,
> Pan can't see all the articles in deeply nested followup trees, and
> sometimes sees a tree broken into smaller ones.
> 
> This was all determined by capturing the stream of alt.test headers,
> from Astraweb and from Blocknews, using the wireshark network packet
> analyzer.  Blocknews formats the header records properly, removing tabs
> from the field values before combining them.
> 
> I'll send another trouble-ticket to Astraweb, but after their previous
> performance I don't know if they'll care.  It shouldn't be hard to
> insert code into Pan to interpret the incoming header stream defensively
> and determine when the References field ends,
> which is what KNode and other newsreaders must be doing, since they work
> OK with Astraweb.

I already spotted this some time ago and modified pan to handle this 
correctly. You could try to compile my branch and try again.

Cheers.



-- 
roses are 0xff0000, violets are 0x0000ff, all my base are belong to you...