Re: [Pan-users] Multiple Server support for PAN: Need a spec

[Frank Bruno]

One suggestion is to look at the database format for BNR which seems to handle
multiple servers quite well: (information from posts regarding BNR1 in the BNR
forum on Yahoo groups) Hopefully this might be useful.




            Here is some description about the cache file structure used
            by BNR
            in version 1.1.3 and later (internally called "cache file
            version 3").

            The file begins with a header record:

            THeaderRec = packed record
            FCount: Integer; // Total count of articles in cache
            FMinIDs: array[0..30] of Integer; // First available article
            nums
            FIDCounts: array[0..30] of Integer; // Number of articles in
            each list
            FHostLen: array[0..30] of Byte; // Server host name lengths
            FVersion: Byte; // Version number of the cached file
            end;

            (This description is only valid for files where FVersion = 3).

            After this header record there are a number of server names whose
            length is given by the FHostLen values above.

            After these the rest of the file contains an article record
followed by
            variable length of text data, followed by the next article record
etc.
            as follows:

            TArtRecV3 = packed record
            FIDs: array[0..30] of Integer; // ID numbers on various servers
            FBytes: Integer; // Size of this article (as text)
            FFileSize: Integer; // Size of the decoded file
            FFlags: Cardinal; // Flags of this article
            FTime: Integer; // Date and time of this article
            FPartSize: Integer; // FileSize of a single part (if MultiPart)
            FCurPart: Word; // Part number (if Multipart)
            FMaxPart: Word; // Number of parts (if MultiPart)
            FCurFile: Word; // File number
            FMaxFile: Word; // Number of files in the posted set
            FPriority: TDownloadPriorityType; // Downloading priority
            FMsgIDLen: Byte; // Length of FMessageID
            FSubLen: Byte; // Length of FSubject
            FNameLen: Byte; // Length of FFilename
            FFromLen: Byte; // Length of FFrom
            end;

            All Integers and Cardinals are 4 bytes. FTime is in DOSFileDateTime
format.
            TDownloadPriorityType = (dptNone, dptLow, dptNormal, dptHigh,
dptHigher,
            dptHighest);
            (in C this would be an enum and takes 1 byte). The last 4 values
above give
            the lengths
            of the following variable-length strings (they can be zero, in
which case
            see below).

            FFlags is a 32-bit vector as follows:

            TArtFlag = ( afRead, // This article has been marked read
            afDownloaded, // This article has been downloaded
            afMarked, // This article is marked for download
            afApproxSize, // We know the approximate file size
            afExactName, // We know the exact file name
            afBadSize, // Article has questionable size
            afExactSize, // We know the exact file size
            afMultiPart, // This article is part of a thread
            // Byte 1: Run-time flags, not saved in cache
            afGoodParts, // All the parts are available
            afBadParts, // Some parts are missing
            afWanted, // This article is Wanted
            afKilled, // This article has been Killed
            afEncodeUUE, // This article is encoded with UUE
            afEncodeMIME, // This article is encoded with MIME
            afExpired, // This article has been expired
            afNowDownloading, // This article is currently being downloaded
            // Byte 2: Extra flags and speedup flags
            afMarkedAlone, // This article has been marked alone
            afOtherServerWaiting, // Another server is waiting for this article
to
            get downloaded
            afHideIfCombined, // Same as (afGoodParts in Flags) and (CurPart >
0)
            and (CurPart < MaxPart)
            afHeaderOnly, // Download header only
            afBuildSubject, // Subject needs to be built before shown
            afHideIfOverview, // Hide this article from overview view mode
            afFree26,
            afFree27,
            // Byte 3: These flags give the file check existence bitmask
            afFree30,
            afEDBSize, // Same size file exists on RemDrvDB
            afESize, // Same size file exists on disk
            afEDBName, // Same name file exists on RemDrvDB
            afEName, // Same name file exists on disk
            afEDBBoth, // Same name&size file exists on RemDrvDB
            afEBoth, // Same name&size file exists on disk
            afReserved // THIS FLAG MUST NOT BE USED!
            );

            FSubLen, FNameLen and FFromLen can be zero, in which case this
            article either has empty strings or it uses the same subject, name
or
            filename as the previous article as follows:

            If afEName flag is set, this article uses the same FileName as the
            previous article in cache.
            If afEBoth flag is set, this article uses the same Subject as the
            previous article in cache.
            If afESize flag is set, this article uses the same From header as
            the previous article in cache.


A couple of things I forgot to mention:

            1) If the afBuildSubject flag is set, the subject in the cache file

            needs some formatting to get it into human-readable form. In that
            case the subject may contain the following tags:

            const
            FILENUMCHAR = #1;
            FILENOFCHAR = #2;
            FILENAMECHAR = #3;
            PARTNUMCHAR = #4;
            FILESIZECHAR = #5;
            FILESIZEKBCHAR = #6;

            (Here #1 is Delphi's way of saying ASCII code 1, '\001' in C)

            When you find those flags from the subject they are replaced by:
            FILENUMCHAR: FCurFile/FMaxFile
            FILENOFCHAR: FCurFile of FMaxFile
            FILENAMECHAR: FFileName
            PARTNUMCHAR: FCurPart/FMaxPart
            FILESIZECHAR: FFileSize
            FILESIZEKBCHAR: If FFileSize / 1024 is an integer, use it, else
            use FFileSize / 1024 formatted to show 2 decimals.

            2) FBytes value is not used in BNR versions 1.1.4 and later,
instead
            the Bytes/PartSize conversion uses the following formula:
            FBytes := Round(FPartSize * 62 / 45) + 500;




"J.B. Moreno" wrote:

> On 10/21/02 12:33 PM, Charles Kerr at <address@hidden> wrote:
>
> -snip-
> > Many servers don't handle retrieving bodies keyed by Message-ID;
> > using the group's index number for the article is more reliable.
>
> Article numbers are certainly more reliable, but do many people have a
> problem with MID?  The only circumstances under which that is likely to be a
> problem is when the history database isn't large enough -- and even then it
> is most likely only a problem with older articles that are about to expire
> (otherwise the size of the history is set way too low).
>
> --
> J.B. Moreno
>
> _______________________________________________
> Pan-users mailing list
> address@hidden
> http://mail.nongnu.org/mailman/listinfo/pan-users