Re: [Sks-devel] sks not serving robots.txt?

[Jason Harris]

On Tue, Aug 31, 2004 at 04:45:34PM -0700, Jack Cummings wrote:
 
> Upon seeing this, I went to create a robots.txt file for the sks server,
> and noticed that I already had one. It just wasn't being served. 
> 
> In fact, if I try and get it, sks won't:
> 
> 2004-08-31 16:33:20 Get request: <ADDR_INET xx.xx.xx.xx:57064> => /robots.txt
> 2004-08-31 16:33:20 Page not found: /robots.txt

OK, as a dirty hack, you can add a rule to dbserver.ml around line 364
to let SKS recognize .txt files:

                 if extension = ".jpg" || extension = ".gif" || extension = 
".png" || extension = ".txt"
                 then
                   let base = base </> (1,0) in
                   let image = read_file ~binary:true (convert_web_fname base) i
n
                   cout#write_string image;
                   (match extension with
                      | ".jpg" -> "image/jpeg" | ".gif" -> "image/gif"
                      | ".png" -> "image/png"  | ".txt" -> "text/plain"
                      | _ -> raise (Wserver.Misc_error
                                      ("internal error: no mimetype " ^

If the crawlers need CR+LF endings, you can cut and paste the code just
above line 364 that handles index.html or add the carriage returns into
the robots.txt file itself.

The code can be made more generic to handle all .html and .txt files in
./web/ as well.

-- 
Jason Harris           |  NIC:  JH329, PGP:  This _is_ PGP-signed, isn't it?
address@hidden _|_ web:  http://keyserver.kjsl.com/~jharris/
          Got photons?   (TM), (C) 2004

pgp9JqQrz8xuk.pgp
Description: PGP signature