Re: LYNX-DEV URL grabber for lynx

[David Woolley]

> I've just written a small C program to filter URLs from stdin, sort

Wouldn't it have been easier to borrow the routine from wget (Gnu sites)
that does this (assuming that you mean filter them from an HTML source)?
(You are allowed to do this providing the result is licensed only under
the GPL.)

> batch or at) to retrieve all found URLs using "lynx -dump" (or

For -dump, and with recursion allowed, Lynx already does this.

> However, I'm hesitating about publishing it in due to possible abuse.
> It is very easy to retrieve 1000 and more URLs at once, so before
> publishing I would like to hear your opinion.

wget has options to insert delays between each fetch to avoid overloading
the server and also respects robots.txt to avoid fetching private or
dynamic information.
;
; To UNSUBSCRIBE:  Send a mail message to address@hidden
;                  with "unsubscribe lynx-dev" (without the
;                  quotation marks) on a line by itself.
;