Re: lynx-dev Quick Question on multiple URLs

[Klaus Weide]

On Mon, 28 Jun 1999, MR SCOTT   FENTON wrote:

> Is there an easy way to submit numerous URLs to lynx at once?  I have a
> routine that executes multiple lynx sessions, one with -auth, -dump, and
> -cookies for each of 400 URLs.  The URLs only vary by three characters, and
> the output is only a couple dozen lines.   But the time required to launch
> lynx, send the URL, process the authorization and cookie options, dump the
> output, and exit on an old Sparc is considerable.  Currently, it takes a
> little over an hour with the Sparc doing nothing but.

Some suggestions for reducing startup time:

- Don't have unnecessary entries in mailcap and mime.types files.
  Better yet, since you probably don't need any of those mappings for
  -dump, set those files to /dev/null in lynx.cfg.

- Use a reduced lynx.cfg, with only a few lines for those options that
  have to differe from compiled-in defaults.

- Don't use a .lynxrc file for personal settings (remove it from the
  batch routine user's home directory if it exists).

- Use a cookie file only if really needed; make sure it doesn't have
  lots of old cookies with domains that don't apply to you batch
  anyway.

- Some tricks may be possible for avoiding the authentication round-trip.
  For example if all requests are to the same server and should have the
  same authentication info (for the same realm?), you could probably
  use junkbuster or a similar proxy to add this as a fixed header to
  each request.  (Use at your own risk!)
  
- If possible, avoid URLs that result in an initial redirection.
  (For "directories" on the https server, use
  <http://example.com/directory/> instead of
  <http://example.com/directory>.

But as someone else mentioned, other tools like wget may be more efficient
anyway.

   Klaus