Re: lynx-dev batch uploads of text files

[Klaus Weide]

On Thu, 8 Jul 1999, Joel F. Rising wrote:

> Folks,
> 
> We have a web-based class administration system to which we upload new
> classlists each day.  It would be nice to do this in batch.  Assuming I
> have the newest version of lynx, with one of the file upload capability
> patches installed, does anyone know what kind of data I pipe to lynx?
> 
> For example if I run:
> 
>   lynx -post_data http://class/process.cgi < infile.txt
> 
> I know infile can contain strings like "a=foo&b=bar", but what does it
> look like if I want to upload a separate file?

1.) Lynx is most likely not the best tool for this.
    a) It should be simpler to just use FTP.
    b) There must be command line tools meant specifically for
       upload via HTTP POST.  I would look around a bit.
       Let us know what you find. :)
    c) There must be command line tools meant specifically for
       upload via HTTP PUT.  I would look around a bit.
       Let us know what you find. :)

(Using the PUT method would be the more logical way for this kind
of thing, but you have to have it supported by the server/script,
of course.)

2.) If you insist on using Lynx:

This shouldn't have anything to do with the file upload patches.
Those only come into play for generating postable data from HTML form
+ user input.  But with -post_data, you already start out with the
postable data, they are (basically) taken as is, *you* have to massage
the data into a shape the server/script understands.

IOW, if you can get it to work with lynx, it should also work without
the file upload code; and if it doesn't work, the file upload patches
won't help.  (Unless the code is changed, of course.)

This is my understanding of those patches, I probably haven't looked
at the latest incarnation but am pretty sure that nothing substantial
has changed in this respect.

Are you in control of the process.cgi script?  If yes, modify it (or
make an alternative version) that understands what you are going to
feed it.  Both sides are in your hands, so it's doable.

If you are limited to the existing process.cgi, it may not be possible.

Lynx sends all POST data given with -post_data in
"application/x-www-form-urlencoded" form.  More exactly: it assumes the
input is in this form and labels it as such when sending.  This
is hardwired in the code.  If process.cgi only understands
"multipart/form-data", you are out of luck.

If process.cgi does understand "application/x-www-form-urlencoded",
you have to figure out how to mangle the input file so it's acceptable
to the script - probably by experimentation and tracing.  Most likely
you will need to
  - URL-encode some characters.  That may include all line endings,
    which should be given as '%0D%0A' (or '%0A'?) instead of '\n'.
    Also, probably, at least the characters '&', '=', and '%'; and
    ' ' (space; as '+' or '%20'?).  Might as well do all except letters
    and numbers...
  - prefix the whole mess with 'fieldname='.
  - add some other 'name=value' pairs separated by '&'.
(But again, it's all between you and process.cgi.  What it actually wants
may be different.)

A not too complicated perl script should be able to do all that for you,
if it's worth the trouble.

Another limitation is imposed by lynx: you cannot have a line starting
with '---' in the data, it would be taken as an end-of-input indication.
That means '---' at the very beginning or else the sequence '\n---';
of course if you have encoded all '\n' chars the sequence cannot occur.
The good thing is, I don't see any fixed length limitations.  (Haven't
actually tested though).  Input cannot contain '\0' bytes, but I doubt
that that is part of your requirements (and the server/script would
probably have problems with that anyway unless, maybe, if it's encoded).

Good luck.

   Klaus