Re: [Bug-wget] [PATCH] New option: --rename-output: modify output filename with perl

[Tim Ruehsen]

On Friday 26 July 2013 14:30:00 Andrew Cady wrote:
> On Fri, Jul 26, 2013 at 11:27:02AM +0200, Tim Rühsen wrote:
> > Thank you for your work, Andrew !
> 
> I appreciate the appreciation :)
> 
> > Call an external program after downloading and saving, not only with
> > filename but also with additional information (e.g. HTTP header stuff
> > like content-type etc.). This external program would be able to rename
> > the file (and telling this Wget via pipe output as Andrew suggested), but
> > also being able to analyze the file content, saving meta-infos into a
> > database, extract and execute javascript, etc.
> > 
> > Two thoughts:
> > - the whole idea is not relevant for single downloads.
> > - the above things could be done by analysing Wgets debugging output. I
> > have done this several times. But the debugging output is not documented,
> > so these solutions are hacks and might break with the next version of
> > Wget.
> This would be useful and would not be hard to do.  For example, instead
> of printing just hstat.local_file to the pipe, you could serialize the
> whole hstat struct (say as JSON).
> 
> However, the big issue is that you would need to define an interface
> for the communication, and then you need to keep that interface
> stable.  Maintaining such an interface would inevitably restrain future
> development.  So, that requires some decision-making and policy.  Or
> else the interface could be unstable, but then you have the same issue
> as parsing the debugging output.
> 
> Niwt apparently uses "an HTTP-based protocol" to communicate between
> plugins.

Any protocol has it's pros and cons. So why not doing it the same/similar way 
as Micah does ? That seems to be intuitive - dumping the original HTTP headers 
and add your extension (e.g. 'X-Wget-Filename: directory/filename').

An additional Version: header as the first line to interpret makes even a 
radical protocol change possible (instead the program could be called with a 
--protocol-version command-line param).

An empty line separates two or more 'jobs', just to make it possible to start 
the external program one time and not for each file downloaded.

The program should just ignore unknown protocol lines.

The called program should 'answer' each 'job' with one or more lines (again, 
an empty line is the end of answer). HTTP like protocol...

Maybe you wait for a GO from Guiseppe...

Regards, Tim