Re: [Bug-wget] [PATCH] New option: --rename-output: modify output filename with perl

[Andrew Cady]

On Fri, Jul 26, 2013 at 11:27:02AM +0200, Tim Rühsen wrote:
> Thank you for your work, Andrew !

I appreciate the appreciation :)

> Call an external program after downloading and saving, not only with filename 
> but also with additional information (e.g. HTTP header stuff like 
> content-type 
> etc.). This external program would be able to rename the file (and telling 
> this 
> Wget via pipe output as Andrew suggested), but also being able to analyze the 
> file content, saving meta-infos into a database, extract and execute 
> javascript, etc.
> 
> Two thoughts:
> - the whole idea is not relevant for single downloads.
> - the above things could be done by analysing Wgets debugging output. I have 
> done this several times. But the debugging output is not documented, so these 
> solutions are hacks and might break with the next version of Wget.

This would be useful and would not be hard to do.  For example, instead
of printing just hstat.local_file to the pipe, you could serialize the
whole hstat struct (say as JSON).

However, the big issue is that you would need to define an interface
for the communication, and then you need to keep that interface
stable.  Maintaining such an interface would inevitably restrain future
development.  So, that requires some decision-making and policy.  Or
else the interface could be unstable, but then you have the same issue
as parsing the debugging output.

Incidentally, the former maintainer of wget, Micah Cowan, actually
started working on a wget "competitor" (so to speak) based on a plugin
architecture designed around this concept:

  
http://micah.cowan.name/2011/02/13/computers/software-development/announcement-niwt-nifty-integrated-web-tools/

  http://niwt.addictivecode.org/

I haven't really looked into it -- in fact I didn't know it was actually
released until I searched for it just now (I just remembered Micah
saying he was going to work on it).  At a glance, it looks to be very
flexible, but also very incomplete.

Niwt apparently uses "an HTTP-based protocol" to communicate between
plugins.