Re: [Bug-wget] Metalink support

[Tim Ruehsen]

Thanks, pushed.

Tim

On Tuesday 01 September 2015 23:37:46 Hubert Tarasiuk wrote:
> Here is a small fix to Metalink support in main function.
> The bug was that if the Metalink file parsing failed, there would be a
> free() call for non-initialized memory pointer.
> 
> W dniu 24.06.2015 o 08:06, Hubert Tarasiuk pisze:
> > I have prepared a Metalink patch for Wget. The two main features are:
> > 
> > - support Metalink v3 and Metalink v4 XML files:
> > https://tools.ietf.org/html/rfc5854
> > using libmetalink parser: https://launchpad.net/libmetalink
> > 
> > - support Metalink in HTTP headers: http://tools.ietf.org/html/rfc6249
> > 
> > 
> > Specifically what the patch implements (for both XML and HTTP):
> > - keep downloading from consecutive resource URLs until a successful
> > retrieval
> > - verify SHA256 digest (this digest is mandatory for Metalink, thus it
> > should be included in all Metalink documents)
> > - verify OpenPGP signatures (using public keys from user's keyring)
> > using GPGME and GnuPG
> > 
> > Checksum mismatch means download failure and it proceeds to try with
> > another resource (if available).
> > If the signature cannot be verified (missing public key would be the
> > most common reason), we do NOT assume download failure.
> > If the signature can be verified and the verification fails (ie. data
> > does not match signature), we assume download failure.
> > 
> > Please note that the PGP signatures are only working for
> > Metalink-over-HTTP at this time due to a bug in libmetalink.
> > 
> > 
> > Following options were added to Wget:
> > 
> > --input-metalink=FILE - download files described in Metalink file FILE
> > (like --input-file)
> > 
> > --metalink-over-http - when downloading from HTTP URLs:
> > -> issue a HEAD request and check for Metalink metadata in reponse
> > -> if found: switch to Metalink-mode
> > -> if not found: fall back to ordinary HTTP download
> > 
> > _Test suite_
> > I have made two modifications to Python test suite:
> > - allow multiple SendHeaders with same name by using a Python list as
> > dictionary value
> > - do not start the HTTP test in constructor; do it in the begin() method
> > instead (as the method name would suggest); original behaviour was to
> > run the test in object constructor and the begin() method would just
> > return the result
> > 
> > Please let me know what do you think about the patches. Some test cases
> > are included. If you would like to test it on actual servers, here is
> > what I found:
> > - Metalink files with PGP signatures: http://curl.haxx.se/download.html
> > - Metalink in HTTP headers:
> > https://download.gnome.org/apps/3.0/3.0.0/sources/
> > 
> > The commits are also available via Github interface:
> > https://github.com/jy987321/Wget/commits/metalink
> > 
> > Hubert
> > 
> > W dniu 28.05.2015 o 00:49, Hubert Tarasiuk pisze:
> >> I have talked with Giuseppe and he suggested that we might not do TCP
> >> Fast Open support for FTP at this time (he argued that FTP is slow
> >> either way :).
> >> 
> >> Instead I might focus on implementing some basics of Metalink protocol
> >> for HTTP and FTP resources in Wget.
> >> 
> >> Do you have any thoughts about that?