Re: [Bug-wget] Metalink support

[Anthony Bryan]

On Fri, Jul 3, 2015 at 5:21 PM, Tim Rühsen <address@hidden> wrote:
> Am Freitag, 3. Juli 2015, 20:23:00 schrieb Hubert Tarasiuk:
>> W dniu 03.07.2015 o 17:12, Tim Ruehsen pisze:
>> > To copy GPGME m4 macro, I have to install libgpgme11-dev. And if I do
>> > that, a ./bootstrap (maybe just a configure) does the job for me.
>>
>> Actually I meant copying over the m4 GPGME macro and adding it to Wget's
>> repository, so that it would be available straight after cloning (I have
>> seen some projects including gpgme.m4 in their repositories to resolve
>> the problem).
>> Or maybe we could provide some kind of wrapper that would call the
>> proper GPGME macro if it is defined elsewhere, and otherwise do nothing.
>> (Not sure how to do that though.)
>
> Let me take a closer look at that in the next days.
>
>
>> > I tried
>> > $ wget http://www.metalinker.org/samples/dsl-3.3.iso.metalink
>> > which I understand as a download link to dsl-3.3.iso.
>> >
>> > That only downloads the .metalink file and that's it !?
>> > I expected Wget to download dsl-3.3.iso (of course first the .metalink
>> > file as intermediate file, which is needed for mirror information).
>> > I am aware of --input-metalink, but would like to see a bit more of
>> > automation in the above example.
>>
>> I kind of agree with that. But we should probably not change the default
>> behaviour of previous versions. How about this:
>
> What is 'previous versions' ? If you are talking about the wget-parallel
> branch, it never became an official release. So there is no 'default'
> behaviour defined as such that we have to take care for.
>
>> $ wget
>> --input-metalink=http://www.metalinker.org/samples/dsl-3.3.iso.metalink
>>
>> Would that work for you? (Allowing to provide an URL instead of local
>> file to the input-metalink option.)
>
> Of course it works for me, but we are not talking about me :-)
>
> The main question is: what does the typical wget user expect ?
> I would thinks (s)he expects the download of the file described by the
> .metalink file. Of course a few users really want to have only the .metalink
> file (for testing/inspection). So there should be a possibility to do exactly
> that (e.g. --keep-metalink).
>
> Before making any decision/action we should wait for some other voices.

I'm not sure what the typical wget user expects, but here are some examples.

if you keep the current behavior, it might be helpful to print to the
user that they've downloaded a metalink, and how to make wget use it.

aria2 (another command line metalink downloader) uses these commands
to download the metalink and process it (download the file described
by it & do a hash check) by default. you have to specify
'--follow-metalink=false' to only download the metalink XML file.

curl on the other hand just downloads the metalink XML file by default
(like any other file), and requires a '--metalink' option to process a
local file or URL. (examples & help/man pages follow)

aria2c URL
or
aria2c --metalink-file=filename

aria2 help:

-M, --metalink-file=METALINK_FILE The file path to the .meta4 and .metalink

                              file. Reads input from stdin when '-' is

                              specified.


                              Possible Values: /path/to/file, -


--follow-metalink=true|false|mem If true or mem is specified, when a file

                              whose suffix is .meta4 or .metalink, or content

                              type of application/metalink4+xml or

                              application/metalink+xml is downloaded, aria2

                              parses it as a metalink file and downloads files

                              mentioned in it.

                              If mem is specified, a metalink file is not

                              written to the disk, but is just kept in memory.

                              If false is specified, the .metalink file is

                              downloaded to the disk, but is not parsed as a

                              metalink file and its contents are not

                              downloaded.


                              Possible Values: true, mem, false

                              Default: true



from the curl man page:

      --metalink

              This option can tell curl to parse and process a given
URI as Metalink file (both version 3 and 4 (RFC 5854)

              are supported) and make use of the mirrors listed within
for failover if there are errors (such as the  file

              or  server  not being available). It will also verify
the hash of the file after the download completes. The

              Metalink file itself is downloaded and processed in
memory and not stored in the local file system.


              Example to use a remote Metalink file:


              curl --metalink http://www.example.com/example.metalink


              To use a Metalink file in the local file system, use
FILE protocol (file://):


              curl --metalink file://example.metalink


              Please note that if FILE protocol is disabled, there is
no way to use a local Metalink file at the  time  of

              this  writing. Also note that if --metalink and
--include are used together, --include will be ignored. This

              is because including headers in the response will break
Metalink parser and if the headers are  included  in

              the file described in Metalink file, hash check will fail.


-- 
(( Anthony Bryan ... Metalink [ http://www.metalinker.org ]
  )) Easier, More Reliable, Self Healing Downloads