Re: [Bug-wget] New option "--no-list-a"

[Giuseppe Scrivano]

Hi Andrea,

would you mind to rework the patch against the current git repository?
Some of the fixes (the documentation bits) are already there.

Also, before I could accept a non-trivial change, you should complete
the copyright assignment to the FSF process.  I will send you a separate
email with more details.

Thanks,
Giuseppe



"Andrea Urbani" <address@hidden> writes:

> Good afternoon,
> some systems (at least info-zip.org) have problems with the ftp command "LIST 
> -a".
> I.e. trying
>
> wget --timestamping --no-directories --recursive --level=1 --accept
> "zip*.t*,unzip*.t*" --reject "*.zip,unzip55?.*,zip23?.*" --tries=5
> --waitretry=10 --timeout=15 --debug
> ftp://ftp.info-zip.org/pub/infozip/src/
>
> you get
>
> Created socket 4.
> done. ==> LIST ...
> --> LIST -a
>
> 150 List started.
> done.
>
>  [ <=> ] 0 --.-K/s in 0s
>
> Closed fd 4
> 226 Transfer completed.
> 2013-08-25 06:40:42 (0.00 B/s) - â.listingâ saved [0]
>
> that means no errors but also an empty list of files.
> Within the attached patch I have added a new parameter, --no-list-a, that 
> tells WGET just to use the "LIST" command.
> If you try the new parameter
>
> wget --timestamping --no-directories --recursive --level=1 --accept
> "zip*.t*,unzip*.t*" --reject "*.zip,unzip55?.*,zip23?.*" --tries=5
> --waitretry=10 --timeout=15 --debug --no-list-a
> ftp://ftp.info-zip.org/pub/infozip/src/
>
> you will get
>
> Created socket 4.
> done. ==> LIST ... (Skipping "LIST -a")
> --> LIST
>
> 150 List started.
> done.
>
>  [ <=> ] 1,855 --.-K/s in 0.06s
>
> Closed fd 4
> 226 Transfer completed.
> 2013-08-25 06:45:08 (30.5 KB/s) - â.listingâ saved [1855]
>
> PLAINFILE; perms 645; size: 1063171; month: Oct; day: 23; year: 1992 (no tm); 
> calgarycorpus.zip
> PLAINFILE; perms 645; size: 24979; month: Feb; day: 9; year: 2004 (no tm); 
> check-4_3-src.zip
> ...
> ...
>
>
> Please, make attention that inside the wget.texi I have changed some "itemx" 
> into "item" as reported me by texinfo ver. 5.1
>
> Bye
> Andrea
> matfanjol
> http://matfanjol.users.sourceforge.net/
>
> Index: src/ftp-basic.c
> ===================================================================
> --- src/ftp-basic.c.orig      2012-05-12 17:18:27.000000000 +0200
> +++ src/ftp-basic.c   2013-08-24 16:40:29.000000000 +0200
> @@ -975,8 +975,14 @@
>       fail, but will never do what is desired here, skip directly to the
>       simple "LIST" command (assumed to be the last one in the list).
>    */
> -  if (rs == ST_VMS)
> -    i = countof (list_commands)- 1;
> +  /* 2013-08-24 matfanjol: Other systems seems to have problem with 
> +                           "LIST -a", so I don't use it if requested 
> +  */
> +  if ( (rs == ST_VMS) || (opt.nolista) )
> +    {
> +      DEBUGP ((" (Skipping \"LIST -a\") "));
> +      i = countof (list_commands)- 1;
> +    }
>  
>    do {
>      /* Send request.  */
> Index: tests/FTPServer.pm
> ===================================================================
> --- tests/FTPServer.pm.orig   2013-08-25 12:10:59.024413164 +0200
> +++ tests/FTPServer.pm        2013-08-25 12:13:31.000000000 +0200
> @@ -78,8 +78,11 @@
>  
>  sub _LIST_command
>  {
> -    my ($conn, $cmd, $path) = @_;
> +    my ($conn, $cmd, $path, $empty_list_if_list_a) = @_;
>      my $paths = $conn->{'paths'};
> +    my $ReturnEmptyList = (defined ($empty_list_if_list_a)
> +                           && $path eq '-a');
> +
>  
>      # This is something of a hack. Some clients expect a Unix server
>      # to respond to flags on the 'ls command line'. Remove these flags
> @@ -94,11 +97,15 @@
>      # working directory.
>      local $_;
>  
> -    $dir = FTPPaths::path_merge($dir, $path);
> -    my $listing = $paths->get_list($dir);
> -    unless ($listing) {
> -        print {$conn->{socket}} "550 File or directory not found.\r\n";
> -        return;
> +    my $listing;
> +    if ( ! $ReturnEmptyList )
> +    {
> +      $dir = FTPPaths::path_merge($dir, $path);
> +      $listing = $paths->get_list($dir);
> +      unless ($listing) {
> +          print {$conn->{socket}} "550 File or directory not found.\r\n";
> +          return;
> +      }
>      }
>  
>      print STDERR "_LIST_command - dir is: $dir\n" if $log;
> @@ -112,8 +119,11 @@
>          return;
>      }
>  
> -    for my $item (@$listing) {
> -        print $sock "$item\r\n";
> +    if ( ! $ReturnEmptyList )
> +    {
> +      for my $item (@$listing) {
> +          print $sock "$item\r\n";
> +      }
>      }
>  
>      unless ($sock->close) {
> @@ -634,7 +644,17 @@
>                  }
>  
>                  # Run the command.
> -                &{$command_table->{$cmd}} ($conn, $cmd, $rest);
> +                # 2013-08-25 matfanjol
> +                # Only for the LIST command I pass also the 
> empty_list_if_list_a behavior
> +                if ( $cmd eq 'LIST' ) 
> +                {
> +                  &{$command_table->{$cmd}} ($conn, $cmd, $rest, 
> +                                             
> $self->{_server_behavior}{empty_list_if_list_a});
> +                }
> +                else
> +                {
> +                  &{$command_table->{$cmd}} ($conn, $cmd, $rest);
> +                }
>              }
>          } else { # Father
>              close $socket;
> Index: src/init.c
> ===================================================================
> --- src/init.c.orig   2012-07-08 11:35:36.000000000 +0200
> +++ src/init.c        2013-08-23 14:33:24.000000000 +0200
> @@ -213,6 +213,7 @@
>    { "mirror",           NULL,                   cmd_spec_mirror },
>    { "netrc",            &opt.netrc,             cmd_boolean },
>    { "noclobber",        &opt.noclobber,         cmd_boolean },
> +  { "nolista",          &opt.nolista,           cmd_boolean },
>    { "noparent",         &opt.no_parent,         cmd_boolean },
>    { "noproxy",          &opt.no_proxy,          cmd_vector },
>    { "numtries",         &opt.ntry,              cmd_number_inf },/* 
> deprecated*/
> Index: src/main.c
> ===================================================================
> --- src/main.c.orig   2012-07-07 10:26:21.000000000 +0200
> +++ src/main.c        2013-08-23 14:23:30.000000000 +0200
> @@ -234,6 +234,7 @@
>      { "mirror", 'm', OPT_BOOLEAN, "mirror", -1 },
>      { "no", 'n', OPT__NO, NULL, required_argument },
>      { "no-clobber", 0, OPT_BOOLEAN, "noclobber", -1 },
> +    { "no-list-a", 0, OPT_BOOLEAN, "nolista", -1 },
>      { "no-parent", 0, OPT_BOOLEAN, "noparent", -1 },
>      { "output-document", 'O', OPT_VALUE, "outputdocument", -1 },
>      { "output-file", 'o', OPT_VALUE, "logfile", -1 },
> @@ -662,6 +663,8 @@
>      N_("\
>         --no-glob               turn off FTP file name globbing.\n"),
>      N_("\
> +       --no-list-a             disable \"LIST -a\" command and use just 
> LIST.\n"),
> +    N_("\
>         --no-passive-ftp        disable the \"passive\" transfer mode.\n"),
>      N_("\
>         --preserve-permissions  preserve remote file permissions.\n"),
> Index: src/options.h
> ===================================================================
> --- src/options.h.orig        2012-06-06 13:42:10.000000000 +0200
> +++ src/options.h     2013-08-23 14:27:29.000000000 +0200
> @@ -280,6 +280,8 @@
>    bool show_all_dns_entries; /* Show all the DNS entries when resolving a
>                                  name. */
>    bool report_bps;              /*Output bandwidth in bits format*/
> +  bool nolista;                 /* Avoid the "LIST -a" command in FTP.
> +                                   "LIST" will be used. */
>  };
>  
>  extern struct options opt;
> Index: tests/run-px
> ===================================================================
> --- tests/run-px.orig 2013-08-25 12:08:09.719417558 +0200
> +++ tests/run-px      2013-08-25 12:08:40.000000000 +0200
> @@ -35,6 +35,7 @@
>      'Test-ftp-iri-fallback.px',
>      'Test-ftp-iri-recursive.px',
>      'Test-ftp-iri-disabled.px',
> +    'Test-ftp-no-list-a.px',
>      'Test-HTTP-Content-Disposition-1.px',
>      'Test-HTTP-Content-Disposition-2.px',
>      'Test-HTTP-Content-Disposition.px',
> Index: tests/Test-ftp-no-list-a.px
> ===================================================================
> --- /dev/null 2013-08-25 06:34:23.227937278 +0200
> +++ tests/Test-ftp-no-list-a.px       2013-08-25 12:14:05.000000000 +0200
> @@ -0,0 +1,54 @@
> +#!/usr/bin/env perl
> +
> +use strict;
> +use warnings;
> +
> +use FTPTest;
> +
> +
> +###############################################################################
> +
> +my $afile = <<EOF;
> +Some text.
> +EOF
> +
> +my $bfile = <<EOF;
> +Some more text.
> +EOF
> +
> +$afile =~ s/\n/\r\n/g;
> +$bfile =~ s/\n/\r\n/g;
> +
> +# code, msg, headers, content
> +my %urls = (
> +    '/afile.txt' => {
> +        content => $afile,
> +    },
> +    '/bfile.txt' => {
> +        content => $bfile,
> +    },
> +);
> +
> +my $cmdline = $WgetTest::WGETPATH . " --no-directories --recursive --level=1 
> --accept \"?file.txt\" --no-list-a ftp://localhost:{{port}}/";;
> +
> +my $expected_error_code = 0;
> +
> +my %expected_downloaded_files = (
> +    'afile.txt' => {
> +        content => $afile,
> +    },
> +    'bfile.txt' => {
> +        content => $bfile,
> +    },
> +);
> +
> +###############################################################################
> +
> +my $the_test = FTPTest->new (name => "Test-ftp-bad-list",
> +                             input => \%urls,
> +                             cmdline => $cmdline,
> +                             errcode => $expected_error_code,
> +                             output => \%expected_downloaded_files,
> +                             server_behavior => {empty_list_if_list_a => 1});
> +exit $the_test->run();
> +
> Index: doc/wget.texi
> ===================================================================
> --- doc/wget.texi.orig        2012-08-04 10:41:52.000000000 +0200
> +++ doc/wget.texi     2013-08-24 17:10:57.000000000 +0200
> @@ -876,7 +876,7 @@
>  actions of one.
>  
>  @cindex proxy
> address@hidden --no-proxy
> address@hidden --no-proxy
>  Don't use proxies, even if the appropriate @code{*_proxy} environment
>  variable is defined.
>  
> @@ -977,9 +977,9 @@
>  whose encoding does not match the one used locally.
>  
>  @cindex IPv6
> address@hidden -4
> address@hidden -4
>  @itemx --inet4-only
> address@hidden -6
> address@hidden -6
>  @itemx --inet6-only
>  Force connecting to IPv4 or IPv6 addresses.  With @samp{--inet4-only}
>  or @samp{-4}, Wget will only connect to IPv4 hosts, ignoring AAAA
> @@ -1764,6 +1764,16 @@
>  system-specific.  This is why it currently works only with Unix @sc{ftp}
>  servers (and the ones emulating Unix @code{ls} output).
>  
> address@hidden list ftp
> address@hidden --no-list-a
> +When a directory list is required within a @sc{ftp} connection WGet sends 
> +the @code{LIST -a} command. If this command fails, WGet tries again with 
> +the @code{LIST} command.
> +Some systems don't work well with @code{LIST -a} command (i.e. the command
> +doesn't fail, but the list is empty). In those cases it is possible to use 
> +the @samp{--no-list-a} option that tells WGet to use just the @code{LIST}
> +command.
> +
>  @cindex passive ftp
>  @item --no-passive-ftp
>  Disable the use of the @dfn{passive} FTP transfer mode.  Passive FTP
> @@ -3094,7 +3104,7 @@
>  Change setting of passive @sc{ftp}, equivalent to the
>  @samp{--passive-ftp} option.
>  
> address@hidden password = @var{string}
> address@hidden password = @var{string}
>  Specify password @var{string} for both @sc{ftp} and @sc{http} file 
> retrieval. 
>  This command can be overridden using the @samp{ftp_password} and 
>  @samp{http_password} command for @sc{ftp} and @sc{http} respectively.