Re: [Bug-wget] wget fails to parse existing .listing files for FTP download

[Nikolay Merinov]

"Mark" <address@hidden> writes:

> I'm using a command like this:
> wget -x -S --no-clobber --recursive --level=inf --debug
> --no-remove-listing --append-output=wget_log.txt --progress=dot:mega
> "ftp://ftp.example.com/path/to/files/";
>
> Would a workaround for the not-able-to-continue-mirroring problem be to
> use --continue and not --no-clobber? Then wget should re-download the
> .listing files, and because the already-downloaded files are all fully
> downloaded, it should skip downloading them again.
>
> Here's some example output.
>
>
> DEBUG output created by Wget 1.13.4 on linux-gnu.
>
> URI encoding = `UTF-8'
> Using `ftp.microsoft.com/.listing' as listing tmp file.
> File `ftp.microsoft.com/.listing' already there; not retrieving.
> UNKNOWN; perms 0;
> Skipping.
> UNKNOWN; perms 0;
> Skipping.
> UNKNOWN; perms 0;
> Skipping.
> UNKNOWN; perms 0;
> Skipping.
> UNKNOWN; perms 0;
> Skipping.
> UNKNOWN; perms 0;
> Skipping.
> UNKNOWN; perms 0;
> Skipping.
> UNKNOWN; perms 0;
> Skipping.
> UNKNOWN; perms 0;
> Skipping.
> UNKNOWN; perms 0;
> Skipping.
> UNKNOWN; perms 0;
> Skipping.
> UNKNOWN; perms 0;
> Skipping.
> UNKNOWN; perms 0;
> Skipping.

Hi, Mark.

The problem is that ftp system type (and ftp listing format) determined
during listing downloading. But when `--no-clobber' option given wget
not re-download .listing file and try to parse it with default (unix)
listing format.

There is two possible solution of this problem:

1. Rewrite listing file even when --no-clobber is given.
2. If file .listing exist on filesystem make additional SYST request.

Patch with first solution attached to this mail.

Good workaround for your problem is not to use --no-remove-listing option
for any purpose except debugging.

>From 2fff576be8988382f06ddecdf839b347e418a8e8 Mon Sep 17 00:00:00 2001
From: Nikolay Merinov <address@hidden>
Date: Wed, 3 Oct 2012 21:31:25 +0600
Subject: [PATCH] Recieve ftp listing even when .listing file exist and
 --no-clobber option is given.

---
 src/ChangeLog |    5 +++++
 src/ftp.c     |    7 ++++++-
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/src/ChangeLog b/src/ChangeLog
index 069478b..a84073d 100644
--- a/src/ChangeLog
+++ b/src/ChangeLog
@@ -1,3 +1,8 @@
+2012-10-03 Merinov Nikolay <address@hidden>
+
+       * ftp.c (ftp_loop_internal): Ignore --no-clobber option when
+       receiving directory listing.
+
 2012-09-29 Merinov Nikolay <address@hidden>
 
        * ftp-ls.c (ftp_parse_winnt_ls): Support filename extracting with
diff --git a/src/ftp.c b/src/ftp.c
index 669e663..b585631 100644
--- a/src/ftp.c
+++ b/src/ftp.c
@@ -1424,7 +1424,12 @@ ftp_loop_internal (struct url *u, struct fileinfo *f, 
ccon *con, char **local_fi
 
   /* If the output_document was given, then this check was already done and
      the file didn't exist. Hence the !opt.output_document */
-  if (opt.noclobber && !opt.output_document && file_exists_p (con->target))
+
+  /* If we receive .listing file it is necessary to determine system type of 
the ftp
+     server even if opn.noclobber is given. Thus we must ignore opt.noclobber 
in
+     order to establish connection with the server and get system type. */
+  if (opt.noclobber && !opt.output_document && file_exists_p (con->target)
+      && !((con->cmd & DO_LIST) && !(con->cmd & DO_RETR)))
     {
       logprintf (LOG_VERBOSE,
                  _("File %s already there; not retrieving.\n"), quote 
(con->target));
-- 
1.7.10.4