[Bug-wget] FTP retry problems

[Mick Brooks]

Hi,

I'm using wget to recursively retrieve a directory tree from an FTP
server. Occasionally there are connection failures (thanks to a flaky
server, or possibly dodgy network infrastructure on the route, but
that's not the problem I'm concerned with here), so I'm relying on
wget to retry failed downloads. I'm seeing a problem where the retry
"succeeds" but results in an empty file.

I've included log output (using the -d switch) from when this happens
below. You'll see that for the retry wget sends a REST command to
reset the start position before starting the RETR command. I'm
confused about the argument to REST: 51132. It's the full length in
bytes of the file to be retrieved. The RETR then shows the entire
contents of the file being skipped, and wget announces that it
successfully retrieved and saved 0 bytes.

Is that argument to REST correct?

$ wget --version
GNU Wget 1.11.4

$ wget ftp://ftp.site.com/Dir/All/dirname --ftp-user=user
--ftp-password=password --recursive --no-host-directories --no-parent
--cut-dirs=2 --timestamping --output-file output.log -P target-dir -d

log output:
... skipped successful stuff ...
--2009-06-10 09:50:54--  ftp://ftp.site.com/Dir/All/dirname/subdir/dirname
           => `target-dir/dirname/subdir/.listing'
changing working directory
Prepended initial PWD to relative path:
   pwd: '/'
   old: 'Dir/All/dirname/subdir'
  new: '/Dir/All/dirname/subdir'
==> CWD /Dir/All/dirname/subdir ...
--> CWD /Dir/All/dirname/subdir

250 Directory successfully changed.
done.
conaddr is: XXX.XXX.XXX.XXX
==> PASV ...
--> PASV

227 Entering Passive Mode (XXX,XXX,XXX,XX,XXX,XXX)
trying to connect to XXX.XXX.XXX.XXX port 32175
Created socket 5.
done.    ==> LIST ...
--> LIST -a

150 Here comes the directory listing.
done.

     0K                                                        82.0K=0.002s

Closed fd 5
226 Directory send OK.
2009-06-10 09:50:54 (82.0 KB/s) - `target-dir/dirname/subdir/.listing'
saved [194]

DIRECTORY; perms 775; size: 2048; month: Jun; day: 9; time: 15:12:00 (no yr);
Ignoring `.' and `..';
Skipping.
DIRECTORY; perms 775; size: 118784; month: Jun; day: 9; time: 23:18:00 (no yr);
Ignoring `.' and `..';
Skipping.
PLAINFILE; perms 664; size: 51132; month: Jun; day: 9; time: 15:12:00
(no yr); filename.txt
Removed `target-dir/dirname/subdir/.listing'.
--2009-06-10 09:50:54--  ftp://ftp.site.com/Dir/All/dirname/subdir
/filename.txt
           => `target-dir/dirname/subdir/filename.txt'
==> CWD not required.
conaddr is: XXX.XXX.XXX.XXX
==> PASV ...
--> PASV

27 Entering Passive Mode (XXX,XXX,XX,XX,XX,XX)

421 Timeout.

Cannot initiate PASV transfer.
conaddr is: XXX.XXX.XXX.XXX
==> PORT ... conaddr is: XXX.XXX.XXX.XXX
Local socket fd 5 bound.
binding to address XXX.XXX.XXX.XXX using port 40152.

--> PORT XX,XX,XXX,XX,XXX,XXX

Closed fd 5

Error in server response, closing control connection.
Closed fd 4
Closed fd 5
Retrying.

--2009-06-10 10:00:55--  ftp://ftp.site.com/Dir/All/dirname/subdir/filename.txt
  (try: 2) => `target-dir/dirname/subdir/filename.txt'
Found ftp.site.com in host_name_addresses_map (0x5567b0)
Connecting to ftp.site.com|XXX.XXX.XXX.XXX|:21... connected.
Created socket 4.
Releasing 0x00000000005567b0 (new refcount 1).
Logging in as user ... 220 Welcome to Site FTP service.

--> USER user

331 Please specify the password.

--> PASS password

230 Login successful.
Logged in!
==> SYST ...
--> SYST

215 UNIX Type: L8
done.    ==> PWD ...
--> PWD

257 "/"
done.
==> TYPE I ...
--> TYPE I

200 Switching to Binary mode.
done.  changing working directory
Prepended initial PWD to relative path:
   pwd: '/'
   old: 'Dir/All/dirname/subdir'
  new: '/Dir/All/dirname/subdir'
==> CWD /Dir/All/dirname/subdir ...
--> CWD /Dir/All/dirname/subdir

250 Directory successfully changed.
done.
conaddr is: XXX.XXX.XXX.XXX
==> PASV ...
--> PASV

227 Entering Passive Mode (XXX,XXX,XXX,XX,XXX,XXX)
trying to connect to XXX.XXX.XXX.XXX port 31681
Created socket 5.
done.    ==> REST 51132 ...
--> REST 51132

350 Restart position accepted (51132).
done.
==> RETR filename.txt ...
--> RETR filename.txt

150 Opening BINARY mode data connection for filename.txt (51132 bytes).
done.
Length: 51132 (50K), 0 (0) remaining

     0K ,,,,,,,,,, ,,,,,,,,,, ,,,,,,,,,, ,,,,,,,,,, ,,,,,,,,, 100% 0.00 =0s

Closed fd 5
226 File send OK.
2009-06-10 10:00:55 (0.00 B/s) -
`target-dir/dirname/subdir/filename.txt' saved [51132]
... skipped more successful stuff ...