Re: [Bug-wget] Bug: resources can't be mirrored via FTP if their name starts with a space character

[Darshit Shah]

* Tim Rühsen <address@hidden> [190408 09:24]:
> Hello Christian,
> 
> please add options '-d -olog --no-remove-listing' and send us (privately
> if you have secrets in the log file) the files 'log' and .listing.
> 
> A problem with FTP directory listings is that there is no standard. A
> leading space could easily be part of the formatting which is meant to
> be human readable, not machine readable. Assuming one space as delimiter
> likely is a regression for other users.
> 
> In the long term, you should consider HTTPS instead of FTP.
> 

Actually, from the provided data, I can already see what is happening and why. 
Just as Tim mentioned, the listing file doesn't have a fixed standard and uses
whitespace as a field separator. 

Looking at the `.listing` file that Wget downloaded, I can see that there is
just whitespace between the size and the filename. Wget has no way to tell if
the whitespace is a field separator or part of the filename. The correct way to
deal with this would be for your FTP server to generate a listing file similar
to ls. Where, it wraps the filename in quotes in order to show that the
whitespace is indeed a part of the filename

What FTP server are you using?

> Regards, Tim
> 
> On 08.04.19 02:13, Christian Rosentreter wrote:
> > 
> > Hi there,
> > 
> > A small bug I encountered: if any file or directory on a remote machine has 
> > pathnames
> > starting with one or more spaces (e.g. " foobar.txt") then Wget fails to 
> > mirror those
> > resources via the good old FTP protocol. Tested with '1.18' and up-to-date 
> > '1.20.3' on
> > Mac OS X.
> > 
> > If required, I could provide a testing FTP account for this particular 
> > setup to a
> > Wget developer (privately.)
> > 
> > 
> > with kind regards,
> > Christian Rosentreter
> > 
> > 
> > 
> > 
> > ----
> > 
> > Example:
> > 
> > wget \
> >     --config auth.config \
> >     --mirror \
> >     --no-host-directories \
> >     --append-output wget.log \
> >     ftp://wget.annex.binaryriot.org/ 
> > 
> > The test setup on the remote server (up-to-date Ubuntu) looks something 
> > like this, 
> > output via 'ls -l':
> > test/
> >     -rw-r--r-- 1 foobar foobar    0 Apr  8 00:31 '  
> > foobar-with-leading-spaces.txt'
> >     -rw-r--r-- 1 foobar foobar    0 Apr  8 00:31 ' foo bar with spaces .txt 
> > '
> >     -rw-r--r-- 1 foobar foobar    0 Apr  8 00:31  foobar-normal.txt
> >     -rw-r--r-- 1 foobar foobar    0 Apr  8 00:31 
> > 'foobar-with-trailing-spaces.txt  '
> >     drwxr-xr-x 2 foobar foobar 4096 Apr  8 00:31 '  dir-with-leading-spaces'
> >     drwxr-xr-x 2 foobar foobar 4096 Apr  8 00:31 'dir-with-trailing-spaces  
> >   '
> >     drwxr-xr-x 2 foobar foobar 4096 Apr  8 00:31 '   dir with spaces  '
> > 
> > 
> > The content of the locally generated .listing file inside the "test" 
> > directory,
> > looks like this:
> >     -rw-r--r--   1 foobar   foobar          0 Apr  8 00:31   
> > foobar-with-leading-spaces.txt
> >     -rw-r--r--   1 foobar   foobar          0 Apr  8 00:31  foo bar with 
> > spaces .txt 
> >     -rw-r--r--   1 foobar   foobar          0 Apr  8 00:31 foobar-normal.txt
> >     -rw-r--r--   1 foobar   foobar          0 Apr  8 00:31 
> > foobar-with-trailing-spaces.txt  
> >     drwxr-xr-x   2 foobar   foobar       4096 Apr  8 00:31   
> > dir-with-leading-spaces
> >     drwxr-xr-x   2 foobar   foobar       4096 Apr  8 00:31 
> > dir-with-trailing-spaces    
> >     drwxr-xr-x   2 foobar   foobar       4096 Apr  8 00:31    dir with 
> > spaces  
> > 
> > 
> > The actual locally mirrored directories and files, output via OS X's 'ls - 
> > l' look
> > like this (any resource with leading spaces is obviously missing in this 
> > local copy
> > now. That's a wee-bit bad for backups when important files can be 
> > unexpectedly
> > M.I.A. ;) )
> > test/
> >     -rw-r--r--  1  foobar   foobar    0 Apr  8 00:31 foobar-normal.txt
> >     -rw-r--r--  1  foobar   foobar    0 Apr  8 00:31 
> > foobar-with-trailing-spaces.txt  
> >     drwxr-xr-x  3  foobar   foobar  102 Apr  8 00:32 
> > dir-with-trailing-spaces    
> > 
> > 
> > In the log file (via --append-output) various errors like this are 
> > generated (the
> > leading spaces are missing), but with the return code of the command 
> > wrongly indicating
> > that the full mirror was actually a success in the end:
> >     ...
> >     No such file 'test/foobar-with-leading-spaces.txt'.
> >     No such file 'test/foo bar with spaces .txt '.
> >     No such directory 'test/dir-with-leading-spaces'.
> >     No such directory 'test/dir with spaces  '.
> >     ...
> > 
> > 
> > 
> 



-- 
Thanking You,
Darshit Shah
PGP Fingerprint: 7845 120B 07CB D8D6 ECE5 FF2B 2A17 43ED A91A 35B6

signature.asc
Description: PGP signature