[Top][All Lists]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bug-wget] HTTP quota bug

From: Andrey Semenchuk
Subject: Re: [Bug-wget] HTTP quota bug
Date: Sun, 03 May 2009 12:39:32 +0300
User-agent: Icedove (X11/20080724)

Micah Cowan wrote:
As it described in documentation, when --quota option is used "download
will be aborted when the quota is exceeded". But HTTP code has no
corresponding lines to break download unlike FTP code. So, if some file
is downloaded via HTTP, it will be fully downloaded and stored (no
matter is --quota option used or not) but with additional warning when
quota is exceeded:  "Download quota (... bytes) EXCEEDED!"

What documentation are you talking about? This is what I see:

     Specify download quota for automatic retrievals.  The value can be
     specified in bytes (default), kilobytes (with `k' suffix), or
     megabytes (with `m' suffix).

     Note that quota will never affect downloading a single file.  So
     if you specify `wget -Q10k ftp://wuarchive.wustl.edu/ls-lR.gz',
     all of the `ls-lR.gz' will be downloaded.  The same goes even when
     several URLs are specified on the command-line.  However, quota is
     respected when retrieving either recursively, or from an input
     file.  Thus you may safely type `wget -Q2m -i sites'--download
     will be aborted when the quota is exceeded.

     Setting quota to 0 or to `inf' unlimits the download quota.

Which is exactly the case, whether you're talking FTP, or HTTP. It
doesn't break download in the middle of a file.

Which, yeah, I agree is counter-intuitive. But with a program like wget,
I can never be sure that changing this won't break someone's script
somewhere. Not that we shouldn't do proper quotas, but we most likely
need to add that feature as a different option.
I don't think that changing this may break someone's script. Most likely current wget's behaviour may cause enexpected results and can occupy match more disk space than it was expected.

Let's assume that someone write script and set quota to 1G.
As it described in manual "Thus you may safely type `wget -Q2m -i sites'--download will be aborted when the quota is exceeded". So, user will expect that command `wget -Q1000m -i sites' will be executed safely and will be aborted as soon as quota is exceed. But, if user download file which size is 5GB (or mirror any site that contain similar file) it will be used 5-6Gb of disk space after script execution.

So, in this case I can't see any safe wget execution with --quota option. At the same time I see that if free space at selected partition was 4,5GB, wget can't download all files (because they are need more disk space) but it can break normal system functionality (if come processes are working with selected partition and will not find free space after wget execution)

I think that manual page should be appended by explanation that "quota is checked only when file download is finished and it doesn't break download in the middle of a file" or something similar. Also, I hope that additional option will be added to future versions of wget to provide breaking file download as soon as quota is exceed.

Best wishes,
Andrey Semenchuk
Trifle Co., Ltd.

reply via email to

[Prev in Thread] Current Thread [Next in Thread]