Re: [Bug-wget] [wget => Kernel message] Too many open files

[Giuseppe Scrivano]

Hi Minato,

I wasn't able to reproduce this problem, I have tried with a directory
containing 50000 files.


What do you get using this command (be sure that website.com is the real
one)?

strace -e open,socket wget -q -nc -r -l inf --no-remove-listing \
       http://website.com/ 2>&1 | tail


Can you check your "ulimit -n"?


Cheers,
Giuseppe



Minato Namikaze <address@hidden> writes:

> First of all : sorry for displaying the french version of wget log messages.
>
> Bug (?) type : Kernel message
> Reproduce : wget -nc -r -l inf --no-remove-listing http://website.com/
> (launched TWICE in different terminals, and with a lot of files already 
> downloaded)
>
> Explanation : The two instances will pass over the numerous files already 
> downloaded. After that, they will try to download the first not downloaded 
> file.
> Then, one of thoses cases appear :
> Case 1 : one will be faster and they will download one file out of two each 
> (during approximatively 10~20 minutes max until case 2 appears)
> Case 2 : one will refuse to download and will display the next error message 
> for all files.
>
> Log :
> --2010-07-13 14:04:35--  http://www.website.com/folders/
> Connexion vers site|ip|:80...connecté.
> requête HTTP transmise, en attente de la réponse...200 OK
> Longueur: 11886 (12K) [text/html]
> www.website.com/folders/file.html: Too many open files
>
> Impossible d'écrire dans <<www.website.com/folders/file.html>> (Too many open 
> files).
> /endlog
>
>
> This is the message displayed for all files by ONE of the wgets running
> Information: if I stop the failing one, relaunch it, and wait until all files 
> have been not-clobbered, I have great chances that the other one will fail 
> with the same message(s).
>
> Information: /proc/sys/fs/file-max is set at 405540.
> Information: the folder where the website is mirrored contains currently 
> 525000 html files and as much folders (total: 6.6 GB).
> Information: When I have a (BIG) network problem, and whet cancels every 
> single file because the address can't be resolved, the elapsed time indicated 
> is always 0s. (and thus the average speed is enormous)
>
>
> If this mailing list was not the place where I should have posted this, 
> please let me know.
>                                         
> _________________________________________________________________
> Vous voulez regarder la TV directement depuis votre PC ? C'est très simple 
> avec Windows 7
> http://clk.atdmt.com/FRM/go/229960614/direct/01/