[Bug-wget] Re: wget 1.12 memory exhausted

[Inaki San Vicente]

> 
> can you also specify -d and post here the last lines you get (or any
> other message that you consider useful for us)?
> 

Hi Giuseppe,

      Sorry if the message is posted twice, I've sent it a second time since it
hasn't been posted since I sent it the first time this morning.


      Thanks for the fast reply, hope this information will help you. 
      here is the -d command result snippet:
...
Closed fd 4
2010-08-05 10:06:50 (68,9 KB/s) - `www.euskadi.net/styles/r33-2220VAStyles.scss
' saved [178662]

Loaded www.euskadi.net/styles/r33-2220VAStyles.scss (size 178662). 
www.euskadi.net/styles/r33-2220VAStyles.scss:
merge(`http://www.euskadi.net/styles/r33-2220VAStyles.scss',
`/AVComun/r01gPageFootVA/images/r01gArrow.gif') ->
http://www.euskadi.net/AVComun/r01gPageFootVA/images/r01gArrow.gif
appending `http://www.euskadi.net/AVComun/r01gPageFootVA/images/r01gArrow.gif'
to urlpos.
Found URI: [url(/AVComun/r01gPageFootVA/images/r01gArrow.gif)] at 1753
[/AVComun/r01gPageFootVA/images/r01gArrow.gif]
www.euskadi.net/styles/r33-2220VAStyles.scss:
merge(`http://www.euskadi.net/styles/r33-2220VAStyles.scss',
`/AVComun/r01gContentListVA/images/r01gArrow2.gif') ->
http://www.euskadi.net/AVComun/r01gContentListVA/images/r01gArrow2.gif
appending
`http://www.euskadi.net/AVComun/r01gContentListVA/images/r01gArrow2.gif' 
to urlpos.
Found URI: [url(/AVComun/r01gContentListVA/images/r01gArrow2.gif)] at 5340
[/AVComun/r01gContentListVA/images/r01gArrow2.gif]

...
here are some more lines with Found URI: another file/merge message/appending
...

www.euskadi.net/styles/r33-2220VAStyles.scss:
merge(`http://www.euskadi.net/styles/r33-2220VAStyles.scss',
`/AVComun/r01gContentListVA/images/r01gSpriteTypoContentList.gif') ->
http://www.euskadi.net/AVComun/r01gContentListVA/images/
r01gSpriteTypoContentList.gif
appending
`http://www.euskadi.net/AVComun/r01gContentListVA/images
/r01gSpriteTypoContentList.gif'
to urlpos.
Found URI:
[url(/AVComun/r01gContentListVA/images/r01gSpriteTypoContentList.gif)] at 6560
[/AVComun/r01gContentListVA/images/r01gSpriteTypoContentList.gif]
www.euskadi.net/styles/r33-2220VAStyles.scss:
merge(`http://www.euskadi.net/styles/r33-2220VAStyles.scss',
`/AVComun/r01gContentListVA/images/r01gSpriteTypoContentList.gif') ->
http://www.euskadi.net/AVComun/r01gContentListVA/images
/r01gSpriteTypoContentList.gif
appending
`http://www.euskadi.net/AVComun/r01gContentListVA/images
r01gSpriteTypoContentList.gif'
to urlpos.
Found URI:
[url(/AVComun/r01gContentListVA/images/r01gSpriteTypoContentList.gif)] at 18833
[/AVComun/r01gContentListVA/images/r01gSpriteTypoContentList.gif]


And every time after this line wget will crash reporting memory exhausted (with
no other information, e.g.,space allocation data).


When I run the wget 1.10.2 version in other computers, the debug information
remains the same until the line:

"Loaded www.euskadi.net/styles/r33-2220VAStyles.scss (size 178662)."

There, instead of loading the scss file (which has been already saved), it will
report: 
"Dequeuing http://www.euskadi.net/scripts/r33-2220VAScripts.sjs at depth 2 "
and the continues downloading files normally.


> As another test, can you use valgrind to trace memory usage of wget?

running massif, this is the information it produces for the last snapshots 
(were the usage is at its maximum peak):

            
------------------------------------------------------------------------------
  n        time(i)         total(B)   useful-heap(B) extra-heap(B)    stacks(B)
------------------------------------------------------------------------------
 95     29,835,721          307,520          289,769        17,751          0
 96     30,846,542          308,872          290,852        18,020          0
 97     31,375,038          309,328          291,210        18,118          0



> 
> What does "ulimit -m" tell you?


Unlimited


Another thing I've notice (not sure if it is related), is that at the beginning
of the execution wget reports the settings applied according to the parameters
you have passed to it. However in the case of the 1.12 version it never reports
the option --force-directories (dirstruct=1) to be set. I've also tried to set
this option manually in the /etc/wgetrc file, but I get no improvement. Looking
to the debug messages could be something related to the recreation of the
original web directory structure?

Best regards,

Iñaki.