Re: [Bug-wget] Memory Leakage

[Tim Ruehsen]

Hi Thomas,

as you say, wget stores the links to traverse in memory. Also the links 
already traversed. That sums up after a while... but what are talking about 
here ? One million links with ~100 bytes sum to ~100MB. That's not very much 
for a modern machine.

To get a better idea:
- what version of wget do you use (wget --version)
- how long is wget running resp. how many pages are referred (rough estimate) 
?
- did you run a top (with pressing 'm' to sort for memory usage) ? if not, 
please do and report memory usage and speed of eating up memory.

You could send me the real URL you are using via PM so I can try to reproduce 
the behavior here. 

Tim

On Monday 14 March 2016 10:59:25 thomas hood wrote:
>   Hey,
> 
> Using wget recently to handle large amounts of data. Finding it frequently
> eats into memory quite a bit.Mar 12 06:32:08 ubuntu kernel:
> [1439386.493486] Out of memory: Kill process 63125 (wget) score 661 or
> sacrifice childMar 12 06:32:08 ubuntu kernel: [1439386.494595] Killed
> process 63125 (wget) total-vm:4206452kB, anon-rss:3161380kB,
> file-rss:0kBThese are the parameters I'm using; wget -c -nd -b -r -e
> robots=off --header="Accept: text/html" --user-agent="Mozilla/5.0 (Windows
> NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1" http://$URL/$3 2>&1 |
> grep '^--' 2>&1 | awk '{ print $3 }' | grep -v
> '\.\(css\|js\|png\|gif\|jpg\|JPG\)$' &Doing a bit of research on Google
> I've found other people have experienced this in the past one or two years.
> Can anyone shed light on the cause? Is it just wget storing the relative
> links it has to traverse in main memory? Obviously there's ways to manage
> that but I'm unsure if it's an issue of memory management.
> 
> Regards,Thomas Hood

signature.asc
Description: This is a digitally signed message part.