findutils-patches
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Findutils-patches] [PATCH] find memory leak


From: Bernhard Voelker
Subject: Re: [Findutils-patches] [PATCH] find memory leak
Date: Tue, 7 Feb 2017 02:04:37 +0100
User-agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0

On 02/06/2017 08:43 PM, Goffredo Baroncelli wrote:
> $ wc -c < /proc/self/mountinfo
> 3988

okay, that's not the reason for the huge memory consumption then.

> However I can't follow you.
> 
> I counted how many time free_file_system_list() is called; it is
> called 113890 times[*]. It seems to me that these functions are
> more often than you think.

outch, that's the reason!  We're coming closer.

> Below the backtrace: the first number is the number of times that 
> free_file_system_list is called. Then the chain of the function call (I had 
> to change the source to print the backtrace every time free_file_system_list 
> is called). Of course I grouped the backtraces on the basis of chain of calls.
> 
> 
> 16269 BT: ./find(print_backtrace+0x1f) ;./find(free_file_system_list+0x41) 
> ;./find(file_system_type_uncached+0xbc) ;./find(filesystem_type+0x72) 
> ;./find(pred_fstype+0x30) ;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
> ;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
> ;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
> ;./find(apply_predicate+0x8c) ;./find(pred_and+0x64) 
> ;./find(apply_predicate+0x8c) ;./find(pred_and+0x42) 
> ;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
> ;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
> ;./find(apply_predicate+0x8c) ;./find(visit+0x85) 
> ;./find(consider_visiting+0x66e) ;./find(find+0x1ad) 
> ;./find(process_all_startpoints+0x64) ;./find(main+0x1df) 
> ;/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) ;./find(_start+0x2a) 
> ;

[...]

> [*]
> Below some stat about my /lib/modules directory
> 
> $ find /lib/modules | wc -l
> 16269

Ha, I got the same factor: for each of those 16269 files, find invokes
the function filesystem_type() 7 times: this is the number of the given
23 file system types which can be found in '/proc/filesystems'; those not
loaded by the kernel are eliminated by the predicate parser - you can
check with 'find -O0 ...' where those unknown types are not optimized
away.

Okay, so back to the question why free_file_system_list() is invoked that
often: usually, filesystem_type() would cache the result ... and I'm assuming
that all of those 16269 files are on the same mounted file system.
A look into the underlying file_system_type_uncached() gives the answer:
either stat() of the mount point of /lib/modules fails, or the returned
'st_dev' number does not match the entry in the mount list - see the
following lines.

What do you get in the /tmp/find.strace.log file when you run the command
via strace:

  $ strace -e stat -vo /tmp/find.strace.log ./find ...

?

I'm getting 7x lines like this:

  stat("/", {st_dev=makedev(8, 2), ...

and then searching for the above major:minor (8, 2) device in the mountinfo
file unveils 'ext4' here (which is not pruned):

  $ grep ' 8:2 ' /proc/self/mountinfo
  35 0 8:2 / / rw,noatime shared:1 - ext4 /dev/sda2 rw,data=ordered

Somewhere in the above chain, find fails to determine the file system type
on your system and therefore re-tries for each entry.  Without the memleak
patch, it's clear that the process would grow excessively.
Can you confirm?

Thanks & have a nice day,
Berny




reply via email to

[Prev in Thread] Current Thread [Next in Thread]