[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Findutils-patches] [PATCH] find memory leak
From: |
Bernhard Voelker |
Subject: |
Re: [Findutils-patches] [PATCH] find memory leak |
Date: |
Tue, 7 Feb 2017 02:04:37 +0100 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0 |
On 02/06/2017 08:43 PM, Goffredo Baroncelli wrote:
> $ wc -c < /proc/self/mountinfo
> 3988
okay, that's not the reason for the huge memory consumption then.
> However I can't follow you.
>
> I counted how many time free_file_system_list() is called; it is
> called 113890 times[*]. It seems to me that these functions are
> more often than you think.
outch, that's the reason! We're coming closer.
> Below the backtrace: the first number is the number of times that
> free_file_system_list is called. Then the chain of the function call (I had
> to change the source to print the backtrace every time free_file_system_list
> is called). Of course I grouped the backtraces on the basis of chain of calls.
>
>
> 16269 BT: ./find(print_backtrace+0x1f) ;./find(free_file_system_list+0x41)
> ;./find(file_system_type_uncached+0xbc) ;./find(filesystem_type+0x72)
> ;./find(pred_fstype+0x30) ;./find(apply_predicate+0x8c) ;./find(pred_or+0x67)
> ;./find(apply_predicate+0x8c) ;./find(pred_or+0x42)
> ;./find(apply_predicate+0x8c) ;./find(pred_or+0x42)
> ;./find(apply_predicate+0x8c) ;./find(pred_and+0x64)
> ;./find(apply_predicate+0x8c) ;./find(pred_and+0x42)
> ;./find(apply_predicate+0x8c) ;./find(pred_or+0x67)
> ;./find(apply_predicate+0x8c) ;./find(pred_or+0x42)
> ;./find(apply_predicate+0x8c) ;./find(visit+0x85)
> ;./find(consider_visiting+0x66e) ;./find(find+0x1ad)
> ;./find(process_all_startpoints+0x64) ;./find(main+0x1df)
> ;/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) ;./find(_start+0x2a)
> ;
[...]
> [*]
> Below some stat about my /lib/modules directory
>
> $ find /lib/modules | wc -l
> 16269
Ha, I got the same factor: for each of those 16269 files, find invokes
the function filesystem_type() 7 times: this is the number of the given
23 file system types which can be found in '/proc/filesystems'; those not
loaded by the kernel are eliminated by the predicate parser - you can
check with 'find -O0 ...' where those unknown types are not optimized
away.
Okay, so back to the question why free_file_system_list() is invoked that
often: usually, filesystem_type() would cache the result ... and I'm assuming
that all of those 16269 files are on the same mounted file system.
A look into the underlying file_system_type_uncached() gives the answer:
either stat() of the mount point of /lib/modules fails, or the returned
'st_dev' number does not match the entry in the mount list - see the
following lines.
What do you get in the /tmp/find.strace.log file when you run the command
via strace:
$ strace -e stat -vo /tmp/find.strace.log ./find ...
?
I'm getting 7x lines like this:
stat("/", {st_dev=makedev(8, 2), ...
and then searching for the above major:minor (8, 2) device in the mountinfo
file unveils 'ext4' here (which is not pruned):
$ grep ' 8:2 ' /proc/self/mountinfo
35 0 8:2 / / rw,noatime shared:1 - ext4 /dev/sda2 rw,data=ordered
Somewhere in the above chain, find fails to determine the file system type
on your system and therefore re-tries for each entry. Without the memleak
patch, it's clear that the process would grow excessively.
Can you confirm?
Thanks & have a nice day,
Berny
- Re: [Findutils-patches] [PATCH] find memory leak, Bernhard Voelker, 2017/02/01
- Re: [Findutils-patches] [PATCH] find memory leak, Goffredo Baroncelli, 2017/02/02
- Re: [Findutils-patches] [PATCH] find memory leak, Bernhard Voelker, 2017/02/02
- Re: [Findutils-patches] [PATCH] find memory leak, Goffredo Baroncelli, 2017/02/03
- Re: [Findutils-patches] [PATCH] find memory leak, Bernhard Voelker, 2017/02/05
- Re: [Findutils-patches] [PATCH] find memory leak, Goffredo Baroncelli, 2017/02/06
- Re: [Findutils-patches] [PATCH] find memory leak,
Bernhard Voelker <=
- Re: [Findutils-patches] [PATCH] find memory leak, Goffredo Baroncelli, 2017/02/07
- Re: [Findutils-patches] [PATCH] find memory leak, Bernhard Voelker, 2017/02/07
- Re: [Findutils-patches] [PATCH] find memory leak, Dale R. Worley, 2017/02/08
- Re: [Findutils-patches] [PATCH] find memory leak, Bernhard Voelker, 2017/02/08
- Re: [Findutils-patches] [PATCH] find memory leak, Goffredo Baroncelli, 2017/02/08
- Re: [Findutils-patches] [PATCH] find memory leak, Bernhard Voelker, 2017/02/08
- Re: [Findutils-patches] [PATCH] find memory leak, Dale R. Worley, 2017/02/09
- Re: [Findutils-patches] [PATCH] find memory leak, Goffredo Baroncelli, 2017/02/09
- Re: [Findutils-patches] [PATCH] find memory leak, Dale R. Worley, 2017/02/10
- Re: [Findutils-patches] [PATCH] find memory leak, Goffredo Baroncelli, 2017/02/12