Re: [Findutils-patches] [PATCH] find memory leak

[Goffredo Baroncelli]

On 02/05/2017 11:52 PM, Bernhard Voelker wrote:
> On 02/03/2017 07:06 PM, Goffredo Baroncelli wrote:
>> On 2017-02-02 23:31, Bernhard Voelker wrote:
>>> yes, I did.  But my /lib/modules only has  9470 files, 
>>
>> my one has 12000 files; so the numbers are comparables.
>>
>>
>>
>>> so the memory usage without
>>> the patches was only
>>>     Maximum resident set size (kbytes): 4136
>>> and with the patches it's down to
>>>     Maximum resident set size (kbytes): 2556
>>
>> Which tools are you using to get these info ?
> 
> That's simply /usr/bin/time -v ...
> 
>>> So I'm wondering why your find uses 110M.  Even if I try for '/' (which has
>>> 1281671 files output by that find command), then only 16-18M (without/with 
>>> the patch)
>>> are used here:
>>>     Maximum resident set size (kbytes): 17892
>>>     Maximum resident set size (kbytes): 15800
>>>
>>> Thus said, 110M sound excessive.
>>>
> 
>> This is not clear to me: "time" states that the maximum memory occupation is 
>> near 110MB (see above). With the patches "time" states that it requires only 
>> 2.7MB
>>
>> /usr/bin/time ./find /lib/modules/ -ignore_readdir_race      \( -fstype NFS 
>> -o -fstype nfs -o -fstype nfs4 -o -fstype afs -o -fstype binfmt_misc -o 
>> -fstype proc -o -fstype smbfs -o -fstype autofs -o -fstype iso9660 -o 
>> -fstype ncpfs -o -fstype coda -o -fstype devpts -o -fstype ftpfs -o -fstype 
>> devfs -o -fstype mfs -o -fstype shfs -o -fstype sysfs -o -fstype cifs -o 
>> -fstype lustre_lite -o -fstype tmpfs -o -fstype usbfs -o -fstype udf -o 
>> -fstype ocfs2 -o      -type d -false  \) -prune -o -print0  | wc -l
>> 0.09user 0.09system 0:00.23elapsed 79%CPU (0avgtext+0avgdata 
>> 2728maxresident)k
>> 1088inputs+0outputs (1major+162minor)pagefaults 0swaps
> 
> That decrease of mem usage looks great.
> 
>> the differences should be the leak....
>> are we looking to the same info ?
> 
> Hmm, if really the 23x -fstype option had a leak and lead to ~110MB of
> memory, then your /proc/self/mountinfo must have about 4.5MB.  How big
> is it on your system ("wc -c < /proc/self/mountinfo")?

$ wc -c < /proc/self/mountinfo
3988

However I can't follow you.

I counted how many time free_file_system_list() is called; it is called 113890 
times[*]. It seems to me that these functions are more often than you think.

Below the backtrace: the first number is the number of times that 
free_file_system_list is called. Then the chain of the function call (I had to 
change the source to print the backtrace every time free_file_system_list is 
called). Of course I grouped the backtraces on the basis of chain of calls.


16269 BT: ./find(print_backtrace+0x1f) ;./find(free_file_system_list+0x41) 
;./find(file_system_type_uncached+0xbc) ;./find(filesystem_type+0x72) 
;./find(pred_fstype+0x30) ;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x64) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(visit+0x85) 
;./find(consider_visiting+0x66e) ;./find(find+0x1ad) 
;./find(process_all_startpoints+0x64) ;./find(main+0x1df) 
;/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) ;./find(_start+0x2a) ;
16269 BT: ./find(print_backtrace+0x1f) ;./find(free_file_system_list+0x41) 
;./find(file_system_type_uncached+0xbc) ;./find(filesystem_type+0x72) 
;./find(pred_fstype+0x30) ;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x64) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(visit+0x85) 
;./find(consider_visiting+0x66e) ;./find(find+0x1ad) 
;./find(process_all_startpoints+0x64) ;./find(main+0x1df) 
;/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) ;./find(_start+0x2a) ;
16269 BT: ./find(print_backtrace+0x1f) ;./find(free_file_system_list+0x41) 
;./find(file_system_type_uncached+0xbc) ;./find(filesystem_type+0x72) 
;./find(pred_fstype+0x30) ;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x64) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(visit+0x85) 
;./find(consider_visiting+0x66e) ;./find(find+0x1ad) 
;./find(process_all_startpoints+0x64) ;./find(main+0x1df) 
;/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) ;./find(_start+0x2a) ;
16269 BT: ./find(print_backtrace+0x1f) ;./find(free_file_system_list+0x41) 
;./find(file_system_type_uncached+0xbc) ;./find(filesystem_type+0x72) 
;./find(pred_fstype+0x30) ;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x64) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(visit+0x85) 
;./find(consider_visiting+0x66e) ;./find(find+0x1ad) 
;./find(process_all_startpoints+0x64) ;./find(main+0x1df) 
;/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) ;./find(_start+0x2a) ;
7 BT: ./find(print_backtrace+0x1f) ;./find(free_file_system_list+0x41) 
;./find(file_system_type_uncached+0xbc) ;./find(filesystem_type+0x72) 
;./find(estimate_fstype_success_rate+0x4d) ;./find(parse_fstype+0x7f) 
;./find(build_expression_tree+0x2aa) ;./find(main+0x1b7) 
;/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) ;./find(_start+0x2a) ;
16269 BT: ./find(print_backtrace+0x1f) ;./find(free_file_system_list+0x41) 
;./find(file_system_type_uncached+0xbc) ;./find(filesystem_type+0x72) 
;./find(pred_fstype+0x30) ;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x64) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(visit+0x85) 
;./find(consider_visiting+0x66e) ;./find(find+0x1ad) 
;./find(process_all_startpoints+0x64) ;./find(main+0x1df) 
;/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) ;./find(_start+0x2a) ;
16269 BT: ./find(print_backtrace+0x1f) ;./find(free_file_system_list+0x41) 
;./find(file_system_type_uncached+0xbc) ;./find(filesystem_type+0x72) 
;./find(pred_fstype+0x30) ;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x64) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(visit+0x85) 
;./find(consider_visiting+0x66e) ;./find(find+0x1ad) 
;./find(process_all_startpoints+0x64) ;./find(main+0x1df) 
;/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) ;./find(_start+0x2a) ;
16269 BT: ./find(print_backtrace+0x1f) ;./find(free_file_system_list+0x41) 
;./find(file_system_type_uncached+0xbc) ;./find(filesystem_type+0x72) 
;./find(pred_fstype+0x30) ;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x64) 
;./find(apply_predicate+0x8c) ;./find(pred_and+0x42) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x67) 
;./find(apply_predicate+0x8c) ;./find(pred_or+0x42) 
;./find(apply_predicate+0x8c) ;./find(visit+0x85) 
;./find(consider_visiting+0x66e) ;./find(find+0x1ad) 
;./find(process_all_startpoints+0x64) ;./find(main+0x1df) 
;/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) ;./find(_start+0x2a) ;




> 
>> Which find version are you testing, maybe that there are other variables 
>> which we aren't considering.
> 
> I was testing both /usr/bin/find on openSUSE:Tumbleweed, and the
> latest built from the findutils.git repo.
> 
> Have a nice day,
> Berny
> 

[*]
Below some stat about my /lib/modules directory

$ find /lib/modules | wc -l
16269
$ find /lib/modules -type f | wc -l
13182
$ find /lib/modules -type d | wc -l
3083



-- 
gpg @keyserver.linux.it: Goffredo Baroncelli <kreijackATinwind.it>
Key fingerprint BBF5 1610 0B64 DAC6 5F7D  17B2 0EDA 9B37 8B82 E0B5