[bug-gawk] Increased RAM memory usage in gawk 5.0.1 compared to gawk 4.1.4

[Finn Magnusson]

Dear gawk developers

Our team recently started using gawk 5.0.1 and found that the RAM memory usage 
seems to have increased compared to gawk version 4.1.4, which we were using 
before.
For info, we develop a gawk-based telecom application (over 110,000 lines of 
gawk code) used for operation of cellular networks. This application stores lot 
of data so each program instance uses over 200 MB of RAM, with peaks up to 1 GB 
in certain cases. As users sometimes execute up to a hundred of instances of 
this application in parallel the RAM memory usage is of concern.

Firstly, we noticed an increase in memory due to the introduction in gawk 4.2.0 
of the feature for easier toggling of the IGNORECASE variable. We made a patch 
to disable this feature (attached file "gawk-5.0.1-IGNORECASE.patch), and this 
helped to reduce memory usage.

Secondly, we noticed (even after applying the patch) that in certain cases gawk 
5.0.1 seemed to be leaking memory. 
To demonstrate it, please see the below test. The test script 
(parse_mofile.awk) parses the same file over and over and stores the data in a 
hashtable. 
Attached are the relevant files in case you'd like to reproduce the issue on 
your end and you can see the command line below for executing the script.
As we can see, with gawk 4.1.4, the memory usage does not increase very much 
for each iteration and stays quite constant after the 10th iteration (at 1,39 
GB RAM).
Whereas with gawk 5.0.1, the memory usage keeps increasing continuously. After 
10 iterations, it is 14% higher than with gawk 4.1.4, after 20 iterations it is 
23% higher, and after 30 iterations it is 30% higher.
It can be interesting to note that when setting the parameter delmib to 1 
(which deletes the main hashtable after iteration) then the memory usage stays 
constant in both cases.

Please let me know if there is any additional information or clarification 
needed and a very big thank you for your great work in making gawk available to 
the world. 

Best regards
Finn

eselivm2v1164l [11:20] [/home/eanzmagn] -> gawk414/gawk-4.1.4/gawk -f 
parse_mofile.awk -v modump=10.220.33.18_153125.log.gz -v iterations=30 
-vdelmib=0
starting the execution of parse_mofile in isolation which will be iterating 30 
time[s] ...
Iteration: 
1...................................................................................................
 RSS=1222496
Iteration: 
2...................................................................................................
 RSS=1290996
Iteration: 
3...................................................................................................
 RSS=1320036
Iteration: 
4...................................................................................................
 RSS=1340052
Iteration: 
5...................................................................................................
 RSS=1354824
Iteration: 
6...................................................................................................
 RSS=1365992
Iteration: 
7...................................................................................................
 RSS=1375268
Iteration: 
8...................................................................................................
 RSS=1382212
Iteration: 
9...................................................................................................
 RSS=1387144
Iteration: 
10...................................................................................................
 RSS=1389108
Iteration: 
11...................................................................................................
 RSS=1391032
Iteration: 
12...................................................................................................
 RSS=1391952
Iteration: 
13...................................................................................................
 RSS=1392576
Iteration: 
14...................................................................................................
 RSS=1393272
Iteration: 
15...................................................................................................
 RSS=1393436
Iteration: 
16...................................................................................................
 RSS=1393872
Iteration: 
17...................................................................................................
 RSS=1393872
Iteration: 
18...................................................................................................
 RSS=1394056
Iteration: 
19...................................................................................................
 RSS=1394056
Iteration: 
20...................................................................................................
 RSS=1394384
Iteration: 
21...................................................................................................
 RSS=1394384
Iteration: 
22...................................................................................................
 RSS=1394384
Iteration: 
23...................................................................................................
 RSS=1394384
Iteration: 
24...................................................................................................
 RSS=1394384
Iteration: 
25...................................................................................................
 RSS=1394384
Iteration: 
26...................................................................................................
 RSS=1394580
Iteration: 
27...................................................................................................
 RSS=1394672
Iteration: 
28...................................................................................................
 RSS=1394672
Iteration: 
29...................................................................................................
 RSS=1394672
Iteration: 
30...................................................................................................
 RSS=1394796

seliius01071 [11:22] [/home/eanzmagn] -> gawk501/gawk-5.0.1/gawk -f 
parse_mofile.awk -v modump=10.220.33.18_153125.log.gz -v iterations=30 
-vdelmib=0
starting the execution of parse_mofile in isolation which will be iterating 30 
time[s] ...
Iteration: 
1...................................................................................................
 RSS=1290080
Iteration: 
2...................................................................................................
 RSS=1361808
Iteration: 
3...................................................................................................
 RSS=1399140
Iteration: 
4...................................................................................................
 RSS=1435264
Iteration: 
5...................................................................................................
 RSS=1468964
Iteration: 
6...................................................................................................
 RSS=1500824
Iteration: 
7...................................................................................................
 RSS=1530220
Iteration: 
8...................................................................................................
 RSS=1553668
Iteration: 
9...................................................................................................
 RSS=1574656
Iteration: 
10...................................................................................................
 RSS=1593976
Iteration: 
11...................................................................................................
 RSS=1612012
Iteration: 
12...................................................................................................
 RSS=1629140
Iteration: 
13...................................................................................................
 RSS=1645132
Iteration: 
14...................................................................................................
 RSS=1659960
Iteration: 
15...................................................................................................
 RSS=1674040
Iteration: 
16...................................................................................................
 RSS=1687364
Iteration: 
17...................................................................................................
 RSS=1699844
Iteration: 
18...................................................................................................
 RSS=1711704
Iteration: 
19...................................................................................................
 RSS=1722820
Iteration: 
20...................................................................................................
 RSS=1733400
Iteration: 
21...................................................................................................
 RSS=1743412
Iteration: 
22...................................................................................................
 RSS=1753180
Iteration: 
23...................................................................................................
 RSS=1762448
Iteration: 
24...................................................................................................
 RSS=1771268
Iteration: 
25...................................................................................................
 RSS=1780140
Iteration: 
26...................................................................................................
 RSS=1788656
Iteration: 
27...................................................................................................
 RSS=1796780
Iteration: 
28...................................................................................................
 RSS=1804956
Iteration: 
29...................................................................................................
 RSS=1812652
Iteration: 
30...................................................................................................
 RSS=1820068

Below is with delmib=1 (the main hashtable "mibtable" is deleted after each 
iteration, then there is no memory leak)
eselivm2v1164l [14:08] [/home/eanzmagn] -> gawk414/gawk-4.1.4/gawk -f 
parse_mofile.awk -v modump=10.220.33.18_153125.log.gz -v iterations=5 -vdelmib=1
starting the execution of parse_mofile in isolation which will be iterating 5 
time[s] ...
Iteration: 
1...................................................................................................
 RSS=1222496
Iteration: 
2...................................................................................................
 RSS=1267640
Iteration: 
3...................................................................................................
 RSS=1267640
Iteration: 
4...................................................................................................
 RSS=1267640
Iteration: 
5...................................................................................................
 RSS=1267640

seliius01071 [10:54] [/home/eanzmagn] -> gawk501/gawk-5.0.1/gawk -f 
parse_mofile.awk -v modump=10.220.33.18_153125.log.gz -v iterations=5 -vdelmib=1
starting the execution of parse_mofile in isolation which will be iterating 5 
time[s] ...
Iteration: 
1...................................................................................................
 RSS=1290080
Iteration: 
2...................................................................................................
 RSS=1335400
Iteration: 
3...................................................................................................
 RSS=1335400
Iteration: 
4...................................................................................................
 RSS=1335400
Iteration: 
5...................................................................................................
 RSS=1335400

parse_mofile.awk
Description: Binary data

10.220.33.18_153125.log.gz
Description: GNU Zip compressed data

gawk-5.0.1-IGNORECASE.patch
Description: Binary data