zzuf :: memory limits established with setrlimit are not applied

[Diego Nieto Cid]

Hi,

One of the zzuf tests involves setting a memory limit with `setrlimit` and then
running a program to exhaust the available memory. While zzuf does
some fancy forking and stdin fuzzing before running such a program, I
could manage to reduce the code to a simpler version [2].

On GNU/Linux the malloc call fails and returns NULL, then the program
goes on and get a SEGFAULT (weird way of testing things :/ )

---->8-------->8----
(gdb) run
Starting program: /home/diego/dev/hurd/zzuf/tests/bug-memory
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

Breakpoint 1, main () at bug-memory.c:24
24     rlim.rlim_cur = 256 * 1024 * 1024;
(gdb) n
25     rlim.rlim_max = 256 * 1024 * 1024;
(gdb)
26     err = setrlimit(ZZUF_RLIMIT_MEM, &rlim);
(gdb)
27     if (err == -1)
(gdb)
34     volatile char *tmp = malloc(375390457);
(gdb)
35     for (int i = 0; i < 1024; i++)
(gdb) p tmp
$1 = 0x0
(gdb) n
36         tmp[0x90 * 1024 * i] = i;
(gdb) n

Program received signal SIGSEGV, Segmentation fault.
0x0000555555555208 in main () at bug-memory.c:36
36         tmp[0x90 * 1024 * i] = i;
---->8-------->8----


However, on GNU/Hurd the limit is not applied (?) and the malloc
returns a valid pointer:

---->8-------->8----
(gdb) run
Starting program: /home/demo/zzuf/tests/bug-memory
[New Thread 842.5]

Thread 4 hit Breakpoint 1, main () at bug-memory.c:24
24          rlim.rlim_cur = 256 * 1024 * 1024;
(gdb) n
25          rlim.rlim_max = 256 * 1024 * 1024;
(gdb)
26          err = setrlimit(ZZUF_RLIMIT_MEM, &rlim);
(gdb)
27          if (err == -1)
(gdb)
34          volatile char *tmp = malloc(375390457);
(gdb)
35          for (int i = 0; i < 1024; i++)
(gdb) p tmp
$1 = 0x1012e8010 ""
(gdb)
---->8-------->8----

Is this a known issue? (besides bug 43320 which involves a child process [1])

--

[1] https://savannah.gnu.org/bugs/?43320
[2] I attach the source file and copy the build output here:

    demo@debian:~/zzuf/tests$ CFLAGS="-g -ggdb" make  bug-memory
    cc -g -ggdb    bug-memory.c   -o bug-memory
    bug-memory.c:10:2: warning: #warning using RLIMIT_AS [-Wcpp]
        10 | #warning using RLIMIT_AS
           |  ^~~~~~~

setrlimit-bug.c
Description: Text document