Re: [Bug 1895080] [NEW] pgb_reserved_va: Assertion `addr == test' failed

[Alex Bennée]

Hansni Bu <1895080@bugs.launchpad.net> writes:

>> > No, it's not set by CentOS-7.5.
>> > Does it mean that we just cannot run the ELF in such a case? I've tried
>> > many times, the assert always fails. Maybe, we can blame CentOS-7.5.
>>
>> The trouble is without MAP_FIXED_NOREPLACE we are at the mercy of the
>> host kernel to allow the address request to be honoured. A plain
>> MAP_FIXED won't do as it can clober existing mappings. In theory a
>> suitable hole has been identified but sometimes the kernel makes a
>> decision to offset the suggested mapping for it's own reasons.
>>
>
> MAP_FIXED_NOREPLACE is quite a new feature.
>
>
>> > BTW: with the option "-p 65536", the case runs successfully.
>>
>> Hmm interesting. I wonder if we are seeing a fail due to mmap_min_addr?
>> What does:
>>
>>   /proc/sys/vm/mmap_min_addr
>>
>> give you on the system?
>>
>
> It gives me 4096. And guest_base has this value. Maybe that's the strange
> point. mmap_min_addr give us 0x1000. While we are requesting this address,
> the kernel gives us 0x10000.

Yeah the meaning of mmap_min_addr should be exactly that which is odd
why the kernel doesn't honour the request. That said it seems to be a
user tweakable knob so you could just up it.

>>
>> You can manually set the reserved_va and the base address using -R and
>> -B although that is more of a developer work around. I think moving the
>> assert to the condition above would be an improvement just because it
>> tells us what the requested base address was and what the kernel decided
>> to give us.
>>
>
> Setting guest_base with -B to 0x10000 works. Tried some -R values, no luck.
> Agree to print a more hintful message.

OK - one other test you try is running inside a chroot which *does not*
have visibility of /proc. That will cause it to fall back to the old
probing technique of trying multiple mmap operations to find the hole. I
suspect that works because the attempt to map 0x1000 would fail the
check. The reason we default to probing /proc/self/map now is that brute
force technique falls over when trying to probe for a hole on 64 bit
systems, especially when built with stack protection/clang debug
instrumentation.

In the meantime I'll spin up a patch to improve the failure mode.

>
>
>>
>> >
>> > On Fri, Sep 11, 2020 at 5:50 PM Alex Bennée <1895080@bugs.launchpad.net>
>> > wrote:
>> >
>> >> Alex Bennée <alex.bennee@linaro.org> writes:
>> >>
>> >> > Laurent Vivier <laurent@vivier.eu> writes:
>> >> >
>> >> <snip>
>> >> >>> Then trying qemu-riscv32 with a simple ELF, I get:
>> >> >>> linux-user/elfload.c:2341: pgb_reserved_va: Assertion `addr == test'
>> >> failed.
>> >> >>>
>> >> >>> strace shows that:
>> >> >>> mmap(0x1000, 4294963200, PROT_NONE,
>> >> MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x10000
>> >> >>> write(2, "qemu-riscv32: ../qemu.git/linux-"..., 103qemu-riscv32:
>> >> ../qemu.git/linux-user/elfload.c:2341: pgb_reserved_va: Assertion `addr
>> ==
>> >> test' failed.
>> >> >>> ) = 103
>> >> >>>
>> >> >>> The source code is in the function pgb_reserved_va (linux-
>> >> >>> user/elfload.c). I think mmap cannot guarantee that the returned
>> >> pointer
>> >> >>> (test) equals to the parameter of addr. So is this a bug to assert
>> >> (addr
>> >> >>> == test)?
>> >> >>
>> >> > I'm assuming CentOS 7.5 actually has a definition for
>> >> > MAP_FIXED_NOREPLACE which should ensure we get what we asked for -
>> >> > otherwise we are in the position of hoping the kernel honours what we
>> >> > asked for.
>> >>
>> >> Doh re-reading I see it's not set in the strace output. Maybe we should
>> >> promote the assert case to the failure leg so we have:
>> >>
>> >>     if (addr == MAP_FAILED || addr != test) {
>> >>         error_report(...)
>> >>     }
>> >>
>> >> so we at least fail with a user friendly error rather than an abort?
>> >>
>> >> --
>> >> Alex Bennée
>> >>
>> >> --
>> >> You received this bug notification because you are subscribed to the bug
>> >> report.
>> >> https://bugs.launchpad.net/bugs/1895080
>> >>
>> >> Title:
>> >>   pgb_reserved_va: Assertion `addr == test' failed
>> >>
>> >> Status in QEMU:
>> >>   New
>> >>
>> >> Bug description:
>> >>   This problem occurs on CentOS-7.5 (64-bit) with qemu-5.1.0, qemu head
>> >>   (commit 9435a8b3dd35f1f926f1b9127e8a906217a5518a) for riscv32-linux-
>> >>   user.
>> >>
>> >>   Firstly, compile fails:
>> >>   Compiling C object
>> libqemu-riscv32-linux-user.fa.p/linux-user_strace.c.o
>> >>   ../qemu.git/linux-user/strace.c:1210:18: error: ‘FALLOC_FL_KEEP_SIZE’
>> >> undeclared here (not in a function)
>> >>        FLAG_GENERIC(FALLOC_FL_KEEP_SIZE),
>> >>
>> >>   I have to add below include to linux-user/strace.c
>> >>   diff --git a/linux-user/strace.c b/linux-user/strace.c
>> >>   index 11fea14fba..22e51d4a8a 100644
>> >>   --- a/linux-user/strace.c
>> >>   +++ b/linux-user/strace.c
>> >>   @@ -7,6 +7,7 @@
>> >>    #include <sys/mount.h>
>> >>    #include <arpa/inet.h>
>> >>    #include <netinet/tcp.h>
>> >>   +#include <linux/falloc.h>
>> >>    #include <linux/if_packet.h>
>> >>    #include <linux/netlink.h>
>> >>    #include <sched.h>
>> >>
>> >>   Then trying qemu-riscv32 with a simple ELF, I get:
>> >>   linux-user/elfload.c:2341: pgb_reserved_va: Assertion `addr == test'
>> >> failed.
>> >>
>> >>   strace shows that:
>> >>   mmap(0x1000, 4294963200, PROT_NONE,
>> >> MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x10000
>> >>   write(2, "qemu-riscv32: ../qemu.git/linux-"..., 103qemu-riscv32:
>> >> ../qemu.git/linux-user/elfload.c:2341: pgb_reserved_va: Assertion `addr
>> ==
>> >> test' failed.
>> >>   ) = 103
>> >>
>> >>   The source code is in the function pgb_reserved_va (linux-
>> >>   user/elfload.c). I think mmap cannot guarantee that the returned
>> >>   pointer (test) equals to the parameter of addr. So is this a bug to
>> >>   assert (addr == test)?
>> >>
>> >>   Attached configure script and test ELF file.
>> >>
>> >>   Thanks.
>> >>
>> >> To manage notifications about this bug go to:
>> >> https://bugs.launchpad.net/qemu/+bug/1895080/+subscriptions
>> >>
>>
>>
>> --
>> Alex Bennée
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1895080
>>
>> Title:
>>   pgb_reserved_va: Assertion `addr == test' failed
>>
>> Status in QEMU:
>>   New
>>
>> Bug description:
>>   This problem occurs on CentOS-7.5 (64-bit) with qemu-5.1.0, qemu head
>>   (commit 9435a8b3dd35f1f926f1b9127e8a906217a5518a) for riscv32-linux-
>>   user.
>>
>>   Firstly, compile fails:
>>   Compiling C object libqemu-riscv32-linux-user.fa.p/linux-user_strace.c.o
>>   ../qemu.git/linux-user/strace.c:1210:18: error: ‘FALLOC_FL_KEEP_SIZE’
>> undeclared here (not in a function)
>>        FLAG_GENERIC(FALLOC_FL_KEEP_SIZE),
>>
>>   I have to add below include to linux-user/strace.c
>>   diff --git a/linux-user/strace.c b/linux-user/strace.c
>>   index 11fea14fba..22e51d4a8a 100644
>>   --- a/linux-user/strace.c
>>   +++ b/linux-user/strace.c
>>   @@ -7,6 +7,7 @@
>>    #include <sys/mount.h>
>>    #include <arpa/inet.h>
>>    #include <netinet/tcp.h>
>>   +#include <linux/falloc.h>
>>    #include <linux/if_packet.h>
>>    #include <linux/netlink.h>
>>    #include <sched.h>
>>
>>   Then trying qemu-riscv32 with a simple ELF, I get:
>>   linux-user/elfload.c:2341: pgb_reserved_va: Assertion `addr == test'
>> failed.
>>
>>   strace shows that:
>>   mmap(0x1000, 4294963200, PROT_NONE,
>> MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x10000
>>   write(2, "qemu-riscv32: ../qemu.git/linux-"..., 103qemu-riscv32:
>> ../qemu.git/linux-user/elfload.c:2341: pgb_reserved_va: Assertion `addr ==
>> test' failed.
>>   ) = 103
>>
>>   The source code is in the function pgb_reserved_va (linux-
>>   user/elfload.c). I think mmap cannot guarantee that the returned
>>   pointer (test) equals to the parameter of addr. So is this a bug to
>>   assert (addr == test)?
>>
>>   Attached configure script and test ELF file.
>>
>>   Thanks.
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/qemu/+bug/1895080/+subscriptions
>>


-- 
Alex Bennée