[PATCH v3] Support madvise(MADV_DONTDUMP) when creating core dumps for qemu-user

[WorksButNotTested]

When running applications which make large (sparsely populated) address ranges
(e.g. when using address sanitizer with LibAFL) the inability to exclude these
regions from any core dump can result in very large files which fill the disk.
A coredump is obvously very useful for performing a post-mortem when fuzzing.

Whilst the man pages state that madvise provides only a hint (and hence can be
ignored), this patch adds support to handle MADV_DONTDUMP and set a
corresponding flag in the page flags, thus allowing QEMU to exclude these
regions from the core file.

Signed-off-by: WorksButNotTested 
<62701594+WorksButNotTested@users.noreply.github.com>
---
 include/exec/page-protection.h |  6 ++++++
 linux-user/elfload.c           |  4 ++++
 linux-user/mmap.c              | 18 ++++++++++++++++++
 3 files changed, 28 insertions(+)

diff --git a/include/exec/page-protection.h b/include/exec/page-protection.h
index c43231af8b..f8826d917e 100644
--- a/include/exec/page-protection.h
+++ b/include/exec/page-protection.h
@@ -38,4 +38,10 @@
  */
 #define PAGE_PASSTHROUGH 0x0800
 
+/*
+ * For linux-user, indicates that the page should not be included in a core 
+ * dump.
+ */
+#define PAGE_DONTDUMP   0x1000
+
 #endif
diff --git a/linux-user/elfload.c b/linux-user/elfload.c
index fbfdec2f17..41c46da055 100644
--- a/linux-user/elfload.c
+++ b/linux-user/elfload.c
@@ -4067,6 +4067,10 @@ static size_t vma_dump_size(target_ulong start, 
target_ulong end,
         return 0;
     }
 
+    if (flags & PAGE_DONTDUMP) {
+        return 0;
+    }
+
     /*
      * Usually we don't dump executable pages as they contain
      * non-writable code that debugger can read directly from
diff --git a/linux-user/mmap.c b/linux-user/mmap.c
index f88a80c31e..016063a8cf 100644
--- a/linux-user/mmap.c
+++ b/linux-user/mmap.c
@@ -1247,6 +1247,24 @@ abi_long target_madvise(abi_ulong start, abi_ulong 
len_in, int advice)
      */
     mmap_lock();
     switch (advice) {
+    case MADV_DONTDUMP:
+        if (len > 0) {
+            /*
+             * To set the page permissons, we must OR our new flags with the
+             * existing flags. Only mark the pages as PAGE_DONTDUMP if the
+             * entire range has the same flags. If any part of the range
+             * differs, we would need to process it one page at a time which
+             * might not be very performant. Since we are not obliged to 
respect
+             * this flag, we will support it for the most likely usage 
scenario.
+             * Note that we don't set PAGE_ANON, since this can only be set 
with
+             * new mappings.
+             */
+            int flg = page_get_flags(start);
+            if (page_check_range(start, len, flg)) {
+                page_set_flags(start, start + len - 1, PAGE_DONTDUMP | (flg & 
~PAGE_ANON) );
+            }
+        }
+        break;
     case MADV_WIPEONFORK:
     case MADV_KEEPONFORK:
         ret = -EINVAL;
-- 
2.43.0