[Gcl-devel] Re: brk emulation, unexec, macosx, openbsd, fedora

[Aurelien Chanudet]

Hi,

For the sake of clarity, I'm pasting here the output of my Mach-O 
object file analysis tool. Below is the memory layout of the raw 
executable file, as produced by the linker and before unexec. As you 
can see, this executable has only one __DATA segment, akin to the .data 
section in ELF parlance.

Load command 1
      cmd LC_SEGMENT
  cmdsize 668
  segname __TEXT
   vmaddr 0x00001000
   vmsize 0x00105000
  fileoff 0
 filesize 1069056
  maxprot rwx
 initprot r-x
   nsects 9
    flags (none)
Load command 2
      cmd LC_SEGMENT
  cmdsize 532
  segname __DATA
   vmaddr 0x00106000
   vmsize 0x00449000
  fileoff 1069056
 filesize 65536
  maxprot rwx
 initprot rw-
   nsects 7
    flags (none)
Load command 3
      cmd LC_SEGMENT
  cmdsize 56
  segname __LINKEDIT
   vmaddr 0x0054f000
   vmsize 0x00235000
  fileoff 1134592
 filesize 2312520
  maxprot rwx
 initprot r--
   nsects 0
    flags NORELOC

When the raw executable as output by the linker is run, I'm detecting 
that the sbrk emulation mechanism needs to be set up. At this stage, 
I'm creating a temporary executable containing a hand-crafted extra 
segment to manage my heap. I'm then using execv(3) to replace the raw 
executable by this temporary executable which will play the role of the 
former raw executable from here one. Below is the memory layout of this 
temporary executable. As you can see, the native __DATA segment size 
plus the hand-crafted one sums up to 0x00449000 + 0x1fbb7000 = 
0x20000000 = MAXPAGE*PAGESIZE. As you can also note, at this stage, the 
disk size of this second data segment is zero because no memory 
allocation has been performed yet.

Load command 1
      cmd LC_SEGMENT
  cmdsize 668
  segname __TEXT
   vmaddr 0x00001000
   vmsize 0x00105000
  fileoff 0
 filesize 1069056
  maxprot rwx
 initprot r-x
   nsects 9
    flags (none)
Load command 2
      cmd LC_SEGMENT
  cmdsize 532
  segname __DATA
   vmaddr 0x00106000
   vmsize 0x00449000
  fileoff 1069056
 filesize 4493312
  maxprot rwx
 initprot rw-
   nsects 7
    flags (none)
Load command 3
      cmd LC_SEGMENT
  cmdsize 124
  segname __DATA
   vmaddr 0x0054f000
   vmsize 0x1fbb7000
  fileoff 5562368
 filesize 0
  maxprot rwx
 initprot rwx
   nsects 1
    flags (none)
Load command 4
      cmd LC_SEGMENT
  cmdsize 56
  segname __LINKEDIT
   vmaddr 0x20106000
   vmsize 0x00235000
  fileoff 5562368
 filesize 2312520
  maxprot rwx
 initprot r--
   nsects 0
    flags NORELOC

All memory allocation takes place in this second data segment 
(including calls to malloc performed by third party libraries). The 
first data segment is frozen in the sense that its size will never 
vary. By contrast, the effective size of the second data segment is 
increasing as the breakpoint is being incremented. When unexec is run, 
the contents of the first data segment is being copied from memory to 
the dumped executable. Accordingly, the contents of the second data 
segment is being copied up to core_end (what's after core_end is just 
thrown away). This scheme is the best one I was able to devise in order 
to make sure the size of the image on disk is as small as possible. 
Below are typical values :

DBEGIN:          0x106000
mach_mapstart:   0x54f000 start of my second data segment
heap_end:       0x19c9000
core_end:       0x19ca000
mach_brkpt:     0xefd2000 emulated breakpoint value
mach_maplimit: 0x20106000 end of my second data segment

> Is the saved image size always at its maximal value?

If you're talking of the disk size, then the answer is no. If you're 
talking of the virtual memory size of the image, then the answer is 
yes. But this does not appear to be a performance issue because the 
virtual memory system is smart enough to know how to cope with 
allocated pages that will never get read from or written to (in fact, 
setting the saved image size to its maximal value simply means 
reserving space where nothing will ever happen in most cases).

> Do you create your own section in the executable for your heap?

Yes, as I tried to explain.

> Is there any way you know of persistently and incrementally
> storing and re-executing a gradually growing heap?

Apparently, the piece of software which is responsible for preparing 
the memory layout of an executable before calling the main entry point 
(on MacOS X, this piece of software is included in the dynamic loader), 
is rather flexible. It lets me decide what to load and where to load 
it. The fact that the heap is gradually growing is transparent for the 
dynamic loader because the virtual memory size of the second memory 
segment does not vary (whereas its file size does). However, had it 
varied, this would not have been much of an issue, as long as we can 
shift what comes after the heap towards higher addresses if needs be. 
On MacOS X, what comes after the heap are the symbol and string tables.

Hope this could cast some light on the issue !

Aurelien