Re: [Qemu-devel] [PATCH v3 0/4] Introduce the microvm machine type

[Sergio Lopez]

Stefan Hajnoczi <address@hidden> writes:

> On Thu, Jul 18, 2019 at 05:21:46PM +0200, Sergio Lopez wrote:
>> 
>> Stefan Hajnoczi <address@hidden> writes:
>> 
>> > On Tue, Jul 02, 2019 at 02:11:02PM +0200, Sergio Lopez wrote:
>> >> Microvm is a machine type inspired by both NEMU and Firecracker, and
>> >> constructed after the machine model implemented by the latter.
>> >> 
>> >> It's main purpose is providing users a KVM-only machine type with fast
>> >> boot times, minimal attack surface (measured as the number of IO ports
>> >> and MMIO regions exposed to the Guest) and small footprint (specially
>> >> when combined with the ongoing QEMU modularization effort).
>> >> 
>> >> Normally, other than the device support provided by KVM itself,
>> >> microvm only supports virtio-mmio devices. Microvm also includes a
>> >> legacy mode, which adds an ISA bus with a 16550A serial port, useful
>> >> for being able to see the early boot kernel messages.
>> >> 
>> >> Microvm only supports booting PVH-enabled Linux ELF images. Booting
>> >> other PVH-enabled kernels may be possible, but due to the lack of ACPI
>> >> and firmware, we're relying on the command line for specifying the
>> >> location of the virtio-mmio transports. If there's an interest on
>> >> using this machine type with other kernels, we'll try to find some
>> >> kind of middle ground solution.
>> >> 
>> >> This is the list of the exposed IO ports and MMIO regions when running
>> >> in non-legacy mode:
>> >> 
>> >> address-space: memory
>> >>     00000000d0000000-00000000d00001ff (prio 0, i/o): virtio-mmio
>> >>     00000000d0000200-00000000d00003ff (prio 0, i/o): virtio-mmio
>> >>     00000000d0000400-00000000d00005ff (prio 0, i/o): virtio-mmio
>> >>     00000000d0000600-00000000d00007ff (prio 0, i/o): virtio-mmio
>> >>     00000000d0000800-00000000d00009ff (prio 0, i/o): virtio-mmio
>> >>     00000000d0000a00-00000000d0000bff (prio 0, i/o): virtio-mmio
>> >>     00000000d0000c00-00000000d0000dff (prio 0, i/o): virtio-mmio
>> >>     00000000d0000e00-00000000d0000fff (prio 0, i/o): virtio-mmio
>> >>     00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi
>> >> 
>> >> address-space: I/O
>> >>   0000000000000000-000000000000ffff (prio 0, i/o): io
>> >>     0000000000000020-0000000000000021 (prio 0, i/o): kvm-pic
>> >>     0000000000000040-0000000000000043 (prio 0, i/o): kvm-pit
>> >>     000000000000007e-000000000000007f (prio 0, i/o): kvmvapic
>> >>     00000000000000a0-00000000000000a1 (prio 0, i/o): kvm-pic
>> >>     00000000000004d0-00000000000004d0 (prio 0, i/o): kvm-elcr
>> >>     00000000000004d1-00000000000004d1 (prio 0, i/o): kvm-elcr
>> >> 
>> >> A QEMU instance with the microvm machine type can be invoked this way:
>> >> 
>> >>  - Normal mode:
>> >> 
>> >> qemu-system-x86_64 -M microvm -m 512m -smp 2 \
>> >>  -kernel vmlinux -append "console=hvc0 root=/dev/vda" \
>> >>  -nodefaults -no-user-config \
>> >>  -chardev pty,id=virtiocon0,server \
>> >>  -device virtio-serial-device \
>> >>  -device virtconsole,chardev=virtiocon0 \
>> >>  -drive id=test,file=test.img,format=raw,if=none \
>> >>  -device virtio-blk-device,drive=test \
>> >>  -netdev tap,id=tap0,script=no,downscript=no \
>> >>  -device virtio-net-device,netdev=tap0
>> >> 
>> >>  - Legacy mode:
>> >> 
>> >> qemu-system-x86_64 -M microvm,legacy -m 512m -smp 2 \
>> >>  -kernel vmlinux -append "console=ttyS0 root=/dev/vda" \
>> >>  -nodefaults -no-user-config \
>> >>  -drive id=test,file=test.img,format=raw,if=none \
>> >>  -device virtio-blk-device,drive=test \
>> >>  -netdev tap,id=tap0,script=no,downscript=no \
>> >>  -device virtio-net-device,netdev=tap0 \
>> >>  -serial stdio
>> >
>> > Please post metrics that compare this against a minimal Q35.
>> >
>> > With qboot it was later found that SeaBIOS can achieve comparable boot
>> > times, so it wasn't worth maintaining qboot.
>> >
>> > Data is needed to show that microvm is really a significant improvement
>> > over a minimal Q35.
>> 
>> I've just ran some numbers using Stefano Garzarella's qemu-boot-time
>> scripts [1] on a server with 2xIntel Xeon Silver 4114 2.20GHz, using the
>> upstream QEMU (474f3938d79ab36b9231c9ad3b5a9314c2aeacde) built with
>> minimal features [2]. The VM boots a minimal kernel [3] without initrd,
>> using a kata container image as root via virtio-blk (though this isn't
>> really relevant, as we're just taking measurements until the kernel is
>> about to exec init).
>> 
>> To try to make the comparison as fair as possible, I've used a minimal
>> q35 machine with as few devices as possible. Disabling HPET and PIT at
>> the same time caused the kernel to get stuck on boot, so I ran two
>> iterations, one without HPET and the other without PIT:
>> 
>> 
>> -----------------
>>  | Q35 with HPET |
>>  -----------------
>> 
>> Command line:
>> 
>> ./x86_64-softmmu/qemu-system-x86_64 -m 512m -enable-kvm -M 
>> q35,smbus=off,nvdimm=off,pit=off,vmport=off,sata=off,usb=off,graphics=off 
>> -kernel /root/src/images/vmlinux-5.2 -append "console=hvc0 reboot=k panic=1 
>> root=/dev/vda quiet" -smp 1 -nodefaults -no-user-config -chardev 
>> pty,id=virtiocon0,server -device virtio-serial -device 
>> virtconsole,chardev=virtiocon0 -drive 
>> id=test,file=/root/src/images/hello-rootfs.ext4,format=raw,if=none -device 
>> virtio-blk,drive=test
>> 
>> Average boot times after 10 consecutive runs:
>> 
>>  qemu_init_end: 77.637936
>>  linux_start_kernel: 117.082526 (+39.44459)
>>  linux_start_user: 364.629972 (+247.547446)
>> 
>> Memory tree:
>> 
>>  address-space: memory
>>   0000000000000000-ffffffffffffffff (prio 0, i/o): system
>>     0000000000000000-000000001fffffff (prio 0, i/o): alias ram-below-4g 
>> @pc.ram 0000000000000000-000000001fffffff
>>     0000000000000000-ffffffffffffffff (prio -1, i/o): pci
>>       00000000000c0000-00000000000dffff (prio 1, rom): pc.rom
>>       00000000000e0000-00000000000fffff (prio 1, i/o): alias isa-bios 
>> @pc.bios 0000000000020000-000000000003ffff
>>       00000000febf4000-00000000febf7fff (prio 1, i/o): virtio-pci
>>         00000000febf4000-00000000febf4fff (prio 0, i/o): virtio-pci-common
>>         00000000febf5000-00000000febf5fff (prio 0, i/o): virtio-pci-isr
>>         00000000febf6000-00000000febf6fff (prio 0, i/o): virtio-pci-device
>>         00000000febf7000-00000000febf7fff (prio 0, i/o): virtio-pci-notify
>>       00000000febf8000-00000000febfbfff (prio 1, i/o): virtio-pci
>>         00000000febf8000-00000000febf8fff (prio 0, i/o): virtio-pci-common
>>         00000000febf9000-00000000febf9fff (prio 0, i/o): virtio-pci-isr
>>         00000000febfa000-00000000febfafff (prio 0, i/o): virtio-pci-device
>>         00000000febfb000-00000000febfbfff (prio 0, i/o): virtio-pci-notify
>>       00000000febfe000-00000000febfefff (prio 1, i/o): virtio-serial-pci-msix
>>         00000000febfe000-00000000febfe01f (prio 0, i/o): msix-table
>>         00000000febfe800-00000000febfe807 (prio 0, i/o): msix-pba
>>       00000000febff000-00000000febfffff (prio 1, i/o): virtio-blk-pci-msix
>>         00000000febff000-00000000febff01f (prio 0, i/o): msix-table
>>         00000000febff800-00000000febff807 (prio 0, i/o): msix-pba
>>       00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
>>     00000000000a0000-00000000000bffff (prio 1, i/o): alias smram-region @pci 
>> 00000000000a0000-00000000000bffff
>>     00000000000c0000-00000000000c2fff (prio 1000, i/o): alias kvmvapic-rom 
>> @pc.ram 00000000000c0000-00000000000c2fff
>>     00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000c0000-00000000000c3fff [disabled]
>>     00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000c0000-00000000000c3fff [disabled]
>>     00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000c0000-00000000000c3fff
>>     00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000c0000-00000000000c3fff [disabled]
>>     00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000c4000-00000000000c7fff [disabled]
>>     00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000c4000-00000000000c7fff [disabled]
>>     00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000c4000-00000000000c7fff
>>     00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000c4000-00000000000c7fff [disabled]
>>     00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000c8000-00000000000cbfff [disabled]
>>     00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000c8000-00000000000cbfff [disabled]
>>     00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000c8000-00000000000cbfff
>>     00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000c8000-00000000000cbfff [disabled]
>>     00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000cc000-00000000000cffff [disabled]
>>     00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000cc000-00000000000cffff [disabled]
>>     00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000cc000-00000000000cffff
>>     00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000cc000-00000000000cffff [disabled]
>>     00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000d0000-00000000000d3fff [disabled]
>>     00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000d0000-00000000000d3fff [disabled]
>>     00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000d0000-00000000000d3fff
>>     00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000d0000-00000000000d3fff [disabled]
>>     00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000d4000-00000000000d7fff [disabled]
>>     00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000d4000-00000000000d7fff [disabled]
>>     00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000d4000-00000000000d7fff
>>     00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000d4000-00000000000d7fff [disabled]
>>     00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000d8000-00000000000dbfff [disabled]
>>     00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000d8000-00000000000dbfff [disabled]
>>     00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000d8000-00000000000dbfff
>>     00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000d8000-00000000000dbfff [disabled]
>>     00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000dc000-00000000000dffff [disabled]
>>     00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000dc000-00000000000dffff [disabled]
>>     00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000dc000-00000000000dffff
>>     00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000dc000-00000000000dffff [disabled]
>>     00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000e0000-00000000000e3fff [disabled]
>>     00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000e0000-00000000000e3fff [disabled]
>>     00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000e0000-00000000000e3fff
>>     00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000e0000-00000000000e3fff [disabled]
>>     00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000e4000-00000000000e7fff [disabled]
>>     00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000e4000-00000000000e7fff [disabled]
>>     00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000e4000-00000000000e7fff
>>     00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000e4000-00000000000e7fff [disabled]
>>     00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000e8000-00000000000ebfff
>>     00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000e8000-00000000000ebfff [disabled]
>>     00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000e8000-00000000000ebfff [disabled]
>>     00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000e8000-00000000000ebfff [disabled]
>>     00000000000ec000-00000000000effff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000ec000-00000000000effff
>>     00000000000ec000-00000000000effff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000ec000-00000000000effff [disabled]
>>     00000000000ec000-00000000000effff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000ec000-00000000000effff [disabled]
>>     00000000000ec000-00000000000effff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000ec000-00000000000effff [disabled]
>>     00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000f0000-00000000000fffff [disabled]
>>     00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000f0000-00000000000fffff [disabled]
>>     00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000f0000-00000000000fffff
>>     00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000f0000-00000000000fffff [disabled]
>>     0000000020000000-0000000020000000 (prio 1, i/o): tseg-blackhole 
>> [disabled]
>>     00000000b0000000-00000000bfffffff (prio 0, i/o): pcie-mmcfg-mmio
>>     00000000fec00000-00000000fec00fff (prio 0, i/o): kvm-ioapic
>>     00000000fed00000-00000000fed003ff (prio 0, i/o): hpet
>>     00000000fed1c000-00000000fed1ffff (prio 1, i/o): lpc-rcrb-mmio
>>     00000000feda0000-00000000fedbffff (prio 1, i/o): alias smram-open-high 
>> @pc.ram 00000000000a0000-00000000000bffff [disabled]
>>     00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi
>> 
>>  address-space: I/O
>>   0000000000000000-000000000000ffff (prio 0, i/o): io
>>     0000000000000000-0000000000000007 (prio 0, i/o): dma-chan
>>     0000000000000008-000000000000000f (prio 0, i/o): dma-cont
>>     0000000000000020-0000000000000021 (prio 0, i/o): kvm-pic
>>     0000000000000060-0000000000000060 (prio 0, i/o): i8042-data
>>     0000000000000064-0000000000000064 (prio 0, i/o): i8042-cmd
>>     0000000000000070-0000000000000071 (prio 0, i/o): rtc
>>       0000000000000070-0000000000000070 (prio 0, i/o): rtc-index
>>     000000000000007e-000000000000007f (prio 0, i/o): kvmvapic
>>     0000000000000080-0000000000000080 (prio 0, i/o): ioport80
>>     0000000000000081-0000000000000083 (prio 0, i/o): dma-page
>>     0000000000000087-0000000000000087 (prio 0, i/o): dma-page
>>     0000000000000089-000000000000008b (prio 0, i/o): dma-page
>>     000000000000008f-000000000000008f (prio 0, i/o): dma-page
>>     0000000000000092-0000000000000092 (prio 0, i/o): port92
>>     00000000000000a0-00000000000000a1 (prio 0, i/o): kvm-pic
>>     00000000000000b2-00000000000000b3 (prio 0, i/o): apm-io
>>     00000000000000c0-00000000000000cf (prio 0, i/o): dma-chan
>>     00000000000000d0-00000000000000df (prio 0, i/o): dma-cont
>>     00000000000000f0-00000000000000f0 (prio 0, i/o): ioportF0
>>     00000000000004d0-00000000000004d0 (prio 0, i/o): kvm-elcr
>>     00000000000004d1-00000000000004d1 (prio 0, i/o): kvm-elcr
>>     0000000000000510-0000000000000511 (prio 0, i/o): fwcfg
>>     0000000000000514-000000000000051b (prio 0, i/o): fwcfg.dma
>>     0000000000000600-000000000000067f (prio 0, i/o): ich9-pm
>>       0000000000000600-0000000000000603 (prio 0, i/o): acpi-evt
>>       0000000000000604-0000000000000605 (prio 0, i/o): acpi-cnt
>>       0000000000000608-000000000000060b (prio 0, i/o): acpi-tmr
>>       0000000000000620-000000000000062f (prio 0, i/o): acpi-gpe0
>>       0000000000000630-0000000000000637 (prio 0, i/o): acpi-smi
>>       0000000000000660-000000000000067f (prio 0, i/o): sm-tco
>>     0000000000000cd8-0000000000000ce3 (prio 0, i/o): acpi-mem-hotplug
>>     0000000000000cf8-0000000000000cfb (prio 0, i/o): pci-conf-idx
>>     0000000000000cf9-0000000000000cf9 (prio 1, i/o): lpc-reset-control
>>     0000000000000cfc-0000000000000cff (prio 0, i/o): pci-conf-data
>>     000000000000c000-000000000000c07f (prio 1, i/o): virtio-pci
>>     000000000000c080-000000000000c0bf (prio 1, i/o): virtio-pci
>> 
>> 
>>  ----------------
>>  | Q35 with PIT |
>>  ----------------
>> 
>> Command line:
>> 
>> ./x86_64-softmmu/qemu-system-x86_64 -m 512m -enable-kvm -M 
>> q35,smbus=off,nvdimm=off,pit=on,vmport=off,sata=off,usb=off,graphics=off 
>> -no-hpet -kernel /root/src/images/vmlinux-5.2 -append "console=hvc0 reboot=k 
>> panic=1 root=/dev/vda quiet" -smp 1 -nodefaults -no-user-config -chardev 
>> pty,id=virtiocon0,server -device virtio-serial -device 
>> virtconsole,chardev=virtiocon0 -drive 
>> id=test,file=/root/src/images/hello-rootfs.ext4,format=raw,if=none -device 
>> virtio-blk,drive=test
>> 
>> Average boot times after 10 consecutive runs:
>> 
>>  qemu_init_end: 77.467852
>>  linux_start_kernel: 116.688472 (+39.22062)
>>  linux_start_user: 363.033365 (+246.344893)
>> 
>> Memory tree:
>> 
>> address-space: memory
>>   0000000000000000-ffffffffffffffff (prio 0, i/o): system
>>     0000000000000000-000000001fffffff (prio 0, i/o): alias ram-below-4g 
>> @pc.ram 0000000000000000-000000001fffffff
>>     0000000000000000-ffffffffffffffff (prio -1, i/o): pci
>>       00000000000c0000-00000000000dffff (prio 1, rom): pc.rom
>>       00000000000e0000-00000000000fffff (prio 1, i/o): alias isa-bios 
>> @pc.bios 0000000000020000-000000000003ffff
>>       00000000febf4000-00000000febf7fff (prio 1, i/o): virtio-pci
>>         00000000febf4000-00000000febf4fff (prio 0, i/o): virtio-pci-common
>>         00000000febf5000-00000000febf5fff (prio 0, i/o): virtio-pci-isr
>>         00000000febf6000-00000000febf6fff (prio 0, i/o): virtio-pci-device
>>         00000000febf7000-00000000febf7fff (prio 0, i/o): virtio-pci-notify
>>       00000000febf8000-00000000febfbfff (prio 1, i/o): virtio-pci
>>         00000000febf8000-00000000febf8fff (prio 0, i/o): virtio-pci-common
>>         00000000febf9000-00000000febf9fff (prio 0, i/o): virtio-pci-isr
>>         00000000febfa000-00000000febfafff (prio 0, i/o): virtio-pci-device
>>         00000000febfb000-00000000febfbfff (prio 0, i/o): virtio-pci-notify
>>       00000000febfe000-00000000febfefff (prio 1, i/o): virtio-serial-pci-msix
>>         00000000febfe000-00000000febfe01f (prio 0, i/o): msix-table
>>         00000000febfe800-00000000febfe807 (prio 0, i/o): msix-pba
>>       00000000febff000-00000000febfffff (prio 1, i/o): virtio-blk-pci-msix
>>         00000000febff000-00000000febff01f (prio 0, i/o): msix-table
>>         00000000febff800-00000000febff807 (prio 0, i/o): msix-pba
>>       00000000fffc0000-00000000ffffffff (prio 0, rom): pc.bios
>>     00000000000a0000-00000000000bffff (prio 1, i/o): alias smram-region @pci 
>> 00000000000a0000-00000000000bffff
>>     00000000000c0000-00000000000c2fff (prio 1000, i/o): alias kvmvapic-rom 
>> @pc.ram 00000000000c0000-00000000000c2fff
>>     00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000c0000-00000000000c3fff [disabled]
>>     00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000c0000-00000000000c3fff [disabled]
>>     00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000c0000-00000000000c3fff
>>     00000000000c0000-00000000000c3fff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000c0000-00000000000c3fff [disabled]
>>     00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000c4000-00000000000c7fff [disabled]
>>     00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000c4000-00000000000c7fff [disabled]
>>     00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000c4000-00000000000c7fff
>>     00000000000c4000-00000000000c7fff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000c4000-00000000000c7fff [disabled]
>>     00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000c8000-00000000000cbfff [disabled]
>>     00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000c8000-00000000000cbfff [disabled]
>>     00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000c8000-00000000000cbfff
>>     00000000000c8000-00000000000cbfff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000c8000-00000000000cbfff [disabled]
>>     00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000cc000-00000000000cffff [disabled]
>>     00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000cc000-00000000000cffff [disabled]
>>     00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000cc000-00000000000cffff
>>     00000000000cc000-00000000000cffff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000cc000-00000000000cffff [disabled]
>>     00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000d0000-00000000000d3fff [disabled]
>>     00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000d0000-00000000000d3fff [disabled]
>>     00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000d0000-00000000000d3fff
>>     00000000000d0000-00000000000d3fff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000d0000-00000000000d3fff [disabled]
>>     00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000d4000-00000000000d7fff [disabled]
>>     00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000d4000-00000000000d7fff [disabled]
>>     00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000d4000-00000000000d7fff
>>     00000000000d4000-00000000000d7fff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000d4000-00000000000d7fff [disabled]
>>     00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000d8000-00000000000dbfff [disabled]
>>     00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000d8000-00000000000dbfff [disabled]
>>     00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000d8000-00000000000dbfff
>>     00000000000d8000-00000000000dbfff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000d8000-00000000000dbfff [disabled]
>>     00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000dc000-00000000000dffff [disabled]
>>     00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000dc000-00000000000dffff [disabled]
>>     00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000dc000-00000000000dffff
>>     00000000000dc000-00000000000dffff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000dc000-00000000000dffff [disabled]
>>     00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000e0000-00000000000e3fff [disabled]
>>     00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000e0000-00000000000e3fff [disabled]
>>     00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000e0000-00000000000e3fff
>>     00000000000e0000-00000000000e3fff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000e0000-00000000000e3fff [disabled]
>>     00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000e4000-00000000000e7fff [disabled]
>>     00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000e4000-00000000000e7fff [disabled]
>>     00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000e4000-00000000000e7fff
>>     00000000000e4000-00000000000e7fff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000e4000-00000000000e7fff [disabled]
>>     00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000e8000-00000000000ebfff
>>     00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000e8000-00000000000ebfff [disabled]
>>     00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000e8000-00000000000ebfff [disabled]
>>     00000000000e8000-00000000000ebfff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000e8000-00000000000ebfff [disabled]
>>     00000000000ec000-00000000000effff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000ec000-00000000000effff
>>     00000000000ec000-00000000000effff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000ec000-00000000000effff [disabled]
>>     00000000000ec000-00000000000effff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000ec000-00000000000effff [disabled]
>>     00000000000ec000-00000000000effff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000ec000-00000000000effff [disabled]
>>     00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-ram @pc.ram 
>> 00000000000f0000-00000000000fffff [disabled]
>>     00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-pci @pc.ram 
>> 00000000000f0000-00000000000fffff [disabled]
>>     00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-rom @pc.ram 
>> 00000000000f0000-00000000000fffff
>>     00000000000f0000-00000000000fffff (prio 1, i/o): alias pam-pci @pci 
>> 00000000000f0000-00000000000fffff [disabled]
>>     0000000020000000-0000000020000000 (prio 1, i/o): tseg-blackhole 
>> [disabled]
>>     00000000b0000000-00000000bfffffff (prio 0, i/o): pcie-mmcfg-mmio
>>     00000000fec00000-00000000fec00fff (prio 0, i/o): kvm-ioapic
>>     00000000fed1c000-00000000fed1ffff (prio 1, i/o): lpc-rcrb-mmio
>>     00000000feda0000-00000000fedbffff (prio 1, i/o): alias smram-open-high 
>> @pc.ram 00000000000a0000-00000000000bffff [disabled]
>>     00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi
>> 
>> address-space: I/O
>>   0000000000000000-000000000000ffff (prio 0, i/o): io
>>     0000000000000000-0000000000000007 (prio 0, i/o): dma-chan
>>     0000000000000008-000000000000000f (prio 0, i/o): dma-cont
>>     0000000000000020-0000000000000021 (prio 0, i/o): kvm-pic
>>     0000000000000040-0000000000000043 (prio 0, i/o): kvm-pit
>>     0000000000000060-0000000000000060 (prio 0, i/o): i8042-data
>>     0000000000000061-0000000000000061 (prio 0, i/o): pcspk
>>     0000000000000064-0000000000000064 (prio 0, i/o): i8042-cmd
>>     0000000000000070-0000000000000071 (prio 0, i/o): rtc
>>       0000000000000070-0000000000000070 (prio 0, i/o): rtc-index
>>     000000000000007e-000000000000007f (prio 0, i/o): kvmvapic
>>     0000000000000080-0000000000000080 (prio 0, i/o): ioport80
>>     0000000000000081-0000000000000083 (prio 0, i/o): dma-page
>>     0000000000000087-0000000000000087 (prio 0, i/o): dma-page
>>     0000000000000089-000000000000008b (prio 0, i/o): dma-page
>>     000000000000008f-000000000000008f (prio 0, i/o): dma-page
>>     0000000000000092-0000000000000092 (prio 0, i/o): port92
>>     00000000000000a0-00000000000000a1 (prio 0, i/o): kvm-pic
>>     00000000000000b2-00000000000000b3 (prio 0, i/o): apm-io
>>     00000000000000c0-00000000000000cf (prio 0, i/o): dma-chan
>>     00000000000000d0-00000000000000df (prio 0, i/o): dma-cont
>>     00000000000000f0-00000000000000f0 (prio 0, i/o): ioportF0
>>     00000000000004d0-00000000000004d0 (prio 0, i/o): kvm-elcr
>>     00000000000004d1-00000000000004d1 (prio 0, i/o): kvm-elcr
>>     0000000000000510-0000000000000511 (prio 0, i/o): fwcfg
>>     0000000000000514-000000000000051b (prio 0, i/o): fwcfg.dma
>>     0000000000000600-000000000000067f (prio 0, i/o): ich9-pm
>>       0000000000000600-0000000000000603 (prio 0, i/o): acpi-evt
>>       0000000000000604-0000000000000605 (prio 0, i/o): acpi-cnt
>>       0000000000000608-000000000000060b (prio 0, i/o): acpi-tmr
>>       0000000000000620-000000000000062f (prio 0, i/o): acpi-gpe0
>>       0000000000000630-0000000000000637 (prio 0, i/o): acpi-smi
>>       0000000000000660-000000000000067f (prio 0, i/o): sm-tco
>>     0000000000000cd8-0000000000000ce3 (prio 0, i/o): acpi-mem-hotplug
>>     0000000000000cf8-0000000000000cfb (prio 0, i/o): pci-conf-idx
>>     0000000000000cf9-0000000000000cf9 (prio 1, i/o): lpc-reset-control
>>     0000000000000cfc-0000000000000cff (prio 0, i/o): pci-conf-data
>>     000000000000c000-000000000000c07f (prio 1, i/o): virtio-pci
>>     000000000000c080-000000000000c0bf (prio 1, i/o): virtio-pci
>> 
>> 
>>  -----------
>>  | microvm |
>>  -----------
>> 
>> Command line:
>> 
>> ./x86_64-softmmu/qemu-system-x86_64 -m 512m -enable-kvm -M microvm -kernel 
>> /root/src/images/vmlinux-5.2 -append "console=hvc0 reboot=k panic=1 
>> root=/dev/vda quiet" -smp 1 -nodefaults -no-user-config -chardev 
>> pty,id=virtiocon0,server -device virtio-serial-device -device 
>> virtconsole,chardev=virtiocon0 -drive 
>> id=test,file=/root/src/images/hello-rootfs.ext4,format=raw,if=none -device 
>> virtio-blk-device,drive=test
>> 
>> Average boot times after 10 consecutive runs:
>> 
>>  qemu_init_end: 64.043264
>>  linux_start_kernel: 65.481782 (+1.438518)
>>  linux_start_user: 114.938353 (+49.456571)
>> 
>> Memory tree:
>> 
>>  address-space: memory
>>   0000000000000000-ffffffffffffffff (prio 0, i/o): system
>>     0000000000000000-000000001fffffff (prio 0, i/o): alias ram-below-4g 
>> @microvm.ram 0000000000000000-000000001fffffff
>>     00000000d0000000-00000000d00001ff (prio 0, i/o): virtio-mmio
>>     00000000d0000200-00000000d00003ff (prio 0, i/o): virtio-mmio
>>     00000000d0000400-00000000d00005ff (prio 0, i/o): virtio-mmio
>>     00000000d0000600-00000000d00007ff (prio 0, i/o): virtio-mmio
>>     00000000fec00000-00000000fec00fff (prio 0, i/o): kvm-ioapic
>>     00000000fee00000-00000000feefffff (prio 4096, i/o): kvm-apic-msi
>> 
>>  address-space: I/O
>>   0000000000000000-000000000000ffff (prio 0, i/o): io
>>     000000000000007e-000000000000007f (prio 0, i/o): kvmvapic
>> 
>> 
>>  --------------
>>  | Conclusion |
>>  --------------
>> 
>> The average boot time of microvm is a third of Q35's (115ms vs. 363ms),
>> and is smaller on all sections (QEMU initialization, firmware overhead
>> and kernel start-to-user).
>> 
>> Microvm's memory tree is also visibly simpler, significantly reducing
>> the exposed surface to the guest.
>> 
>> While we can certainly work on making Q35 smaller, I definitely think
>> it's better (and way safer!) having a specialized machine type for a
>> specific use case, than a minimal Q35 whose behavior significantly
>> diverges from a conventional Q35.
>
> Interesting, so not a 10x difference!  This might be amenable to
> optimization.
>
> My concern with microvm is that it's so limited that few users will be
> able to benefit from the reduced attack surface and faster startup time.
> I think it's worth investigating slimming down Q35 further first.
>
> In terms of startup time the first step would be profiling Q35 kernel
> startup to find out what's taking so long (firmware initialization, PCI
> probing, etc)?

Some findings:

 1. Exposing the TSC_DEADLINE CPU flag (i.e. using "-cpu host") saves a
    whooping 120ms by avoiding the APIC timer calibration at
    arch/x86/kernel/apic/apic.c:calibrate_APIC_clock

Average boot time with "-cpu host"
 qemu_init_end: 76.408950
 linux_start_kernel: 116.166142 (+39.757192)
 linux_start_user: 242.954347 (+126.788205)

Average boot time with default "cpu"
 qemu_init_end: 77.467852
 linux_start_kernel: 116.688472 (+39.22062)
 linux_start_user: 363.033365 (+246.344893)

 2. The other 130ms are a direct result of PCI and ACPI presence (tested
    with a kernel without support for those elements). I'll publish some
    detailed numbers next week.

Sergio.

signature.asc
Description: PGP signature