Re: [Qemu-devel] [Xen-devel] [RFC PATCH 0/2] support xen HVM direct kernel boot

[Ian Campbell]

On Tue, 2014-06-03 at 03:16 -0600, Chun Yan Liu wrote:
> 
> >>> On 6/2/2014 at 11:14 PM, in message
> <address@hidden>, Stefano
> Stabellini <address@hidden> wrote: 
> > On Fri, 30 May 2014, Konrad Rzeszutek Wilk wrote: 
> > > On Thu, May 29, 2014 at 11:23:22AM +0800, Chunyan Liu wrote: 
> > > > Following previous discussion: 
> > > > https://lists.gnu.org/archive/html/qemu-devel/2014-04/msg03843.html 
> > > >  
> > > > I spent lot of time to understand xen xc_hvm_build, hvmloader code, 
> > > > and qemu pc_memory_init, xen_ram_init code, and wrote experimental 
> > > > code to test. Now it's basically working based on seabios. 
> > >  
> > > Excellent! 
> > > >  
> > > > Talking about qemu linuxboot.bin/multiboot.bin first, they are 
> > > > implemented to intercept boot process, originally by override int19, 
> > > > later changed to provide BEV entry which will be called by seabios 
> > > > if option rom is selected for booting. linuxboot.bin/multiboot.bin 
> > > > will load kernel/initrd and jump to execute kernel directly. 
> > > >  
> > > > In qemu load_linux() code, it first read out kernel/initrd file and 
> > > > store related ADDRs to fw_cfg (from where linuxboot.bin/multiboot.bin 
> > > > will get the ADDRs and load kernel/initrd data), then add 
> > > > linuxboot.bin/multiboot.bin to option rom with bootindex=0. 
> > > >  
> > > > In seabios side, it could load and execute ROMs by itself, and 
> > > > according 
> > > > to boot order, it will boot from the BEV entry on the option rom. That 
> > > > will then be taken over by linuxboot.bin/multiboot.bin. 
> > > >  
> > > > So, in theory, qemu load_linux() code almost does the work. I just 
> > > > tried 
> > > > to reuse it for xen. 
> > > >  
> > > > xen side patch: pass kernel/initrd/append parameters to qemu-dm 
> > > > qemu side patch: reuse load_linux() for xen hvm direct kernel boot. 
> > > >     Different from pc_memory_init which does lots of ram alloc work 
> > > >     and rom/bios loading work, for xen, we only need to init a basic 
> > > >     fw_cfg device used by load_linux() to store ADDRs and 
> > > >     linuxboot.bin/multiboot.bin to retrive ADDRs, then load_linux(), 
> > > >     after that, do real add option rom work to add 
> > > >     linuxboot.bin/multiboot.bin to system option rom. Other things 
> > > >     would be done by seabios smoothly. 
> > > >  
> > > > In my testing, it's working based on seabios. 
> > > > Rombios case is not working. But now that xen prefers qemu upstream, 
> > > > seabios will be default, do we need to put effort to make rombios work? 
> > >  
> > > No. But the code (libxl) should detect whether you are using the 
> > > traditional QEMU - which would be using ROM BIOS. If it detects that 
> > > it shouldn't allow you to pass the arguments. 
> > >  
> > > You probably will also need to an docs/ change to document these 
> > > new parameters. 
> > > > And stubdom case, since I didn't make my guest started even without 
> > > > direct 
> > > > kernel boot, I don't know if it works. And I still could not well 
> > > > understand about the differences to stubdom and non stubdom this work 
> > > > may cover. Any problems please point out. 
> > >  
> > > Hmm, you would have to somehow 'transfer' the images from one domain 
> > > (the control domain) to the stubdomain which would do the execution. 
> > >  
> > > At least the multiboot.bin image. 
> >  
> > Yeah, unfortunately the approach of implemeting direct kernel boot for 
> > HVM guests via QEMU wouldn't work for stubdomains. 
> >  
> > If we want this to work with stubdoms, it would be best to reimplement 
> > kernel and initrd loading in libxl/libxc. 
> 
> Thanks. Rouphly thinking, the implementation might be totally different.
> For seabios, could load kernel/initrd (and linuxboot.bin) in libxc, and in
> qemu find a way to get data and continue with existing code logic (qemu
> side changes may be not very clean).

I suppose the native qemu code path is roughly
      * qemu:
              * parse command line
              * load file from disk into appropriate place in guest RAM
              * insert the BIOS extension
              * go

For Xen it would perhaps be something like:
      * libxc:
              * load file from disk into appropriate place in guest RAM
              * communicate this to qemu somehow (xenstore, command
                line, in memory
              * data structure at known location, carrier pigeons)
      * qemu:
              * discover images in RAM according to scheme decided above
              * (perhaps) move those images to the appropriate place in
                guest ram
              * insert the BIOS extension
              * go

So at least the second half of the qemu side could be common/normal.
Perhaps the first half on the qemu side could be abstracted away as some
sort of magic "file like" thing backed by bytes already present in guest
RAM? (it should be pretty clear by now that I don't really know much
about the qemu internals side of this...!)

>  And stubdom-dm uses
> qemu-xen-traditional, that means rombios by default, that method doesn't
> work.

Sorry, I should have said: support for direct kernel from a stub qemu is
dependent on someone implementing stubdm with qemu upstream first. There
is no need at all to implement this for the existing qemu-trad stub dm,
in fact I'm actively against doing so (no new features for qemu-trad).

Ian Jackson is working on making qemu upstream work as a stubdom.

I think I'd be happy if we simply had a plausible plan for how we would
support direct kernel boot with upstream qemu in a stubdm, rather than
blocking these changes on that or expecting you to do that porting work.
Ian -- does that sound OK?

>  Maybe need to change hvmloader? I'll have a look.
> 
> > I have to admit that given the extreme simplicity of this patch series, 
> > I wouldn't be against merging it, even if it cannot be made to work with 
> > stubdoms.  However like in the rombios case, libxl would need to 
> > understand that this feature doesn't work with stubdoms and print an 
> > appropriate message. 
> 
> I'll update printing messages. But now, as you prefer, should I update
> existing patch series to let it be in good shape for merge, or should I
> abandon it and reimplement to try to make it work with stubdom?

You can update this series, I think.

Ian.