Re: [qemu-s390x] [Qemu-devel] [PATCH v3 09/20] Boot Linux Console Test: update the x86_64 kernel

[Gerd Hoffmann]

  Hi,

> > I'd like automate some of the graphics testing.  For starters boot
> > guest, run test utility to display something, take a screendump, check
> > whenever it actually shows what I expect it to show.  Maybe later run
> > test suites like https://cgit.freedesktop.org/drm/igt-gpu-tools/.
> > 
> > Ideally it would allow to not only test qemu, but to also test the
> > kernel drivers (i.e. allows to boot and test with some custom
> > development kernel).
> > 
> > For that I need a bit more than just a kernel.  Suggestions how to do
> > that best with autotest?
> > 
> > Fetch distro image, install/compile what I need, cache the image
> > somehow, then use it to run the tests?  Would the vmimage library allow
> > to do that?  From a brief look it seems to only support fetching the
> > images, not customizing them ...
> 
> Do you feel like most of the customization would be done constantly?

For qemu testing no.
For linux kernel testing yes.

> I'm trying to grasp if it would make sense to have an "almost ready"
> image, and adding a final level of customization (adding freshly built
> drivers?) on top of that.
> 
> If that sounds like a possibility, than it should be possible to host
> your images somewhere[1] and let the vmimage library get and cache them
> (you can expand the list of providers).

Hmm.

Put kernel testing aside for now.  I don't think it is that an good idea
to host a bunch of custom images just because some test tools are needed
for some test.  The cloud images are typically pretty stripped down, so
when using them for testing the need to install additional is probably
not rare.  Be it my graphics test tools, or usbutils for usb testing, or
something else network/block testing.

I think it would make sense to support creating and caching such custom
images, i.e. download standard fedora image, install test tools (fetch
rpms or compile stuff), then put that image into a local cache and use
it for test runs.

> Then, when it comes to the customization part, do you expect your
> running environments to have a "reasonable" level of high level tools
> such as SSH and Python?

Wouldn't hurt, but isn't required.  At least initially.

When maybe running a whole gfx test suite inside the guest some day
it'll be a different story.  That'll probably out of scope for
acceptance tests, but using avocado might still be useful for that.

> PS: I have strong feelings about what a test should do, and that should not
> include any kind of setup (including downloading files it'll use).  This
> is somehow loosely expressed in work I intend to do at a given point[2].

Yep.  Setup and test run should be separate, so you setup once and don't
fetch stuff from the Internet for each test run.

cheers,
  Gerd