Re: [qemu-s390x] [Qemu-devel] [PATCH v3 09/20] Boot Linux Console Test: update the x86_64 kernel

[Cleber Rosa]

On Tue, Mar 12, 2019 at 10:35:15AM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > possible that these "kernel_urls" can be turned into parameters
> > that will eventually be expand based on the user's selected distro
> > and version (similar to the Avocado vmimage library[1]).
> 
> Speaking of distros ...
> 
> I'd like automate some of the graphics testing.  For starters boot
> guest, run test utility to display something, take a screendump, check
> whenever it actually shows what I expect it to show.  Maybe later run
> test suites like https://cgit.freedesktop.org/drm/igt-gpu-tools/.
> 
> Ideally it would allow to not only test qemu, but to also test the
> kernel drivers (i.e. allows to boot and test with some custom
> development kernel).
> 
> For that I need a bit more than just a kernel.  Suggestions how to do
> that best with autotest?
> 
> Fetch distro image, install/compile what I need, cache the image
> somehow, then use it to run the tests?  Would the vmimage library allow
> to do that?  From a brief look it seems to only support fetching the
> images, not customizing them ...
>

Do you feel like most of the customization would be done constantly?
I'm trying to grasp if it would make sense to have an "almost ready"
image, and adding a final level of customization (adding freshly built
drivers?) on top of that.

If that sounds like a possibility, than it should be possible to host
your images somewhere[1] and let the vmimage library get and cache them
(you can expand the list of providers).

Then, when it comes to the customization part, do you expect your
running environments to have a "reasonable" level of high level tools
such as SSH and Python?  The reason I ask is, as much as it sounds
like overkill, tools like ansible (that depend on both of these in the
managed system) can take a considerable amount of complexity out of
test.  In my experience it's only when tools like ansible are used as
programming languages that everything starts to crumble and fall apart.

In the context of "acceptance" tests, an "ansible" dependency can be
expressed as a requirement on "tests/requirements.txt" so the
bootstrap should not be a problem.  Then, I'd really like to add some
level of integration between the:

 * the avocado.utils.vmimage library
 * the avocado.utils.cloudinit library
 * the avocado.utils.ssh library
 * the QEMUMachine instances we spawn
 * running an ansible playbook on the previous

Brainstorming about it:

   def setUp(self):
       with self.get_vm() as vm:
          vm.add_drive(utils.vmimage.get("mydistro"))
          vm.add_drive(utils.cloudinit.iso())
          vm.start()
          utils.ansible.run(vm, "graphic_drivers.yml")
          # failure to get here can CANCEL the test

   def test(self):
       with utils.ssh.SSHSession(self.get_vm("default_vm")) as ssh:
           ...

> Create an appliance using supermin?
> 
> Create an special initrd using dracut?
>

If your systems are not expected to contain higher level tools, than,
I'm afraid those are good options indeed.

> Something else?
> 
> thanks,
>   Gerd
>

PS: I have strong feelings about what a test should do, and that should not
include any kind of setup (including downloading files it'll use).  This
is somehow loosely expressed in work I intend to do at a given point[2].

[1] - There's lot of unused disk space and bandwidth available in the
      "avocado-project.org" server.

[2] - 
https://trello.com/c/WPd4FrIy/1479-add-support-to-specify-assets-in-test-docstring