Re: [PATCH 1/4] Acceptance tests: use an available kernel image package for arm

[Philippe Mathieu-Daudé]

On 9/7/20 11:39 AM, Daniel P. Berrangé wrote:
> On Mon, Sep 07, 2020 at 10:06:13AM +0200, Philippe Mathieu-Daudé wrote:
>> [Cc'ing Daniel who usually have good ideas for that
>> kind if project-wide problem]
>>
>> On 9/7/20 6:19 AM, Cleber Rosa wrote:
>>> Which means a newer kernel version.  Expected output was changed
>>> to match the new kernel too.
>>
>> Nack.
>>
>> Acceptance tests are not to test the latest Linux kernel,
>> they aim to assert a specific kernel tested by some developer
>> still works while QEMU evolves.
>> QEMU doesn't have to adapt to the latest kernel;
>> QEMU should keep boot an old kernel.
>>
>> Testing new kernels is good, you are adding coverage. But
>> this break the acceptance testing contract "keep testing
>> the same thing over time".
>>
>> The problem you are trying to fix is the "where to keep
>> assets from public locations where they are being removed?"
>> one. Two years ago [*] you suggested to use some storage on
>> the avocado-project.org:
>>
>>   For Avocado-VT, there are the JeOS images[1], which we
>>   keep on a test "assets" directory.  We have a lot of
>>   storage/bandwidth availability, so it can be used for
>>   other assets proven to be necessary for tests.
>>
>>   As long as distribution rights and licensing are not
>>   issues, we can definitely use the same server for kernels,
>>   u-boot images and what not.
>>
>>   [1] - https://avocado-project.org/data/assets/
> 
> If I look at stuff under that directory I see a bunch of "Jeos" qcow2
> images, and zero information about the corresponding source for the
> images, nor any information about the licenses of software included.
> IOW what is stored their right now does not appear to comply with the
> GPL licensing requirements for providing full and corresponding source.
> 
>> It is time to have QEMU assets managed the same way.
> 
> I'd rather we didn't do anything relying on binary blobs with no
> info about how they were built. Pointing to the 3rd party download
> URLs was the easy way to ensure we don't have to worry about licensing
> problems.

I tried to be very strict including the recipe about how to rebuild
and description of the source (for licensing) in each commits (Alex
Bennée once said Debian/Fedora based was OK):

commit 784b8796380532351737d753a6a092ed4e1ffebf
Author: Philippe Mathieu-Daudé <f4bug@amsat.org>
Date:   Wed Mar 11 23:18:52 2020 +0100

    tests/boot_linux_console: Add a SLOW test booting Ubuntu on OrangePi PC

    This test boots Ubuntu Bionic on a OrangePi PC board.

    As it requires 1GB of storage, and is slow, this test is disabled
    on automatic CI testing.

    It is useful for workstation testing. Currently Avocado timeouts too
    quickly, so we can't run userland commands.

    The kernel image and DeviceTree blob are built by the Armbian
    project (based on Debian):
    https://www.armbian.com/orange-pi-pc/

    The Ubuntu image is downloaded from:
    https://dl.armbian.com/orangepipc/Bionic_current

commit de2749bcb2459d0722022fb70c70310a04413fc1
Author: Philippe Mathieu-Daudé <f4bug@amsat.org>
Date:   Wed Mar 11 23:18:51 2020 +0100

    tests/boot_linux_console: Add a SD card test for the OrangePi PC board

    The kernel image and DeviceTree blob are built by the Armbian
    project (based on Debian):
    https://www.armbian.com/orange-pi-pc/

    The SD image is from the kernelci.org project:
    https://kernelci.org/faq/#the-code

commit c40b1ded0af6ee5b26e4b2eb5e0ce68c308e8de3
Author: Philippe Mathieu-Daudé <f4bug@amsat.org>
Date:   Wed Mar 11 23:18:50 2020 +0100

    tests/boot_linux_console: Add initrd test for the Orange Pi PC board

    This test boots a Linux kernel on a OrangePi PC board and verify
    the serial output is working.

    The kernel image and DeviceTree blob are built by the Armbian
    project (based on Debian):
    https://www.armbian.com/orange-pi-pc/

    The cpio image used comes from the linux-build-test project:
    https://github.com/groeck/linux-build-test

commit e33ee3097f705bd7d3ae14a177e12a4b16055970
Author: Philippe Mathieu-Daudé <f4bug@amsat.org>
Date:   Fri Jan 17 14:09:30 2020 +0000

    tests/boot_linux_console: Add a SD card test for the CubieBoard

    The kernel image and DeviceTree blob are built by the Armbian
    project (based on Debian):
    https://docs.armbian.com/Developer-Guide_Build-Preparation/

    The cpio image used comes from the linux-build-test project:
    https://github.com/groeck/linux-build-test

commit efdb45bfd72745038909dfd1e970a827cb8d5d7e
Author: Philippe Mathieu-Daudé <f4bug@amsat.org>
Date:   Mon Oct 28 19:04:04 2019 -0400

    tests/boot_linux_console: Run BusyBox on 5KEc 64-bit cpu

    This tests boots a Linux kernel on a Malta machine up to a
    busybox shell on the serial console. Few commands are executed
    before halting the machine (via reboot).

    We use the Fedora 24 kernel extracted from the image at:
    https://fedoraproject.org/wiki/Architectures/MIPS
    and the initrd cpio image from the kerneltests project:
    https://kerneltests.org/

commit 92d9361255a0862379866f156686180902bb63d2
Author: Philippe Mathieu-Daudé <f4bug@amsat.org>
Date:   Mon Oct 28 19:04:04 2019 -0400

    tests/boot_linux_console: Add a test for the Raspberry Pi 2

    Similar to the x86_64/pc test, it boots a Linux kernel on a raspi2
    board and verify the serial is working.

    The kernel image and DeviceTree blob are built by the Raspbian
    project (based on Debian):
    https://www.raspbian.org/RaspbianImages
    as recommended by the Raspberry Pi project:
    https://www.raspberrypi.org/downloads/raspbian/

> 
> IIUC, the majority of our acceptance tests needs should be satisfied
> with a combination of a Linux kernel and busybox tools. We already
> use cross-compilers to build TCG test cases.
> 
> I'd like to see us provide a minimal "config" file for each Linux
> kernel combo we need to test. We should be able to define a fairly
> simple script that do the Linux build, busybox build and then puts
> the combination of them into a disk image.  Something 100% automated
> and reviewable.

For Debian/Fedora this config is available in their package (or source
package).

> 
> Even a minimal linux build takes some time, so we would need to cache
> the binary result in some suitable manner. At least this way we have
> a clear record of what we're building & how and can thus satisfy the
> license rules.

Yes, but if the Debian/Fedora build farms already build/published
their packages, it is easier to use it directly.

QEMU developers are very reluctant to add tests. I suppose we prefer
to develop or fix bugs rather than write tests. If we ask full build
recipe for each tests, I doubt we'll have much testing.

> 
>>
>> [*] https://www.mail-archive.com/avocado-devel@redhat.com/msg00860.html
>>
>>>
>>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>>> ---
>>>  tests/acceptance/boot_linux_console.py | 44 +++++++++++++-------------
>>>  tests/acceptance/replay_kernel.py      |  8 ++---
>>>  2 files changed, 26 insertions(+), 26 deletions(-)
>>>
>>> diff --git a/tests/acceptance/boot_linux_console.py 
>>> b/tests/acceptance/boot_linux_console.py
>>> index aaa781a581..751b47b8fd 100644
>>> --- a/tests/acceptance/boot_linux_console.py
>>> +++ b/tests/acceptance/boot_linux_console.py
>>> @@ -490,12 +490,12 @@ class BootLinuxConsole(LinuxKernelTest):
>>>          :avocado: tags=arch:arm
>>>          :avocado: tags=machine:cubieboard
>>>          """
>>> -        deb_url = ('https://apt.armbian.com/pool/main/l/'
>>> -                   
>>> 'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
>>> -        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
>>> +        deb_url = ('https://apt.armbian.com/pool/main/l/linux-5.8.0-sunxi/'
>>> +                   'linux-image-dev-sunxi_20.08_armhf.deb')
>>> +        deb_hash = 'ae553a9f7d43b18abfa8f3e64bf2d31878b9be89'
>>>          deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
>>>          kernel_path = self.extract_from_deb(deb_path,
>>> -                                            '/boot/vmlinuz-4.20.7-sunxi')
>>> +                                            '/boot/vmlinuz-5.8.0-sunxi')
>>>          dtb_path = 
>>> '/usr/lib/linux-image-dev-sunxi/sun4i-a10-cubieboard.dtb'
>>>          dtb_path = self.extract_from_deb(deb_path, dtb_path)
>>>          initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
>>> @@ -522,7 +522,7 @@ class BootLinuxConsole(LinuxKernelTest):
>>>          exec_command_and_wait_for_pattern(self, 'cat /proc/cpuinfo',
>>>                                                  'Allwinner sun4i/sun5i')
>>>          exec_command_and_wait_for_pattern(self, 'cat /proc/iomem',
>>> -                                                'system-control@1c00000')
>>> +                                                '1c00000.system-control')
>>>          # cubieboard's reboot is not functioning; omit reboot test.
>>>  
>>>      def test_arm_cubieboard_sata(self):
>>> @@ -530,12 +530,12 @@ class BootLinuxConsole(LinuxKernelTest):
>>>          :avocado: tags=arch:arm
>>>          :avocado: tags=machine:cubieboard
>>>          """
>>> -        deb_url = ('https://apt.armbian.com/pool/main/l/'
>>> -                   
>>> 'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
>>> -        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
>>> +        deb_url = ('https://apt.armbian.com/pool/main/l/linux-5.8.0-sunxi/'
>>> +                   'linux-image-dev-sunxi_20.08_armhf.deb')
>>> +        deb_hash = 'ae553a9f7d43b18abfa8f3e64bf2d31878b9be89'
>>>          deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
>>>          kernel_path = self.extract_from_deb(deb_path,
>>> -                                            '/boot/vmlinuz-4.20.7-sunxi')
>>> +                                            '/boot/vmlinuz-5.8.0-sunxi')
>>>          dtb_path = 
>>> '/usr/lib/linux-image-dev-sunxi/sun4i-a10-cubieboard.dtb'
>>>          dtb_path = self.extract_from_deb(deb_path, dtb_path)
>>>          rootfs_url = ('https://github.com/groeck/linux-build-test/raw/'
>>> @@ -573,12 +573,12 @@ class BootLinuxConsole(LinuxKernelTest):
>>>          :avocado: tags=arch:arm
>>>          :avocado: tags=machine:orangepi-pc
>>>          """
>>> -        deb_url = ('https://apt.armbian.com/pool/main/l/'
>>> -                   
>>> 'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
>>> -        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
>>> +        deb_url = ('https://apt.armbian.com/pool/main/l/linux-5.8.0-sunxi/'
>>> +                   'linux-image-dev-sunxi_20.08_armhf.deb')
>>> +        deb_hash = 'ae553a9f7d43b18abfa8f3e64bf2d31878b9be89'
>>>          deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
>>>          kernel_path = self.extract_from_deb(deb_path,
>>> -                                            '/boot/vmlinuz-4.20.7-sunxi')
>>> +                                            '/boot/vmlinuz-5.8.0-sunxi')
>>>          dtb_path = 
>>> '/usr/lib/linux-image-dev-sunxi/sun8i-h3-orangepi-pc.dtb'
>>>          dtb_path = self.extract_from_deb(deb_path, dtb_path)
>>>  
>>> @@ -598,12 +598,12 @@ class BootLinuxConsole(LinuxKernelTest):
>>>          :avocado: tags=arch:arm
>>>          :avocado: tags=machine:orangepi-pc
>>>          """
>>> -        deb_url = ('https://apt.armbian.com/pool/main/l/'
>>> -                   
>>> 'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
>>> -        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
>>> +        deb_url = ('https://apt.armbian.com/pool/main/l/linux-5.8.0-sunxi/'
>>> +                   'linux-image-dev-sunxi_20.08_armhf.deb')
>>> +        deb_hash = 'ae553a9f7d43b18abfa8f3e64bf2d31878b9be89'
>>>          deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
>>>          kernel_path = self.extract_from_deb(deb_path,
>>> -                                            '/boot/vmlinuz-4.20.7-sunxi')
>>> +                                            '/boot/vmlinuz-5.8.0-sunxi')
>>>          dtb_path = 
>>> '/usr/lib/linux-image-dev-sunxi/sun8i-h3-orangepi-pc.dtb'
>>>          dtb_path = self.extract_from_deb(deb_path, dtb_path)
>>>          initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
>>> @@ -629,7 +629,7 @@ class BootLinuxConsole(LinuxKernelTest):
>>>          exec_command_and_wait_for_pattern(self, 'cat /proc/cpuinfo',
>>>                                                  'Allwinner sun8i Family')
>>>          exec_command_and_wait_for_pattern(self, 'cat /proc/iomem',
>>> -                                                'system-control@1c00000')
>>> +                                                '1c00000.system-control')
>>>          exec_command_and_wait_for_pattern(self, 'reboot',
>>>                                                  'reboot: Restarting 
>>> system')
>>>          # Wait for VM to shut down gracefully
>>> @@ -641,12 +641,12 @@ class BootLinuxConsole(LinuxKernelTest):
>>>          :avocado: tags=machine:orangepi-pc
>>>          :avocado: tags=device:sd
>>>          """
>>> -        deb_url = ('https://apt.armbian.com/pool/main/l/'
>>> -                   
>>> 'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
>>> -        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
>>> +        deb_url = ('https://apt.armbian.com/pool/main/l/linux-5.8.0-sunxi/'
>>> +                   'linux-image-dev-sunxi_20.08_armhf.deb')
>>> +        deb_hash = 'ae553a9f7d43b18abfa8f3e64bf2d31878b9be89'
>>>          deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
>>>          kernel_path = self.extract_from_deb(deb_path,
>>> -                                            '/boot/vmlinuz-4.20.7-sunxi')
>>> +                                            '/boot/vmlinuz-5.8.0-sunxi')
>>>          dtb_path = 
>>> '/usr/lib/linux-image-dev-sunxi/sun8i-h3-orangepi-pc.dtb'
>>>          dtb_path = self.extract_from_deb(deb_path, dtb_path)
>>>          rootfs_url = 
>>> ('http://storage.kernelci.org/images/rootfs/buildroot/'
>>> diff --git a/tests/acceptance/replay_kernel.py 
>>> b/tests/acceptance/replay_kernel.py
>>> index b79fc8daf8..80a460814f 100644
>>> --- a/tests/acceptance/replay_kernel.py
>>> +++ b/tests/acceptance/replay_kernel.py
>>> @@ -132,12 +132,12 @@ class ReplayKernel(LinuxKernelTest):
>>>          :avocado: tags=arch:arm
>>>          :avocado: tags=machine:cubieboard
>>>          """
>>> -        deb_url = ('https://apt.armbian.com/pool/main/l/'
>>> -                   
>>> 'linux-4.20.7-sunxi/linux-image-dev-sunxi_5.75_armhf.deb')
>>> -        deb_hash = '1334c29c44d984ffa05ed10de8c3361f33d78315'
>>> +        deb_url = ('https://apt.armbian.com/pool/main/l/linux-5.8.0-sunxi/'
>>> +                   'linux-image-dev-sunxi_20.08_armhf.deb')
>>> +        deb_hash = 'ae553a9f7d43b18abfa8f3e64bf2d31878b9be89'
>>>          deb_path = self.fetch_asset(deb_url, asset_hash=deb_hash)
>>>          kernel_path = self.extract_from_deb(deb_path,
>>> -                                            '/boot/vmlinuz-4.20.7-sunxi')
>>> +                                            '/boot/vmlinuz-5.8.0-sunxi')
>>>          dtb_path = 
>>> '/usr/lib/linux-image-dev-sunxi/sun4i-a10-cubieboard.dtb'
>>>          dtb_path = self.extract_from_deb(deb_path, dtb_path)
>>>          initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
>>>
>>
>>
> 
> Regards,
> Daniel
>