Re: avocado test failing INTERRUPTED for "Missing asset"

[Cleber Rosa]

On Wed, Jul 26, 2023 at 3:34 AM Philippe Mathieu-Daudé
<philmd@linaro.org> wrote:
>
> On 25/7/23 19:13, Peter Maydell wrote:
> > Currently this CI job is failing:
> >
> > https://gitlab.com/qemu-project/qemu/-/jobs/4737819946
> >
> > because:
> >
> > (05/59) 
> > tests/avocado/boot_linux_console.py:BootLinuxConsole.test_arm_exynos4210_initrd:
> > INTERRUPTED: Missing asset
> > https://snapshot.debian.org/archive/debian/20190928T224601Z/pool/main/l/linux/linux-image-4.19.0-6-armmp_4.19.67-2+deb10u1_armhf.deb\nRunner
> > error occurred: Timeout reached\nOriginal status: CANCEL\n{'name':
> > '05-tests/avocado/boot_linux_console... (90.67 s)
> >
> > Why is a "Missing asset" causing a timeout after 90 seconds,
> > rather than being accounted as a "SKIP" ("missing requirements
> > in the test environment" sounds like what we have here) ?
>
> Maybe something to report to upstream Avocado.
>
>

Hi Philippe,

Please check my response to Peter's first message on this thread.
It's a rather long answer, but I hope this behavior is understandable.

> That said, we want CI to be reproducible. If we fetch assets from
> unreliable sources, we can't be reproducible. If we are unable to
> provide a assets cache, we'll keep hitting this problem. If we can
> not find a way to have assets stored (requiring sysadmin time setting
> up some infra, possibly only for GitLab), then I'd consider stopping
> running tests depending on external assets on CI; otherwise at some
> point we'll get tired to waste time figuring out the same problem.
>

Right, in an ideal world, we could have a master list of all the
assets that every single job will need, and require an admin to make
sure each and every one of them is cached before running a job.  The
current approach with Avocado's pre-job "fetch asset" plugin is to do
as much as possible without having duplication of the assets URLs in
such a "asset master list" and in the test code.

Also, it will not abort a job if any of these assets fail to be
fetched.  It's a convenient choice that, on the other hand, yields
lower reproducibility and reliability.

> As a maintainer I'm happy to run the avocado tests using my local
> assets cache, and I would rather keep using the framework. But then
> my cache is likely different from others (users, maintainers, CI).
> Similarly few users/maintainers end up having the same cache and
> running the same set of tests.
>
> $ du -chs ~/avocado/data/cache/
> 5.7G    /Users/philmd/avocado/data/cache/
>
> Some files are older than 3 years, and I'm happy to still run the
> tests depending on them (although they disappeared from their
> original http server).
>

This is a well maintained cache! :) Can we rsync from it? :)

Jokes aside, I'm open for ideas on how to better balance this
convenience versus reliability question.

Thanks,
- Cleber.