Re: [PATCH v3 05/24] tests/functional: Prepare the meson build system for the functional tests

[Thomas Huth]

On 01/08/2024 18.12, Philippe Mathieu-Daudé wrote:
> Hi,
>
> On 30/7/24 19:03, Daniel P. Berrangé wrote:
> > From: Thomas Huth <thuth@redhat.com>
> >
> > Provide a meson.build file for the upcoming python-based functional
> > tests, and add some wrapper glue targets to the tests/Makefile.include
> > file. We are going to use two "speed" modes for the functional tests:
> > The "quick" tests can be run at any time (i.e. also during "make check"),
> > while the "thorough" tests should only be run when running a
> > "make check-functional" test run (since these tests might download
> > additional assets from the internet).
> >
> > The changes to the meson.build files are partly based on an earlier
> > patch by Ani Sinha.
> >
> > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > ---
> >   tests/Makefile.include       | 11 ++++++
> >   tests/functional/meson.build | 66 ++++++++++++++++++++++++++++++++++++
> >   tests/meson.build            |  1 +
> >   3 files changed, 78 insertions(+)
> >   create mode 100644 tests/functional/meson.build
>
>
> > +foreach speed : ['quick', 'thorough']
> > +  foreach dir : target_dirs
> > +    if not dir.endswith('-softmmu')
> > +      continue
> > +    endif
> > +
> > +    target_base = dir.split('-')[0]
> > +    test_emulator = emulators['qemu-system-' + target_base]
> > +
> > +    if speed == 'quick'
> > +      suites = ['func-quick', 'func-' + target_base]
> > +      target_tests = get_variable('tests_' + target_base + '_quick', []) 
> > + tests_generic
> > +    else
> > +      suites = ['func-' + speed, 'func-' + target_base + '-' + speed, speed]
> > +      target_tests = get_variable('tests_' + target_base + '_' + speed, [])
> > +    endif
> > +
> > +    test_deps = roms
> > +    test_env = environment()
> > +    if have_tools
> > +      test_env.set('QEMU_TEST_QEMU_IMG', meson.global_build_root() / 
> > 'qemu-img')
> > +      test_deps += [qemu_img]
> > +    endif
> > +    test_env.set('QEMU_TEST_QEMU_BINARY',
> > +                 meson.global_build_root() / 'qemu-system-' + target_base)
> > +    test_env.set('QEMU_BUILD_ROOT', meson.project_build_root())
> > +    test_env.set('PYTHONPATH', meson.project_source_root() / 'python:' +
> > +                               meson.current_source_dir())
> > +
> > +    foreach test : target_tests
> > +      test('func-@0@/@1@'.format(target_base, test),
> > +           python,
> > +           depends: [test_deps, test_emulator, emulator_modules],
> > +           env: test_env,
> > +           args: [meson.current_source_dir() / 'test_' + test + '.py'],
> > +           protocol: 'tap',
> > +           timeout: test_timeouts.get(test, 60),
> > +           priority: test_timeouts.get(test, 60),
> > +           suite: suites)
> > +    endforeach
> > +  endforeach
> > +endforeach
>
> Can we have meson report why a test is skipped? All Python
> unittest @skip* decorators provide a 'reason' argument, so
> we already have that information.

I guess you'd need to extend the meson test runner for this...

> Not much help on meson doc:
> https://mesonbuild.com/Unit-tests.html#skipped-tests-and-hard-errors
>
> If I run a test manually I see the reason in the tap output:
>
> $ FOO=bar tests/functional/test_mips64el_loongson3v.py
> TAP version 13
> ok 1 test_mips64el_loongson3v.MipsLoongson3v.test_pmon_serial_console # SKIP 
> untrusted code
> 1..1

You can also see this information in the txt file in the meson-logs folder, 
so it is certainly not completely lost.

 Thomas