Re: [PATCH v5 1/5] iotests: remove 'linux' from default supported platforms

[Peter Maydell]

On Fri, 4 Oct 2019 at 14:50, Max Reitz <address@hidden> wrote:
> On 04.10.19 15:16, Peter Maydell wrote:
> > 'make check' does have the restriction
> > that we don't want the tests to take too long to run, but in
> > general the block layer should be running some reasonable subset
> > of tests in the project's standard "run the tests please" command.
> > (I have no opinion on exactly what that subset ought to be, beyond
> > that it would be good if that subset doesn't have known intermittent
> > failures in it.)
>
> Deciding the subset is difficult.  My stance was that it’s better to not
> choose an arbitrary subset here but ensure that really everything gets
> run as part of CI, that is asynchronously so it doesn’t block anyone and
> can take as long as it wants.

I wonder if we have a terminology difference confusion here.
To me "CI" means "we have tests which we automatically run
before pushing commits to master, and if they fail then we
don't push". So (a) they have to run synchronously and (b) there
is a need for them to run in a reasonable period of time because
otherwise it takes too long to run the tests before pushing
to master and we get a backlog of unprocessed pullrequests
and annoying delays.

> If we decide to get pull requests based on that, then that’d bring even
> more pain, but at least it’d be honest.  But just running half of the
> qcow2 tests to me seems only like we want to pretend we ran the iotests.
>  So for me, iotests still break, and we need to deal with make check
> failures on top.  I’d at least prefer one or the other.

I think the ideal we're aiming for here is:
 (1) people doing active work in the block subsystem are probably
often going to want to run all the iotests, and certainly the
subsystem maintainers will want to do that as they put together
pull requests.
 (2) but people who *don't* do active work in the block subsystem
still sometimes touch it in passing as part of things like global
refactorings or other changes that touch big parts of the tree,
or accidentally break it with a change to some other bit of QEMU
entirely. These people won't run the full iotests, but it is
reasonable to expect them to run 'make check', and it would be good
if that caught "whoops you broke the block subsystem".

Similarly, "make check" has incredibly spotty coverage of
various arm boards and devices, but it does do some basic
checks which do catch accidental breakage.

thanks
-- PMM