Re: issue verification

[Michael Käppler]

Am 18.09.2020 um 23:56 schrieb Jean Abou Samra:
> Hi,
>
> Le 18/09/2020 à 11:15, Phil Holmes a écrit :
>
> > I don't know if this isn't clear, but just to state the original
> > point of verifying issues.
> >
> > If the change is claimed to fix a bug, we compile the previously
> > buggy code with the release in which the bug is claimed fixed and
> > check that the bug is no longer there.  It it has really disappeared,
> > we change the status of the issue from Fixed to Verified.  i.e. we
> > are certain that the bug is no longer there.
> >
> > If the change is to provide updated functionality, then it can be
> > really quite hard to verify the new functionality, and in any case
> > the patch review system should do that.  So we simply check the patch
> > was pushed into the claimed build.  If it's clear that it was, we
> > mark the status as Verified.
> >
> > That was the original intention of verifying issues.
>
> Thanks for the insight.
>
> So, Michael and I verified nearly all issues in Status::Fixed.
Actually, Jean did the vast majority, because I had to leave on Friday
afternoon for a rehearsal weekend.

> The ones remaining are listed here:
>
> https://gitlab.com/lilypond/lilypond/-/issues?scope=all&utf8=%E2%9C%93&state=closed&label_name[]=Status%3A%3AFixed&not[milestone_title]=2.21.7
>
>
> Among these,
>
> https://gitlab.com/lilypond/lilypond/-/issues/6027
>
> which I don't know how to test (Jonas?).
>
> The others (5 items) are Windows-related. Maybe Michael or someone
> else with access to a Windows machine can go over them?
I verified them and noticed a couple of another issues.
(#6044-#6046)

Lilypond-book on Windows seems broken to me, because
the correct way to run it is neither obvious nor documented.
The script lacks a file ending (2.20.0 had lilypond-book.py, 2.21.6 only
lilypond-book) and can only be run like 'python lilypond-book foo.lytex'

Jean, what would you propose as a good way to ship these python scripts
for the Windows releases?

> Otherwise, part of the work was extremely simple, namely check that
> the commits had made it into some branch under release/, which is fast;
> and part of it was much harder, when trying to reproduce memory-related
> bugs, for example. I reopened a few of these issues.
>
> This doesn't answer Jonas' initial question: what should be done,
> in terms of policies, with regard to issue verification? I don't
> know. Personally I think it's not really needed as long as every
> bug fix contains a regression test.
Do we agree that 'verification' in the sense of
'checking whether a particular commit made it into a release x.y.z'
is not necessary anymore, because attached merge requests show that
the commit has gone in?

> Speaking of regression tests, I also went through comparison pages
> from "2.21.6 vs. 2.21.5" to "2.21.2 vs. 2.21.1" included. Honestly,
> I've had my share of this; could somebody jump in to verify the
> last two pages as well? These are:
>
> lilypond.org/test/v2.21.1-1/compare-v2.21.0-1/index.html
>
> http://lilypond.org/test/v2.21.0-1/compare-v2.20.0-1/index.html
I find that very difficult, because some changes may be intended, others
not...
Also there are often slight spacing changes that are probably "Heisenbugs",
like James mentioned.
> Depending on your time and energy available, you may even go
> farther. Or simply (I mean simple, not costless) verify the
> entire test suite:
>
> http://lilypond.org/doc/v2.21/input/regression/collated-files.html
>
> That'll be all for today.
Thank you very much, Jean!
> Best,
> Jean