Re: Reg test diffs for XML - noise in all tests

[Jonas Hahnfeld]

Am Samstag, den 18.07.2020, 17:31 +0200 schrieb Han-Wen Nienhuys:
> On Sat, Jul 18, 2020 at 11:13 AM Jonas Hahnfeld <hahnjo@hahnjo.de> wrote:
> > > Since a few days ago I am now nearly always getting two or four reg test
> > > diffs show, albeit just for cell counts, for XML tests.
> > > 
> > > See example attached.
> > > 
> > > It's not stopping me doing anything but it does add noise and I
> > > wondered, as there are no image diffs ever for XML tests, if this is
> > > pointing to something more serious or potentially problematic?
> > 
> > thanks for raising this. Since MR !230, 'make check' started comparing
> > the regression tests for musicxml and there should be image diffs in
> > case of changes (see the long output of 'make check' for !230 which
> > also contained the fix for chords).
> 
> I actually think this is not the way to go for XML. Rather than
> compare PNG outputs (which mainly measures changes in the formatting
> engine), we should compare the XML files against baseline as text
> files.

I think that's a misunderstanding: The tests take MusicXML, run
musicxml2ly to produce an input for LilyPond and compare the result. We
could of course compare the (textual) input to LilyPond, but I'd argue
that comparing visual representations is easier for humans.

> [...]
> > Moreover, cell counts won't be available with Guile v2.x (at least the
> > property 'total-cells-allocated doesn't exist anymore). Would anybody
> > object to dropping this for Guile v1.8 already? That would potentially
> > also simplify the introduction of automated regression test comparison
> > in the CI pipeline...
> 
> I'm also in favor of dropping this.

Good, I'll prepare the changes. Can I just drop the options dump-
profile and dump-cpu-profile or are there external users?

signature.asc
Description: This is a digitally signed message part