Re: Reg test diffs for XML - noise in all tests

[Han-Wen Nienhuys]

On Sat, Jul 18, 2020 at 11:13 AM Jonas Hahnfeld <hahnjo@hahnjo.de> wrote:
> > Since a few days ago I am now nearly always getting two or four reg test
> > diffs show, albeit just for cell counts, for XML tests.
> >
> > See example attached.
> >
> > It's not stopping me doing anything but it does add noise and I
> > wondered, as there are no image diffs ever for XML tests, if this is
> > pointing to something more serious or potentially problematic?
>
> thanks for raising this. Since MR !230, 'make check' started comparing
> the regression tests for musicxml and there should be image diffs in
> case of changes (see the long output of 'make check' for !230 which
> also contained the fix for chords).

I actually think this is not the way to go for XML. Rather than
compare PNG outputs (which mainly measures changes in the formatting
engine), we should compare the XML files against baseline as text
files.

> I guess the cell count diffs have always been there, just not noticed
> due to lack of comparison. In the past, I also found the counts to be
> very dependent on the number of processes and file ordering, ie the
> order LilyPond processes the input. I haven't investigated yet if
> there's a sort() missing somewhere...

The cell counts have always been iffy.  They will highlight some
grotesque errors, but structural increases in allocation usually also
affect runtimes, so they show up once when doing performance
measurements.

> Moreover, cell counts won't be available with Guile v2.x (at least the
> property 'total-cells-allocated doesn't exist anymore). Would anybody
> object to dropping this for Guile v1.8 already? That would potentially
> also simplify the introduction of automated regression test comparison
> in the CI pipeline...

I'm also in favor of dropping this.

-- 
Han-Wen Nienhuys - hanwenn@gmail.com - http://www.xs4all.nl/~hanwen