I've finished the core of the regression tester. You can now run it and generate a html report but you will need a few tools installed: imagick, xwd, npm, pretty-diff and xvfb. This should all be listed in ft-regression.sh's comments.If you want to run it locally make sure you have a ~/test-fonts with .ttf files in it. You will also need freetype2 and freetype2-demos sources next to each other. You can run it locally by calling: ./CI/ft-regression.sh <commit> eg: ./CI/ft-regression.sh HEAD~1 from inside your freetype2 directory. It should generate an easy to digest report to /tmp/ft-tests/index.html. This is basically what I wanted to accomplish for my GSOC project but I can do a lot of the things mentioned in my previous message if I pass my evaluation.
I've also got the testing apparatus running on azure pipelines with the other tests here:
Ignore the mingw build failures as they seem to be in the midst of a regime change and their new signed key databases aren't in sync with azure.
You can download the report it generates from here:
The file is rather large. In the future, I could probably shrink it quite a bit by using 7zip instead of regular zip or I could split the tests into smaller chunks too.
The report doesn't demonstrate the image comparison because there are obviously no regressions between my commit and master. However, if there is one it will generate a page where you can see the differences between images on mouse over. For text files it generates a diff report using pretty-diff. You can see this in the freetype-bench comparisons.
As I've said this should be mostly working now. I also believe I've commented the source pretty thoroughly. I've also moved the scripts to a subdir as requested. If there are any changes / improvements you would like to see to the reports or scripts please let me know. If not, I have listed several things I would like to do to polish it in my previous message.