Re: New "make benchmark" target

[Pip Cet]

"Eli Zaretskii" <eliz@gnu.org> writes:

>> Date: Mon, 30 Dec 2024 21:34:55 +0000
>> From: Pip Cet <pipcet@protonmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>, stefankangas@gmail.com, mattiase@acm.org, 
>> eggert@cs.ucla.edu, emacs-devel@gnu.org
>>
>> > I'm open to patches to elisp-benchmarks (and to its hypothetical copy in
>> > emacs-core).  My opinion that something can potentially be improved in
>>
>> What's the best way to report the need for such improvements?
>
> Since you've pushed that to a branch, I suggest to submit bug reports
> about these issues, using "[scratch/elisp-benchmarks]" in the Subject
> of the bug.

Okay.

>> > it (why not), but I personally ATM don't understand the need for ERT.
>>
>> Let's focus on the basics right now: people know how to write ERT tests.
>> We have hundreds of them.  Some of them could be benchmarks, and we want
>> to make that as easy as possible.
>
> We can later add more benchmarks using ERT.  There's no contradiction.

I agree.  There's definitely no "right now" need for ERT, I was
explaining why it's the change I'll be investigating.

>> It also allows a third class of tests: stress tests which we want to
>> execute more often than once per test run, which identify occasional
>> failures in code that needs to be executed very often to establish
>> stability (think bug#75105: (cl-random 1.0e+INF) produces an incorrect
>> result once every 8 million runs).  IIRC, right now ERT uses ad-hoc
>> loops for such tests, but it'd be nicer to expose the repetition count
>> in the framework (I'm not going to run the non-expensive testsuite on
>> FreeDOS if that means waiting for a million iterations on an emulated
>> machine).
>>
>> (I also think we should introduce an ert-how structure that describes how
>> a test is to be run: do we want to inhibit GC or allow it?  Run some
>> warm-up test runs or not? What's the expected time, and when should we
>> time out? We can't run the complete matrix for all tests, so we need
>> some hints in the test, and the lack of a test declaration in
>> elisp-benchmarks hurts us there).
>
> These seem to be long-term goals of improving the benchmark suite.
> They are fine by me, but I don't see why they should preclude
> installing the benchmarks we have without first converting them to
> ERT.  We can do that later, if we decide it's worth the effort.

We seem to agree here: my intention, too, is to merge the
elisp-benchmarks branch ASAP.  Let's establish which changes are
required on that branch, then do a synchronized rebase-merge to preserve
history?

Pip