Re: [PATCH] Patches for problems discovered with interactive testing

[gerard . vermeulen]

On 19.01.2024 13:50, Ihor Radchenko wrote:
> gerard.vermeulen@posteo.net writes:
>
> > This patch addresses a number of glitches discovered during 
> > interactive
> > testing and contains possible fixes (sub-patch test-ob-maxima fixes
> > batch testing on my system, but not interactive testing).
>
> Thanks for your interest in improving Org mode tests!
>
> > The test-ob-maxima patch may be due to a different Maxima version, but
> > I do not think so.  In batch testing test-ob-maxima passes contrary to
> > interactive mode.
>
> On my side, Maxima tests pass both interactively and non-interactively.
> I use Maxima 5.47.0
> What is your version?
The same: maxima --version
Maxima 5.47.0

In batch testing without the patch I get:

Test ob-maxima/batch+verbatim condition:
    (ert-test-failed
     ((should
       (equal (org-babel-execute-src-block)
              "(assume(z > 0),integrate(exp(-t)*t^z,t,0,inf))\n
                gamma(z + 1)"))
      :form
      (equal
       "(linenum:0,\n(assume(z > 0),integrate(exp(-t)*t^z,t,0,inf))\n    
                             gamma(z + 1)"
       "(assume(z > 0),integrate(exp(-t)*t^z,t,0,inf))\n                 
                gamma(z + 1)")
      :value nil :explanation
      (arrays-of-different-length 104 92
                                  "(linenum:0,\n(assume(z > 
0),integrate(exp(-t)*t^z,t,0,inf))\n                                 
gamma(z + 1)"
                                  "(assume(z > 
0),integrate(exp(-t)*t^z,t,0,inf))\n                                 
gamma(z + 1)"
                                  first-mismatch-at 1)))
   FAILED   162/1252  ob-maxima/batch+verbatim (0.694916 sec) at 
../lisp/test-ob
-maxima.el:71

I do not know where the difference comes from.  I am not a Maxima user
and not inclined to track this down unless you tell me it is important.
> > The test-ob-python patch unsets org-src-preserve-indentation which I
> > have set.
> >
> > The test-ob patch also unsets org-src-preserve-indentation.
>
> > The test-org patch also unsets org-src-preserve-indentation.  This
> > reduces the number of failures to 2 while testing test-org
> > interactively.
>
> It is generally not expected that Org tests will pass when you try to
> run them on Emacs instance with your personal config. This is because
> you can have arbitrary customizations, hooks, or advices that change 
> the
> Org mode defaults.
>
> let-binding a couple of variables cannot fix this general problem for
> all scenarios. So, I do not see any reason to accept patches that make
> the tests run with personal config of a single user.

Before I made the patch, I had reverted changes that worked 
interactively
in my personal config but caused failures with make test.

Because of the Maxima test failure (also in batch), I was not sure that 
all
tests pass on a reference system.

> What might be useful, however, is making sure that tests do not fail
> interactively with a clean Emacs config (make repro).
> Check out testing/README file.

I have a super-weird system where make repro opens Emacs with
as default-directory my home directory.  So I have to make symbolic
links from my home to the testing and lisp directories.
But make test works.

Because of testing my 
0001-org-babel-demarcate-block-split-using-element-API.patch
interactively in my personal config (before make test which takes
quite some time), I figured out that it would guess to suppress false
failures.
> >  (ert-deftest ob-maxima/string-input ()
> >    "Test of string input"
> >    (org-test-at-id "b5842ed4-8e8b-4b18-a1c9-cef006b6a6c8"
> >      (org-babel-next-src-block 2)
> > -    (should (equal "- sin(x)" (org-babel-execute-src-block)))))
> > +    (should (equal '(("(linenum:0," "") ("-" "sin(x)"))
> > +                   (org-babel-execute-src-block)))))
>
> The test in question is testing the output of
>
> #+begin_src maxima :var fun="sin(x)" :var q=2 :results silent
> print(diff(fun, x, q))$
> #+end_src
>
> It is certainly not expected that such code block returns "linenum:0".

I would say that it is an unexplained difference, I still get (with 
"linenum:0") :

#+header: :wrap "src text -n"
#+name: lst:maxima-example
#+begin_src maxima -n :batch batch :cmdline --quiet :exports both 
:results raw
rat(1/(x+1) + x/(x-1));
#+end_src

#+name: lst:maxima-example-results
#+results: lst:maxima-example
#+begin_src text -n
(%i1) (linenum:0,
(%i1) rat(1/(x+1)+x/(x-1))
                                  2
                                 x  + 2 x - 1
(%o1)/R/                         ------------
                                     2
                                    x  - 1
#+end_src

> Tests in Org mode are verifying that Org mode behaves correctly. It
> makes no sense to change the test when Org mode result is not correct -
> we should instead fix Org mode.
>
> In the majority of cases, if the test is failing, it is not something
> wrong with the test, but something wrong with Org mode code that test 
> is
> verifying.
>
> >  (ert-deftest test-ob/does-not-replace-a-block-with-the-results ()
> >    (org-test-with-temp-text "#+NAME: foo
> > diff --git a/testing/lisp/test-org-element.el 
> > b/testing/lisp/test-org-element.el
> > index ca7d77e28..f0958ff59 100644
> > --- a/testing/lisp/test-org-element.el
> > +++ b/testing/lisp/test-org-element.el
> > @@ -4457,7 +4457,7 @@ Text
> >       '(paragraph nil "  Two spaces\n\n \n  Two spaces"))
> >      '(paragraph nil "Two spaces\n\n\nTwo spaces")))
> >    (should
> > -   (equal
> > +   (equal ;; Fails in test. Why? (equal ...) form works in ielm.
>
> It would help if you provided the error ERT displayed when the test 
> failed.

I cannot reproduce it.  Now it works in interactive testing (personal 
config
as when I first saw this).

If you ask me, I am still willing to collect the 2 sub-patches with the
let bindings which improve testing interactively in personal configs
somewhat.  If not, it is OK (I understand now that it is not important).

Regards -- Gerard