Re: Automate Emacs UI testing?

[joakim]

John Wiegley <address@hidden> writes:

>>>>>> Óscar Fuentes <address@hidden> writes:
>
>> As Emacs has no strong test suite (automatic testing of UI is hard, so no
>> cure for this on sight) neither has many resources for steady manual testing
>> and bug-fixing, I'm afraid that switching to the cadence model would
>> deteriorate the quality of releases.
>
> Web developers have some pretty awesome tools for driving a UI and observing
> the resulting behavior. I wonder what it would take to have a framework like
> that for Emacs...

I work with test automation periodically.

Web testing is a special case, because there are frameworks that can
work with the browser document object model, to simulate a user, and verify
results. These tools wouldn't work with Emacs.

There is another class of tools that work at the windowing level,
including http://www.sikuli.org/. It works by simulating events, and
then analyzing the on-screen results with OpenCV, which is a computer
vision framework. I have worked with this tool, and it works, but is
tricky to set up. I have been wanting to set up my own tests with Sikuli
and Emacs, because many emacs packages I use randomly fail after
upgrade. If I could verify that everything works in a separate
environment I wouldn't need to upgrade my main emacs. 

This is the testing scenario I have worked on.

- Isolate an Emacs build and environment in a Docker container
  environment. This works well, but it's a little bit tricky getting the
  containerized Emacs to access the host screen. It is a nice strategy,
  because you can easily simulate a number of gnu/linux distributions. I
  used this strategy for my xwidget branch.
  
- Make sikuli tests that test things like Helm and Yasnippets. I haven't
  done more than verifying that the strategy works. Here is a container
  with Sikuli that I havent tested: 
https://hub.docker.com/r/kkochubey1/sikuli-chrome-x11vnc/

- Do these tests on each commit in a Jenkins build server that I
  have. It could be any build server with the right configuration though.






>
> John
>

-- 
Joakim Verona