Re: make check is broken (again) - patch testing seeming to taking more of my time than I like

[David Kastrup]

Dan Eble <address@hidden> writes:

> In the past month, I've devoted many hours to testing my submissions,
> but clearly the effort is not achieving the goal.

If the goal is perfection, it is hard to achieve.

> I request some help to understand how I can improve my pre-commit
> testing procedures, and where my responsibilities begin and end.

Responsibilities don't end.  We are a community project: anybody who can
change something to the better is advised not to consider stuff
"somebody else's problem" since we don't really have _assigned_
responsibilities.  In return, we don't have assigned blame.  Tracking
down the source of a problem may seem like that, but it's not like
people aren't trying their best, and it's not like it's always easy to
achieve.

> I enjoy having my commits reverted as much as others enjoy having
> their build broken--it is a big waste of pro-bono time--so I want to
> understand the issues clearly.

To put that in perspective: a revert is not a big waste of time: in
contrast it puts a problem from affecting everybody's time back into a
state where it only affects a single person's time.  Fixes to reverted
material often end up trivial, not costing much actual personal work
time (though a time lag).

The main resource we are trying to preserve with our procedures is
mental energy and goodwill of everybody involved.  Most of the time, it
works out well enough.

A larger part than initially planned for is done by James: this part has
become larger since our automated testing procedures for individual
patches have not survived the migration from Google Code to Sourceforge.
Since he now does quite a bit of work manually, the reproducibility has
suffered a bit: maybe we would need to provide more scripts/automation
for offline testing for the case that the online test processing is not
fully operative (like it is now).

> How are build-breaking changes getting into the master branch? CG
> section 3.4.10 says that the reason for pushing to staging is that
> automated tests are run before changes are moved to master.  What
> specifically is being tested?

Basically make, make test, and make doc.  At the current point of time,
the main testing platform for the staging branch testing is my personal
laptop which tends to run a current or beta version of Ubuntu.

GUB is usually (but not necessarily so) run on LilyDev, a virtual Ubuntu
image.  The tools in GUB tend to be updated as needed, so there often
are older GCC and/or Python versions involved here.

It has not been rare in the past that development went forward for a
minor release and just at release time it was detected that the tools
used in GUB would not support some of the new code.

> And days before that happens, patches are announced as having been
> tested with the feedback "Passes make. make check and a full make
> doc."  The evidence suggests that that does not include running
> autogen, otherwise it should have caught the problem with "tidy" that
> my own testing failed to catch.

Check James' procedure: he has posted it a few times.  Formalizing it
would create scripts for it.

> Should things such as missing optional programs and new-ish Python
> syntax be rejected at either of these stages?  If not, then it would
> seem to fall to the submitter to set up an alternate development
> environment with Python 2.4, GCC 3.4, and similarly aged versions of
> other tools, and run additional tests in that environment.

We don't have as comprehensive tests as that and don't really demand
it.  One should just be conservative.  The cost of a mistake in that
regard may well be a revert or a timely followup patch.

-- 
David Kastrup