Re: [RFC] Automated QA build system

[Richard Frith-Macdonald]

On 24 Feb 2005, at 05:39, Alex Perez wrote:

> Fred Kiefer wrote:
> > I want to second Adam on this, especially as it was me who noticed 
> > the missing files in the first place. CVS is not the greatest of all 
> > version management tools, it is very easy to miss out on submitting 
> > single files. (You would know, if you were coding)
>
> Is this supposed to be an insult? If so, it fell flat on its face. 
> I've used CVS for years. Also, the GNUstep website itself is in CVS, 
> which I update periodically on a fairly regular basis. Surely Richard 
> has far more experience with CVS than I do.

I think he was just suggesting adopting a reasonable/patient attitude 
to CVS ... and that perhaps
doing lots of cvs commits would help (of course, if yours never, ever 
go wrong, it's unlikely to promote tolerance ... but clearly he assumes 
yours would occasionally go wrong).

> > All of us really should try hard to avoid this from happening, but 
> > when it happens its fine to just correct it as soon as possible.
>
> If you tested the actual code which gets committed (which IMHO is 
> really a basline requirement) then this would not happen.

That's logical nonsense ... you can't test the code which gets 
committed until after it has been committed ... at which point the 
commit has already happened.

In this case the code was tested on three operating systems before the 
commit, (several cycles of test/fix before testing was ok) and the 
results of the commit itsself were tested (and fixed) afterwards (with 
a gap of a few hours as I had other things to do, but never the less, 
it was tested).

>  Things which break the entire build cycle should be taken more 
> seriously.
>
> Anyways, my comments were merely meant as constructive criticism. It's 
> frustrating to check out CVS two times in a week and have it fail to 
> build for such a trivial reason.

Yes ... but in large part the point of CVS is so you can get out 
different versions in the event of a problem at point in time.
While I like to keep CVS as stable as reasonably possible, I do not 
believe that developers should waste a lot of time/energy on the issue 
... I seem to recall that someone was doing regular snapshots of stable 
systems for people who didn't want to spend the few extra minutes 
needed to revert their checked-out state by a few hours when they 
happen to check out a broken version (which happens occasionally even 
when no mistakes have been make committing).

> After a discussion on #GNUstep with Stefan Urbanek (stiivi), I will 
> pass along his idea/suggestion. Some sort of automated QA mechanism 
> would be great, where, upon a new CVS check-in to -base, for example, 
> it would be checked out by an automated QA system and compiled on 
> Linux/x86 as a baseline test. The results of the compilation would be 
> placed on a webpage. If compilation breakage to any part of -core 
> happened, a message would be sent to -dev notifying the core team that 
> this had happpened, the offending file, and the version of the file in 
> CVS at which the breakage occurred.
>
> Does anyone think this would be valuable? It could be expanded 
> upon/combined to do unit testing as well as pretty much any other 
> Quality Assurance task you could dream up.

I doubt that there is a lot of point doing it for gnu/linux/x86 ... as 
that's what most (all?) the people doing
work on the system will be testing on anyway ...  it might notify us of 
a few problems quicker though ... which is a good thing as long as we 
don't rely on it as a substitute for the testing we are doing now 
(which would certainly be a temptation).

> I'd appreciate hearing everyone's thoughts on this (that includes all 
> of the core devs).
>
> I'd also like to set this up for win32, since that seems to be the 
> thing that breaks fairly often as other patches go into -core.

Yes ... for win32 and bsd (and perhaps solaris) this would be really 
useful, as these systems are infrequently used and a problem there can 
go unnoticed in cvs for a long time as most developers do not have the 
time (or the machines available) to test on these systems.