Re: [Chicken-janitors] #630: Test scripts that don't properly report exit status

[Chicken Trac]

#630: Test scripts that don't properly report exit status
-------------------------+--------------------------------------------------
  Reporter:  mario       |       Owner:                     
      Type:  defect      |      Status:  new                
  Priority:  major       |   Milestone:                     
 Component:  extensions  |     Version:  4.7.x              
Resolution:              |    Keywords:  test infrastructure
-------------------------+--------------------------------------------------

Old description:

> The eggs listed bellow don't properly report their exit code to the test
> infrastructure.
>
> Please, make them exit 0 when all tests pass or a non-zero value when one
> or more tests fail, so salmonella can properly indicate the tests status.
> Hint: for those using the [http://wiki.call-cc.org/egg/test test egg],
> you just need {{{(test-exit)}}} at the end of the test script.
>
> If you test script relies on errors (or {{{assert}}}) to test egg
> features, you can assume it is ok (I'm sorry for the false alert), since
> on error the test script will exit 70 (which is ok for the test
> infrastructure).
>
> I took a look at the last 20 lines of the run.scm files (or at the files
> they include/load) in the trunk dir for each egg (yeah, it's not exactly
> a pleasant work).  If you don't intend to make a new release of your
> egg(s) to fix bugs or to add new features soon, please consider making a
> new release just to fix the tests, so we can have a more accurate idea
> about the shape of the set of eggs.  Also consider that egg tests help to
> test the compiler too.
>
> - condition-utils (Kon Lovett)
> - discount (Stephen Eilert)
> - dsssl-utils (Kon Lovett)
> - elliptic-curves (Thomas Chust)
> - err5rs-arithmetic (Kon Lovett)
> - fmt (Alex Shinn)
> - format (Felix Winkelmann, Kon Lovett)
> - list-utils (Kon Lovett)
> - lookup-table (Kon Lovett)
> - macosx (Kon Lovett, unsure whether this can be automated)
> - mailbox (Felix Winkelmann, Kon Lovett)
> - mathh (Kon Lovett, John Cowan)
> - modular-arithmetic (Thomas Chust)
> - mw (Tony Sidaway. Uses testeez, which has no way to exit on failure?)
> - mw-core (Tony Sidaway. Uses testeez, which has no way to exit on
> failure?)
> - numspell (Felix Winkelmann(?) uses testeez)
> - operations (Felix Winkelmann)
> - pandora (Thomas Chust)
> - procedure-decoration (Kon Lovett)
> - sexp-diff (Felix Winkelmann)
> - specialized-io (Jeronimo Pellegrini)
> - srfi-101 (Kon Lovett.  Note: replacing {{{system}}} with {{{system*}}}
> in {{{run.scm}}} is enough)
> - srfi-19 (Kon Lovett. Note: {{{run-s19-tests}}} already returns number
> of failures)
> - srfi-27 (Kon Lovett. Note: {{{system}}} -> {{{system*}}})
> - srfi-45 (Kon Lovett)
> - stack (Kon Lovett)
> - statistics (Peter Lane. Uses srfi-78, which has no {{{test-exit}}}
> equivalent)
> - sxml-templates (Moe Aboulkheir, Peter Danenberg)
> - symbol-utils (Kon Lovett)
> - synch (Kon Lovett)
> - system (Felix Winkelmann)
> - tinyclos (Felix Winkelmann.  Uses testeez and is deprecated)
> - uuid-ossp (Kon Lovett. Note: meta-file says it needs testeez, but it
> uses test egg)
> - yasos (Kenneth Dickey, Juergen Lorenz. Uses testeez)
> - yelp (Derrell Piper)
>
> The following eggs have tests that aren't really automated.  If they
> can't be automated or you don't feel like doing that, please remove your
> eggs from the list:
>
> - kanren (Alex Shinn)
> - lazyffi (Felix Winkelmann)
> - openal (Felix Winkelmann)
> - progress-indicators (Felix Winkelmann)
> - remote-mailbox (Kon Lovett)
> - xosd (Felix Winkelmann)

New description:

 The eggs listed bellow don't properly report their exit code to the test
 infrastructure.

 Please, make them exit 0 when all tests pass or a non-zero value when one
 or more tests fail, so salmonella can properly indicate the tests status.
 Hint: for those using the [http://wiki.call-cc.org/egg/test test egg], you
 just need {{{(test-exit)}}} at the end of the test script.

 If you test script relies on errors (or {{{assert}}}) to test egg
 features, you can assume it is ok (I'm sorry for the false alert), since
 on error the test script will exit 70 (which is ok for the test
 infrastructure).

 I took a look at the last 20 lines of the run.scm files (or at the files
 they include/load) in the trunk dir for each egg (yeah, it's not exactly a
 pleasant work).  If you don't intend to make a new release of your egg(s)
 to fix bugs or to add new features soon, please consider making a new
 release just to fix the tests, so we can have a more accurate idea about
 the shape of the set of eggs.  Also consider that egg tests help to test
 the compiler too.

 - discount (Stephen Eilert)
 - elliptic-curves (Thomas Chust)
 - fmt (Alex Shinn)
 - lookup-table (Kon Lovett)
 - macosx (Kon Lovett, unsure whether this can be automated)
 - mailbox (Felix Winkelmann, Kon Lovett)
 - mathh (Kon Lovett, John Cowan)
 - modular-arithmetic (Thomas Chust)
 - mw (Tony Sidaway. Uses testeez, which has no way to exit on failure?)
 - mw-core (Tony Sidaway. Uses testeez, which has no way to exit on
 failure?)
 - numspell (Felix Winkelmann(?) uses testeez)
 - operations (Felix Winkelmann)
 - pandora (Thomas Chust)
 - procedure-decoration (Kon Lovett)
 - sexp-diff (Felix Winkelmann)
 - specialized-io (Jeronimo Pellegrini)
 - srfi-101 (Kon Lovett.  Note: replacing {{{system}}} with {{{system*}}}
 in {{{run.scm}}} is enough)
 - srfi-19 (Kon Lovett. Note: {{{run-s19-tests}}} already returns number of
 failures)
 - srfi-27 (Kon Lovett. Note: {{{system}}} -> {{{system*}}})
 - srfi-45 (Kon Lovett)
 - stack (Kon Lovett)
 - statistics (Peter Lane. Uses srfi-78, which has no {{{test-exit}}}
 equivalent)
 - sxml-templates (Moe Aboulkheir, Peter Danenberg)
 - symbol-utils (Kon Lovett)
 - synch (Kon Lovett)
 - system (Felix Winkelmann)
 - tinyclos (Felix Winkelmann.  Uses testeez and is deprecated)
 - uuid-ossp (Kon Lovett. Note: meta-file says it needs testeez, but it
 uses test egg)
 - yasos (Kenneth Dickey, Juergen Lorenz. Uses testeez)
 - yelp (Derrell Piper)

 The following eggs have tests that aren't really automated.  If they can't
 be automated or you don't feel like doing that, please remove your eggs
 from the list:

 - kanren (Alex Shinn)
 - lazyffi (Felix Winkelmann)
 - openal (Felix Winkelmann)
 - progress-indicators (Felix Winkelmann)
 - remote-mailbox (Kon Lovett)
 - xosd (Felix Winkelmann)

--

Comment(by sjamaan):

 Thanks Kon!  I've removed these eggs from the master list so we can more
 clearly see which ones still need fixing

-- 
Ticket URL: <http://bugs.call-cc.org/ticket/630#comment:24>
Chicken Scheme <http://www.call-with-current-continuation.org/>
Chicken Scheme is a compiler for the Scheme programming language.