Re: Mauve test question

[Thomas Zander]

On Thursday 30 December 2004 08:58, you wrote:
> Thomas Zander wrote:
> > This brings the running of mauve back to one 'java -jar' and
> > then just following instructions.  (that 'java' is just an
> > example; it should work on any JVM)
> > here is the jar file: http://members.home.nl/zander/alltests.jar (1mb)
>
> Thank you! I really like this idea. A couple of thoughts/requests:
> - We need to get rid of the gnu.testlet.config interface for this to
> work
Agreed.
The simplest solution to this problem IMO is to provide a .bat and .sh 
script.  The 'java -jar mauve.jar' would then not only write the "tests" 
file, but also these files.  This means that the shell-script would be 
responsible for providing the pathseparator and all those variables, since 
we obviously can't rely on the chosen jvm to provide them.   Right?
The script would then pass the variables to the runner on the command line.
That a good idea?
  User does a jamvm -jar mauve.jar
  types which JDK version he wants to test and 
  is then instructed to start 'mauve.sh' or 'mauve.bat'

> - Can you please write out the tests file with the correct line
> separator?
with 'correct' you mean the one from config.java?  It now writes a '\n'. 
Which I thought would be correct. No?

> - Can you please compile the classes to target a 1.4 VM?

Ehh; good point!  Done.

> Another thing (unrelated to your work) is that Mauve really should clean
> its temp directory before starting the test run (or maybe even generate
> a unique temp directory for each run).

Since the output of SimpleTestHarness is insufficient for parsing and 
aggregating the data, as well as the wanted new feature to pass 
path-separator and friends, these things should be used to create a new 
TestHarness.
I've looked at Davids work:
http://sources.redhat.com/ml/mauve-discuss/2004-q4/msg00045.html
Looks good and I think I can add a XMLOutputter to it, plus fix some of the 
other long standing issues; like a good tmp dir, I already fixed some 
mistakes he made :-)

The XML outputter should then only output failed values.  The testruns of 
all JVMs combined give a number of XML files, which can then be read and 
aggregated by a webserver or a GUI tool.
What about this:
<testrun tests="jdk1.5">
<test class="java/awt/geom/CubicCurve2D/solveCubic">
    <fail>
        <!-- checkpoint and number are unique per test, right ? -->
        <checkpoint>0.25x^3 - 1.0x^2 = 0</checkpoint>
        <number>1</number>
        <!-- this run resulted in the following value: -->
        <result>2</result>
        <!-- the test checked and expected this value: -->
        <expected>1</expected>
        <!-- the default value is the value that the Sun JVM produced, this
            tag should only be added when its different from the expected
            value -->
        <default>2</default>
    </fail>
</test>
</testrun>

Comments?
-- 
Thomas Zander

pgpvpJycmia8C.pgp
Description: PGP signature