Re: [pdf-devel] Test Specification Document

[jemarch]

Hello.

   > I introduced a new texinfo document in `doc/' named `gnupdf-tsd.texi'
   > (for GNU PDF Library Test Specification Document). It is the place
   > where we will document all the unit tests, subsystem tests and system
   > tests of the library.
   Is this really worth while? It seems quite imprecise - example:

   pdf_realloc 
   Test: pdf_realloc_001  
   Reallocate some bytes. 
   Success condition  
   The call should not produce an error.

   It would be better to express some initial conditions, a set of
   interesting variations (realloc smaller, realloc same size, realloc
   larger), and then check it actually worked (not just that the error
   handling is turned off :-))

We can do it more precise by introducing more fields into the test
descriptions. For example:

Test: pdf_realloc_001
Reallocate some bytes.
Initial Conditions: foo bar
Success condition: ...

As for the variations, can be implemented with more tests. That was the
reason to identify functions with testcases:

Test: pdf_realloc_002
Realloc smaller
...

Test: pdf_realloc_003
Realloc same size
...

Test: pdf_realloc_004
Realloc larger
...

etc

   At this level, expressing the test in C would seem better.

I agree in that it would be quite convenient to document the tests in
the source code. The reason I introduced a separated test
specification document is that we need a place to define the tests
before to implement these: there are separate tasks in the project
plan involving testing:

   - Design the unit tests for foo
   - Implement the unit tests for foo

In that way we can start the design of the unit tests for the
functions of a given module as early as we have the module API
designed.

The testmaster (I would like to appoint someone to play that role;
would you like to dedicate some time for this? :)) will need to
continuously maintain some statistics about the state of the library
validation system:

- How much functions have unit tests designed?
- How much functions have unit tests implemented?
- ...

A script could be written to inspect the TSD, the Reference Manual and
the torture/unit/ source tree and automatically generate such
statistics. 

An alternative is to generate the TSD from the torture/unit/
sources. What do you think about that?

   I am not saying that there shouldn't be a test plan / spec, just that it 
   should be at a higher level - expressing goals rather than detail:

   * Goal: all functions should have unit tests that verify correct operation.
   * Goal: all functions should have unit tests that check bad parameter 
   combinations result in meaningful errors.
   * Goal: subsystems shall...
   * Goal: systems shall...

Something like a test plan is already depicted in the
http://gnupdf.org/Lib:Torture_Chamber wiki page. Maybe would be good
to put that information in a texinfo document instead.