Re: Tests, Documentation

[graydon hoare]

Sascha Brawer <address@hidden> writes:

> First, please write more test cases. I agree that it is boring, dumb, and
> taxing; testing slows down development. But it's also the most reliable
> way to make sure that code really works. If anyone needs to change our
> code in five years, they'll want to know whether their change has any
> serious impact. Without a large test suite, this is close to impossible.

I agree that we should be writing more test cases. but I think of it
in terms of a technical rather than moral failure: our testing system
requires mental energy beyond what we are willing to exert, and
rewards you with a sense of grinding tedium rather than excitement and
success. I think this is a result of mostly technical issues:

  - mauve should be arranged such that:
    - it's fetched, configured and built as part of classpath
    - it builds and runs with the JVM / compiler combination you
      configured classpath with
    - tests can be added inline with a source file, or "right next
      door" in a similarly-named class or package.
    - tests are run with "make check" from classpath, using $CWD as
      the package restriction when in a subdirectory
    - it uses simple introspection conventions rather than scanning
      text in a source tree and writing makefiles for itself

  - it should provide sub-frameworks for easily testing:
    - serializability
    - comparability, equality, hashing, toString, etc.
    - container protocols
    - threading protocols
    - property (bean) protocols
    - event listening / observer protocols
    - null-pointer sensitivity

  - it should provide (or at least provide a place to hook up)
    loopback, in-memory versions of:
    - "remote" web server
    - filesystem
    - window system
    - database server

  - it should provide generic facilities for doing:
    - pseudorandom / antirandom test vectors
    - all pairs / all n-tuples test reduction
    - noise-tolerant boundary / range checking
    - coverage and profiling analysis
    - establishing and releasing invariants across methods
    - establishing and releasing test prerequisites ("fixtures")

these are not too much to ask; existing JUnit-based suites often
provide some or all these as extensions to the basic TestCase class,
and they make testing feel much more productive. 

without enhancing the framework -- judging from the tests we've
accumulated so far -- I don't think the mauve suite is ever going to
offer you much assurance about whether a change has "serious impact".

> Second, please write understandable documentation for every single
> method. Developers should be able to create free programs without
> referring to external documents. These could disappear at any time.

I think javadoc is a very poor documentation system. it makes the same
mistake as the rest of the java libraries: that verbosity and
factoring into ever-smaller parts is a prima facie duty.

imo it should not be. it only works in your favour up to a point, then
it works against you. to understand a package or logical concept in
javadoc requires you to have dozens of browser tabs open, flipping
back and forth and assembling knowledge in pieces, like a jigsaw
puzzle. it's a very unpleasant way of reading.

I would much rather each class -- or each *package* -- had 1 javadoc
comment, maybe with some diagrams or the like, explaining its
operation in broad detail and pointing out salient features and how
they relate to one another. method-by-method operation is not
particularly useful to document, especially since it's free code and
you can always just read what the method does.

-graydon