t-test gets NaN instead of 0 for significance on x86-64

[Ben Pfaff]

Some time ago, Matej Cepl <address@hidden> reported that
t-test-alpha3.sh in 0.6.2-pre5 failed on x86-64 with GCC 4.4:

> PASS: tests/bugs/t-test-alpha3.sh
> 19c19
> < #Pair 0|A & B#3|      1.000| NaN#
> ---
>> #Pair 0|A & B#3|      1.000|.000#
> compare output
> FAILED
> FAIL: tests/bugs/t-test-paired.sh

This evening, I've run the same test on an x86-64 machine
(bellini.debian.org) with GCC 4.3.2 and I reproduce this test
failure.  I also get the same failure with 0.6.1, although I have
to run the test by hand there since this test was new in
0.6.2-pre5.

After some fussing, I tracked the source of the NaN to this
calculation in pscbox() in src/language/stats/t-test.q:

      double correlation_t =
        pairs[i].correlation * sqrt (df) /
        sqrt (1 - pow2 (pairs[i].correlation));

In this particular test case, pairs[i].correlation is almost
exactly 1.0, such that 1 - pow2 (pairs[i].correlation) comes out
just slightly negative, making the square root yield NaN.

John, do you have a suggestion for the correct fix?  I don't know
enough about the math here to say.
-- 
Ben Pfaff 
http://benpfaff.org