[Octave-bug-tracker] [bug #30817] truncation vs. rounding

[Jaroslav Hajek]

Follow-up Comment #1, bug #30817 (project octave):

What you should realize is that when you type 0.999995, the value is already
rounded at that point to nearest (or, in general, a close-by) floating-point
number. It has to be; there is no other way. So you may be deceived at the
spot:

octave:51> x = 9007154218744718 * 2^-53
x =  0.99999
octave:52> x == 0.999995
ans =  1

now with aid of PARI:

? x = 9007154218744718 * 2^-53
%1 = 4503577109372359/4503599627370496
? x - 99999/10^5 < 1 - x
%2 = 1

so, as you see, Octave's result is mathematically correct. Such are the
subtleties of binary floating point values.

    _______________________________________________________

Reply to this item at:

  <http://savannah.gnu.org/bugs/?30817>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.gnu.org/