[Octave-bug-tracker] [bug #63848] Mean: some single class inputs can hit precision limit with sum

[Markus Mützel]

Follow-up Comment #28, bug #63848 (group octave):

The expected value for that average is not 35184372088832 like you wrote.

`flintmax("double")` is 2^53 (or 9007199254740992). Take that to Wolfram Alfa
(or another "calculator" of your choice that has the required precision) and
you'll see that the expected result is actually what the test expects
currently:
https://www.wolframalpha.com/input?i=%289007199254740992+%2B+2%5E8+-1%29%2F2%5E8



    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?63848>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/