[Octave-bug-tracker] [bug #65495] nchoosek error "gcd: all values must be integers"

[Hendrik K]

Follow-up Comment #15, bug #65495 (group octave):

Correct means of course analytically correct. 

A numerical floating point solution is beyond flintmax not always analytically
correct but is an approximate solution. Specifically in our case beyond
flintmax one cannot assume, that i/(g1 * g2) is identical to one and therefore
one needs to cater for these cases in the programming logic.


The analytical correct solution of the gcd between 879236959347161415 and 9 is
actually 9. 
That 879236959347161415 cannot exactly be represented as a double and that its
"representation" is 879236959347161472 and that the gcd of its
"representation" and 9 is 3 is true, but somewhat besides the point made
below: one cannot assume that i/(g1 * g2) is equal to 1. I did not mean that
the gcd does provide a numerically incorrect result beyond flintmax...


@Side note:
I am not aware of any major programming language which treats numerical
literals/constants as doubles when they are used in the program to e.g. set
integer variable values.  In fact, I would consider such a behaviour a major
limitation/error. 
So I am not sure that one needs to document normal/expected behaviour.


 

    


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?65495>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/