[Octave-bug-tracker] [bug #65712] sort over InF dimension coredumps

[Arun Giridhar]

Follow-up Comment #45, bug #65712 (group octave):

This is baffling behavior.

Stable is using nint_value, and this happens when I add cout statements:


octave:2> sort ([1 2; 3 4], inf)
Fsort: dim = 2147483647
ans =
   1   2
   3   4

octave:3> sort ([1 2; 3 4], -inf)
Fsort: dim = 2147483647
ans =
   1   2
   3   4


Apparently nint_value treats both +inf and -inf as intmax(), so they are both
treated as +inf.

But on the default branch with its strict_int_value, this happens:

octave:2> sort ([1 2; 3 4], inf)
Fsort: dim = 2147483646
ans =
   1   2
   3   4


octave:3> sort ([1 2; 3 4], -inf)
Fsort: dim = 2147483647
error: sort: DIM must be a positive scalar integer


So how is it that *both* cases convert -inf to either 2^31-1 or 2^31-2, but
one works and the other fails?


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?65712>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/