[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Octave-bug-tracker] [bug #36133] num2str displays more than 16 signific
From: |
Rik |
Subject: |
[Octave-bug-tracker] [bug #36133] num2str displays more than 16 significant digits for large integer inputs |
Date: |
Wed, 25 Apr 2018 12:58:54 -0400 (EDT) |
User-agent: |
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:55.0) Gecko/20100101 Firefox/55.0 |
Follow-up Comment #8, bug #36133 (project octave):
Adding in jwe because the behavior is fascinating. From Hartmut's tests in
comment #3, the Matlab sprintf function returns
sprintf ('%30.0f', 1e25)
ans =
10000000000000001000000000
where the '1' in the sectond position is very real.
Octave's answer for the same code is
ans = 10000000000000000905969664
If I align the two
10000000000000001000000000 # Matlab
10000000000000000905969664 # Octave
it seems pretty clear what Matlab has done. First, Octave uses the C library
routine for sprintf so we get whatever default behavior is implemented when
the precision exceeds that of the underlying number. But Matlab has written a
shim layer for sprintf which calls the sprintf library, but then cleans up the
result. However, they haven't been too clever about it. IEEE-854 doubles
have a variable precision of 15-17 decimal digits. The extra '1' occurs
because they are rounding the digit '9' which is the 18 significant figure and
not valid.
In any case, once Matlab has printed 17 digits, it pads the rest of the string
with zeros. This is pretty easy to do in C++ with the substr() function or
operator[] access and a for loop.
So should Octave do something similar? And if so, where should it be done?
It could be done just in our sprintf routine, but then other actions like
printf would diverge in behavior. That means it would probably be better to
do it in octave::stream.
_______________________________________________________
Reply to this item at:
<http://savannah.gnu.org/bugs/?36133>
_______________________________________________
Message sent via/by Savannah
http://savannah.gnu.org/