[Octave-bug-tracker] [bug #65291] [octave forge] (mapping) degrees2dms produces incorrect results

[Philip Nienhuis]

Update of bug#65291 (group octave):

                  Status:         Patch Submitted => Confirmed              
             Assigned to:          philipnienhuis => None                   
        Operating System:       Microsoft Windows => Any                    

    _______________________________________________________

Follow-up Comment #8:

OK I'll set status back to just "confirmed".

These functions (degrees2dms, dms2degrees, dm2degrees, dms2degrees) haven't
been written by me.  While their code looks quite straightforward, corner
cases like losing seconds or minutes easily fall victim to double
(im)precision that isn't catched properly.
I'm also a bit surprised that this bug pops up only now.

As an aside:
Degrees-minutes-seconds display output like [1, 11, 60] looks a bit wonky as
it suggests that a minute can be subdivided in 61 ... yeah, what? 'intervals'?
rather than 60 seconds (0..60 rather than 0..59). OTOH display output like in
comment #2, where output precision is much higher, is accurate. It is the
display *rounding* that's hard to get right.
So I can understand e.g. Matlab in emitting such results - it's a lot easier
to not have to worry about carry-over when rounding to display precision. But
in most practical use (e.g., navigating at sea) it would be frowned upon and
most navigation SW I know either doesn't accept such input or morphs it into
"proper" display rounding.

>From comment #2:
> when rounded to nearest double is [1, 11,
59.9999999999998436805981327779591083526611328125]
Rounding to display precision (IIUC different than "double precision") should
give [1, 12, 0].  On the surface of the Earth the inaccuracy would be in the
order of 3^(-10) m:

>> (60 - 59.9999999999998436805981327779591083526611328125) * 1852
ans = 2.895035322580952e-10

(one minute ~ a nautical mile = 1852 m)

I might give fixing it another try later on, but I have little time to devote
to it, sorry. Hopefully someone beats me to it.



    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?65291>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/