Re: [gpsd-dev] ✘CEP(50) 0.285 metersMime-Version: 1.0

[Fred Wright]

On Fri, 22 Jul 2016, Gary E. Miller wrote:
> On Fri, 22 Jul 2016 13:55:52 -0700 (PDT)
> Fred Wright <address@hidden> wrote:
>
> > On Fri, 22 Jul 2016, Gary E. Miller wrote:
> >
> > > My main attack for a while will we be in geting the numeric
> > > precision solid.  Calculating things to about 11 significant
> > > decimal digits is harder than it looks.  A 64 bit long is only
> > > about 10 decimal digits.
> >
> > Umm, a *64-bit* long is about 19 decimal digits.  Though a double is
> > only good for about 15.
>
> And I can't see how to force Python to use a 64-bit long.  Got a clue
> for me?

It shouldn't be a problem in Python.  Python has the concept of a "long
int" (sometimes just called "long"), which uses software multiple
precision.  It should automatically promote as needed, and it isn't even
limited to 64 bits.  E.g. (on the BBB):

Python 2.7.3 (default, Jun 21 2016, 21:00:47)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> x = sys.maxint
>>> x
2147483647
>>> type(x)
<type 'int'>
>>> y = x * x
>>> y
4611686014132420609L
>>> type(y)
<type 'long'>
>>> z = y * y
>>> z
21267647892944572736998860269687930881L
>>> type(z)
<type 'long'>

Essentially, Python has built-in "bignum" support.

Of course it's best to stick to normal ints where possible, for
efficiency.  Sometimes it's useful to wrap an int() around something to
get rid of gratuitous longness.

> 15 from a double is probably close enough, but I would go long double
> if I could figure out to get Python to do that.

Long doubles tend to be pretty iffy in general, in many cases happily
falling back to normal doubles.

Fred Wright