Re: trying to understand why I get 2.2204e-16 rather than 0

[Maynard Wright]

On Wednesday, August 03, 2016 08:59:15 PM Dmitri A. Sergatskov wrote:
> On Wed, Aug 3, 2016 at 7:41 PM, naryana.shankara <address@hidden
> > wrote:
> > 
> > If I enter cot (pi/4)-1 I get 2.2204e-16
> > From my trig days I remember cot (pi/4) is 1 and 1-1 is 0
> > Octave also confirms cot (pi/4) is 1 and 1-1 is 0
> 
> octave:6> format hex
> octave:7> 1
> ans = 3ff0000000000000
> octave:8> cot(pi/4)
> ​​
> ans = 3ff0000000000001
> 
> > so why does Octave spit out 2.2204e-16 when cot(pi/4)-1 is entered?
> 
> https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
> 
> 
> ​Dmitri.


Although cot(pi/4)-1 returns 0, tan(pi/4)-1 returns -1.11022e-16.   One might 
therefore suspect that cot() is not obtained by taking the inverse of tan(), 
but 1 / (tan(pi/4))-1 returns 0, so the macro cot(x) = 1 / tan(x) may be used, 
although I haven't checked.   

To see exactly why these anomalies occur, one would have to look into the 
floating point representations of the numbers involved and the procedure used 
to compute tan().   The references given by others in this thread should be 
helpful in pointing out the limitations of representing numbers in floating 
point format.


Maynard Wright