[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Possibly wrong statistics calculation
From: |
Mohammad Akhlaghi |
Subject: |
Re: Possibly wrong statistics calculation |
Date: |
Tue, 12 May 2020 22:02:29 +0100 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.8.0 |
Hi,
Thank you very much for reporting this.
This is an interesting situation. From the command below, we see that
the image has 223112 pixels.
astfits 019_00.fits -h0 | awk '$1=="NAXIS1"{x=$3} $1=="NAXIS2"{y=$3}
END{print x*y}'
But when I look at the "Number of elements" reported by the
aststatistics program is one less: 223111
Then I noticed that in your calculation, the maximum is 65535 (the
maximum value for unsigned 16-bit integers) while aststatistics says it
is 65184.
This explains the problem: in Gnuastro, for unsigned types, the maximum
value is defined as a blank value. In a single exposure this is natural:
this value can only occur when the pixel is saturated and in such cases,
indeed, it shouldn't be included in the statistics. This is explained in
this part of the Gnuastro book:
https://www.gnu.org/software/gnuastro/manual/html_node/Blank-pixels.html
But if in your scenario, that single saturated pixel is meaningful,
indeed it is a bug.
So I just defined bug #58360 for this [1]. Currently I am too busy to
implement it immediately, but hopefully we'll implement a fix soon. I
have added you in the "Carbon copy" of that bug so you are informed when
it is fixed ;-).
If there is any other thoughts on that bug, it would be great if we
discuss it on Savannah so it is recorded under there. You can just
register on Savannah and post bugs there.
Thanks again for reporting this,
Cheers,
Mohammad
[1] https://savannah.gnu.org/bugs/?58360
On 5/12/20 9:27 PM, rmorales wrote:
Summary:
Possibly wrong statistics calculation
gnuastro version:
0.11
test image set:
https://cloud.iaa.csic.es/public.php?service=files&t=9344cdb85c640c7dd74446d8c266591d
image pixel sequence as csv file:
https://cloud.iaa.csic.es/public.php?service=files&t=56b7ade952ede68f496b313c52887aa5
command:
aststatistics 019_00.fits --hdu=0
result :
Number of elements: 223111
Minimum: 0
Maximum: 65184
Median: 1295
Mean: 1340.290403
Standard deviation: 524.5495077
question:
Using my own statistics some values are different: average, stdDev and max:
average = 1340.57812668077
variance = 486126.1640894976
stdDev = 697.2274837450813
mode = 0
median = 1295
min = 0
max = 65535
Writing all pixels in to a calc sheet (see download link at beginning of
this message) :
Mean 1340.57812668077
Mode 0
Median 1295
Variance 486126.164089498
Standard Deviation 697.227483745081
Minimum 0
Maximum 65535
suggestion:
Review how the pixel values are obtained from FITS file applying the
linear transformation with BSCALE and BZERO keywords.