[gnuastro-commits] master 62ce84b 3/7: Book: extended introduction added to bitwise operators

[Mohammad Akhlaghi]

branch: master
commit 62ce84b54429bf793e41dd283b96e8c2254c2248
Author: Mohammad Akhlaghi <mohammad@akhlaghi.org>
Commit: Mohammad Akhlaghi <mohammad@akhlaghi.org>

    Book: extended introduction added to bitwise operators
    
    Until now, there was no introductory explanation on the conditional,
    bitwise, and building new dataset operators. While the first and third were
    pretty easy to understand for a random astronomer, the second (bitwise
    operators) could be a little hard to understand, so it was necessary to
    give a good introduction in astronomical data analysis scenarios.
    
    With this commit, an introduction has been added for all three sections and
    the bitwise operators now have an extended discussion on why they are
    useful for a modern astronomer.
---
 doc/gnuastro.texi | 64 ++++++++++++++++++++++++++++++++++++++++++++++---------
 1 file changed, 54 insertions(+), 10 deletions(-)

diff --git a/doc/gnuastro.texi b/doc/gnuastro.texi
index 8b51bc4..0686743 100644
--- a/doc/gnuastro.texi
+++ b/doc/gnuastro.texi
@@ -12744,11 +12744,12 @@ This is a very powerful notation and is used in 
languages like Postscript @footn
 @node Arithmetic operators, Invoking astarithmetic, Reverse polish notation, 
Arithmetic
 @subsection Arithmetic operators
 
-The recognized operators in Arithmetic are listed below.
-See @ref{Reverse polish notation} for more on how the operators and operands 
should be ordered on the command-line.
-The operands to all operators can be a data array (for example a FITS image) 
or a number, the output will be an array or number according to the inputs.
+In this section, list of recognized operators in Arithmetic (and the Table 
program's @ref{Column arithmetic}) and discussed in detail with examples.
+As mentioned before, to be able to easily do complex operations on the 
command-line, the Reverse Polish Notation is used (where you write 
`@mymath{4\quad5\quad+}' instead of `@mymath{4 + 5}'), if you aren't already 
familiar with it, before continuing, please see @ref{Reverse polish notation}.
+
+The operands to all operators can be a data array (for example a FITS image or 
data cube) or a number, the output will be an array or number according to the 
inputs.
 For example a number multiplied by an array will produce an array.
-The conditional operators will return pixel, or numerical values of 0 (false) 
or 1 (true) and stored in an @code{unsigned char} data type (see @ref{Numeric 
data types}).
+The numerical data type of the output of each operator is described within it.
 
 @cartouche
 @noindent
@@ -13209,6 +13210,9 @@ The output will have the same type as the inputs.
 This is natural for the @command{min} and @command{max} operators, but for 
other similar operators (for example @command{sum}, or @command{average}) the 
per-pixel operations will be done in double precision floating point and then 
stored back in the input type.
 Therefore, if the input was an integer, C's internal type conversion will be 
used.
 
+@item
+The operation will be multi-threaded, grealy speeding up the process if you 
have large and numerous data to stack.
+You can disable multi-threaded operations with the @option{--numthreads=1} 
option (see @ref{Multi-threaded operations}).
 @end itemize
 
 @item max
@@ -13225,13 +13229,15 @@ $ astarithmetic a.fits b.fits c.fits 3 min -omax.fits
 For each pixel count the number of non-blank pixels in all given datasets.
 The output will be an unsigned 32-bit integer datatype (see @ref{Numeric data 
types}).
 This operator is called similar to the @command{min} operator, please see 
there for more.
-Note that some datasets may have blank values (which are also ignored in all 
similar operators like @command{min}, @command{sum}, @command{mean} or 
@command{median}).
-Hence final pixel values of this operator will not, in general, be equal to 
the number of inputs.
 For example
 @example
 $ astarithmetic a.fits b.fits c.fits 3 number -onum.fits
 @end example
 
+Some datasets may have blank values (which are also ignored in all similar 
operators like @command{min}, @command{sum}, @command{mean} or 
@command{median}).
+Hence, the final pixel values of this operator will not, in general, be equal 
to the number of inputs.
+This operator is therefore mostly called in parallel with those operators to 
know the ``weight'' of each pixel (in case you want to only keep pixels that 
had the full exposure for example).
+
 @item sum
 For each pixel, calculate the sum in all given datasets.
 The output will have the a single-precision (32-bit) floating point type.
@@ -13520,13 +13526,19 @@ $ astarithmetic img1.fits img2.fits img3.fits 3 
add-dimension
 @node Conditional operators, Mathematical morphology operators, Dimensionality 
changing operators, Arithmetic operators
 @subsubsection Conditional operators
 
+Conditional operators take two inputs and return a binary output that can only 
have two values 0 (for pixels where the condition was false) or 1 (for the 
pixels where the condition was true).
+Because of the binary (2-valued) nature of their outputs, the output is 
therefore stored in an @code{unsigned char} data type (see @ref{Numeric data 
types}) to speed up process and take less space in your storage.
+There are two exceptions to the general features above: @code{isblank} only 
takes one input, and @code{where} takes three, while not returning a binary 
output, see their description for more.
+
 @table @command
 @item lt
 Less than: creates a binary output (values either 0 or 1) where each pixel 
will be 1 if the second popped operand is smaller than the first popped operand 
and 0 otherwise.
 If both operands are images, then all the pixels will be compared with their 
counterparts in the other image.
-For example:
+
+For example, the pixels in the output of the command below will have a value 
of 1 (true) if their value in @file{image1.fits} is less than their value in 
@file{image2.fits}.
+Otherwise, their value will be 0 (false).
 @example
-$ astarithmetic image1.fits image2.fits -g1 lt
+$ astarithmetic image1.fits image2.fits lt
 @end example
 If only one operand is an image, then all the pixels will be compared with the 
single value (number) of the other operand.
 For example:
@@ -13756,6 +13768,38 @@ $ astarithmetic image.fits invert
 @node Bitwise operators, Numerical type conversion operators, Mathematical 
morphology operators, Arithmetic operators
 @subsubsection Bitwise operators
 
+@cindex Bitwise operators
+Astronomical images are usually stored as an array multi-byte pixels with 
different sizes for different precision levels (see @ref{Numeric data types}).
+For example images from CCDs are usually in the unsigned 16-bit integer type 
(each pixel takes 16 bits, or 2 bytes, of memory) and fully reduced deep images 
have a 32-bit floating point type (each pixel takes 32 bits or 4 bytes).
+
+On the other hand, during the data reduction, we need to preserve a lot of 
meta-data about some pixels.
+For example, if a cosmic ray had hit the pixel during the exposure, or if the 
pixel was saturated, or is known to have a problem, or if the optical 
vignetting is too strong on it, and etc.
+A crude solution is to make a new image when checking for each one of these 
things and make a binary image where we flag (set to 1) pixels that satisfy any 
of these conditions above, and set the rest to zero.
+However, processing pipelines sometimes need more than 20 flags to store 
important per-pixel meta-data, and recall that the smallest numeric data type 
is one byte (or 8 bits, that can store up to 256 different values), while we 
only need two values for each flag!
+This is a major waste of storage space!
+
+@cindex Flag (mask) images
+@cindex Mask (flag) images
+A much more optimal solution is to use the bits within each pixel to store 
different flags!
+In other words, if you have an 8-bit pixel, use each bit as as a flag to mark 
if a certain condition has happened on a certain pixel or not.
+For example, let's set the following standard based on the four cases 
mentioned above: the first bit will show that a cosmic ray has hit that pixel.
+So if a pixel is only affected by cosmic rays, it will have this sequence of 
bits (note that the bit-counting starts from the right): @code{00000001}.
+The second bit shows that the pixel was saturated (@code{00000010}), the third 
bit shows that it has known problems (@code{00000100}) and the fourth bit shows 
that it was affected by vignetting (@code{00001000}).
+
+Since each bit is independent, we can thus mark multiple metadata about that 
pixel in the actual image, within a single ``flag'' or ``mask'' pixel of a flag 
or mask image that has the same number of pixels.
+For example a flag-pixel with the following bits @code{00001001} shows that it 
has been affected by cosmic rays @emph{and} it has been affected by vignetting 
at the same time.
+The common data type to store these flagging pixels are unsigned integer types 
(see @ref{Numeric data types}).
+Therefore when you open an unsigned 8-bit flag image in a viewer like DS9, you 
will see a single integer in each pixel that actually has 8 layers of metadata 
in it!
+For example the integer you will see for the bit sequences given above will 
respectively be: @mymath{2^0=1} (for a pixel that only has cosmic ray), 
@mymath{2^1=2} (for a pixel that was only saturated), @mymath{2^2=4} (for a 
pixel that only has known problems), @mymath{2^3=8} (for a pixel that is only 
affected by vignetting) and @mymath{2^0 + 2^3 = 9} (for a pixel that has a 
cosmic ray @emph{and} was affected by vignetting).
+
+You can later use this bit information to mark objects in your final analysis 
or to mask certain pixels.
+For example you may want to set all pixels affected by vignetting to NaN, but 
can interpolate over cosmic rays.
+You therefore need ways to separate the pixels with a desired flag(s) from the 
rest.
+It is possible to treat a flag pixel as a single integer (and try to define 
certain ranges in value to select certain flags).
+But a much more easier and robust way is to actually look at each pixel as a 
sequence of bits (not as a single integer!) and use the bitwise operators below 
for this job.
+For more on the theory behind bitwise operators, see 
@url{https://en.wikipedia.org/wiki/Bitwise_operation, Wikipedia}.
+
+
 @table @command
 @item bitand
 Bitwise AND operator: only bits with values of 1 in both popped operands will 
get the value of 1, the rest will be set to 0.
@@ -13976,7 +14020,7 @@ $ asttable catalog.fits \
 @node Building new dataset, Operand storage in memory or a file, Elliptical 
shape operators, Arithmetic operators
 @subsubsection Building new dataset
 
-
+With the operator here, you can create a new dataset from scratch to start 
certain operations without any input data.
 
 @table @command
 @item makenew
@@ -14000,7 +14044,7 @@ $ astarithmetic 100 200 2 makenew 5 mknoise-sigma
 @node Operand storage in memory or a file,  , Building new dataset, Arithmetic 
operators
 @subsubsection Operand storage in memory or a file
 
-In your early days of using Gnuastro, to do multiple operations, it is likely 
that you will simply call Arithmetic (or Table, with column crithmetic) 
multiple times.
+In your early days of using Gnuastro, to do multiple operations, it is likely 
that you will simply call Arithmetic (or Table, with column arithmetic) 
multiple times: feed the output file of the first call to the second call.
 But as you get more proficient in the reverse polish notation, you will find 
yourself combining many operations into one call.
 This greatly speeds up your operation, because instead of writing the dataset 
to a file in one command, and reading it in the next command, it will just keep 
the intermediate dataset in memory!