[task #16555] Very high RAM usage of Warp in WCS warpings

[Mohammad Akhlaghi]

URL:
  <https://savannah.gnu.org/task/?16555>

                 Summary: Very high RAM usage of Warp in WCS warpings
                   Group: GNU Astronomy Utilities
               Submitter: makhlaghi
               Submitted: Wed 19 Jun 2024 10:33:25 PM CEST
         Should Start On: Wed 19 Jun 2024 12:00:00 AM CEST
   Should be Finished on: Wed 19 Jun 2024 12:00:00 AM CEST
                Category: Warp
                Priority: 5 - Normal
              Item Group: Enhancement
                  Status: Postponed
                 Privacy: Public
             Assigned to: glorenzetti
        Percent Complete: 0%
             Open/Closed: Open
         Discussion Lock: Any
                  Effort: 0.00


    _______________________________________________________

Follow-up Comments:


-------------------------------------------------------
Date: Wed 19 Jun 2024 10:33:25 PM CEST By: Mohammad Akhlaghi <makhlaghi>
When the Warp program does
[https://www.gnu.org/software/gnuastro/manual/html_node/Align-pixels-with-WCS-considering-distortions.html
WCS operations] (which is the default). It consumes a very large amount of
RAM! Such that in large pipelines, the warping step becomes the main
bottleneck (in RAM). 

Here is an example (the second Warp command) 


$ echo "1 101 101 1 20 0.5 30 0.7 15 5" \
       | astmkprof --mergedsize=201,201 --oversample=1 -oraw.fits \
                   --pc=-0.939692,-0.3420201,-0.3420201,0.9396926
                       
$ /usr/bin/time --format=%M astwarp raw.fits
23680

$ ls -lh raw.fits 
-rw-r--r-- 1 mohammad mohammad 169K Jun 19 22:16 raw.fits


We see that while the file is only 169K, at its maximum, Warp is consuming
23680K of memory!!!! In a normal reduction pipeline these days images are
about 300-400 Megaybtes, so this excessive usage of RAM by Warp easily fills
up any RAM and creates a strong bottleneck for parallel operations!

This excessive usage occurs because of the two phase operation of Warp in WCS
mode: first it initializes the output image's pixel grid, and then it will
warp the input pixels into it. But since the operation is non-linear, we need
the coordinates of all four corners of every pixel. 

My first guess is that if we can re-design the program to get the output pixel
grid with minimal points (for example only warp the outer corners of the the
pixels on the border of the image), and then do the warp of each input pixel
independently in its own thread, this excessive RAM usage will be removed. But
we should look into it closer! 

I should greatly thank Pedram for his wonderful work on completing the
WCS-based feature of Warp! We knew about this problem at that time, but in the
spirit of progress, needed to finish the work and couldn't invest more in the
RAM optimization at that time.







    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/task/?16555>

_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/