[sr #110636] asttable is very slow with many columns

[Mohammad Akhlaghi]

Follow-up Comment #1, sr #110636 (project gnuastro):

Thanks a lot for reporting this.

I was able to reproduce the problem with this manually created table of 10e5
columns:


echo 1 100000 | awk '{for(i=0;i<$2;++i) printf "%d ", $1+i; print ""}' >
many-columns.txt

asttable many-columns.txt -i


Increasing the column count to 1e6 did indeed take very long and forced me to
manually close it too.

The problem with plain-text tables is that the reading can't be easily
parallelized (with FITS tables, since we know the number a-priori, in
Gnuastro, we parallelize the column reading).

If you do come up with a good solution for plain-text tables in NEMO, please
share it here. 

In Gnuastro, we do have a format
<https://www.gnu.org/software/gnuastro/manual/html_node/Gnuastro-text-table-format.html>
to give metadata to plain-text tables. But that also won't necessarily help
unless all columns are string (their number of characters are known
a-priori).

Generally, besides theoretical checking of code, I am curious if there is any
real-world scenario that would require so many columns? Usually the workflow
can be re-designed (optimized) to avoid so many columns in one file. For
example FITS only accepts 999 columns.


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/support/?110636>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/