[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
loading big gzip compressed and uncompressed numerical matrices
From: |
Andreas Weber |
Subject: |
loading big gzip compressed and uncompressed numerical matrices |
Date: |
Sat, 21 Dec 2019 15:20:37 +0100 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.3.0 |
Dear all,
I've implemented a CSV parser as OCT file [1] to load gzip compressed
numerical matrices.
The reason for this was that I'm running GNU Octave on embedded systems
and I had to deal with increasing, gzip compressed files (see my
question 3 weeks ago [2]).
The result is quite surprising (at least for me): It's approx. 12 times
faster than the builtin "load" even for non-incremental reads.
I hope it can be useful for others who deal with large numerical CSVs.
-- Andy
[1] https://github.com/Andy1978/load_gz/tree/master
[2]
https://octave.1599824.n4.nabble.com/incremental-read-of-gzipped-matrix-td4694821.html
[Prev in Thread] |
Current Thread |
[Next in Thread] |
- loading big gzip compressed and uncompressed numerical matrices,
Andreas Weber <=