[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Need a faster way of reading files.
From: |
Bård Skaflestad |
Subject: |
Re: Need a faster way of reading files. |
Date: |
Wed, 02 Jun 2010 18:24:43 +0200 |
On Wed, 2010-06-02 at 17:23 +0200, Ole Jacob Hagen wrote:
> Hi,
>
> I have large data files, with 90 columns. Typical size is 600-700 MB.
> Data is stored in ascii format. This is data collected for 7200 s,
> with sampling interval 100 Hz.
>
> I am using data = load('mydata.txt'), but this is very very slow,
> compared to SciPy. SciPy uses one minute, while Octave is using approx
> 5-10 minutes. Haven't measured time consumption, but can come back
> with that one. ;-)
>
> I've tried to use fread, but the data is not recognizable at all. I
> know how many columns I have in my data file.
>
> How can I accelerate my data loading into memory in Octave?
Since you're using LOAD, I'll assume that the values are floating point
numbers. This is a shot in the dark, but have you tried FSCANF? Note
that FREAD is (mainly?) for binary data. Would you mind trying the
following statements
fid = fopen('mydata.txt', 'rt')
tic; table = fscanf(fid, '%f', [90, inf]).'; toc
and reporting back on the tic/toc pair (as well as if the reading was
successful or not)?
Best regards,
--
Bård Skaflestad <address@hidden>
SINTEF ICT, Applied mathematics