Re: Import large field-delimited file with strings and numbers

[CdeMills]

Joao Rodrigues wrote
> I need to import a large CSV file with multiple columns with mixed 
> string and number entries, such as:
> 
> field1, field2, field3, field4
> A,        a,        1,       1.0,
> B,        b,        2,        2.0,
> C,        c,        3,        3.0,
> 
> and I want to pass this on to something like
> 
> cell1 ={[1,1] = A; [2,1] = B; [3,1] = C};
> cell2 ={[1,1] = a; [2,1] = b; [3,1] = c};
> arr3 =[1 2 3]';
> arr4 =[1.0 2.0 3.0]';
> 
> furthermore, some columns can be ignored, the total number of entries is 
> known and there is a header.
> 
> How can I perform the import within reasonable time and little memory 
> overhead? Below are a few of my attempts.
> 
> Octave offers a wide range of functions to import files (csvread, 
> dlmread, textscan, textread, fscanf, fgetline) but as far as I can tell 
> none seems to get the job done.
> 
> csvread and dlmread don't work because they only handle numerical data.
> 
> textscan works eats up all the memory (the file is 200 MB, textscan's 
> memory usage was into the GB's). It doesn't allow to provide a priori 
> the size of the object.

Hello Joaõ,
this is the perfect job for the "dataframe" package. It reads a CSV file and
returns an object which try to mimics a matrix. You can select ranges,
perform arithmetic operations, ...

You'll find in on the octave-forge source.

Regards

Pascal




--
View this message in context: 
http://octave.1599824.n4.nabble.com/Import-large-field-delimited-file-with-strings-and-numbers-tp4666380p4666422.html
Sent from the Octave - General mailing list archive at Nabble.com.