[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Import large field-delimited file with strings and numbers
From: |
Philip Nienhuis |
Subject: |
Re: Import large field-delimited file with strings and numbers |
Date: |
Mon, 8 Sep 2014 09:49:36 -0700 (PDT) |
Joao Rodrigues wrote
>>> I need to import a large CSV file with multiple columns with mixed
>>> string and number entries, such as:
>>>
>>> field1, field2, field3, field4
>>> A, a, 1, 1.0,
>>> B, b, 2, 2.0,
>>> C, c, 3, 3.0,
>>>
>>> and I want to pass this on to something like
>>>
>>> cell1 ={[1,1] = A; [2,1] = B; [3,1] = C};
>>> cell2 ={[1,1] = a; [2,1] = b; [3,1] = c};
>>> arr3 =[1 2 3]';
>>> arr4 =[1.0 2.0 3.0]';
>>>
>>> furthermore, some columns can be ignored, the total number of entries is
>>> known and there is a header.
> <snip>
> Yet, csv2cell is orders of magnitude faster. I will break the big file
> into chunks (using fileread, strfind to determine newlines and fprintf)
> and then apply csv2cell chunk-wise.
Why do you need to break it up using csv2cell? AFAICS that reads the entire
file and directly translates the data into "values" in the output cell
array, using very little temporary storage (the latter quite unlike
textscan/strread).
It does read the entire file twice, once to assess the required dimensions
for the cell array, the second (more intensive) pass for actually reading
the data.
BTW, on my TODO list is an old (2+ years?) entry: adding a "headerlines"
parameter to csv2cell..... (but OK my TODO list is looong)
Philip
--
View this message in context:
http://octave.1599824.n4.nabble.com/Import-large-field-delimited-file-with-strings-and-numbers-tp4666380p4666420.html
Sent from the Octave - General mailing list archive at Nabble.com.
- Re: Import large field-delimited file with strings and numbers, (continued)
- Re: Import large field-delimited file with strings and numbers, Ben Abbott, 2014/09/07
- Re: Import large field-delimited file with strings and numbers, Philip Nienhuis, 2014/09/08
- how does fscanf with "C" work?, Francesco Potortì, 2014/09/08
- Re: how does fscanf with "C" work?, Thomas D. Dean, 2014/09/08
- Re: how does fscanf with "C" work?, Francesco Potortì, 2014/09/09
- Re: Import large field-delimited file with strings and numbers, Petri Piila, 2014/09/07
- Re: Import large field-delimited file with strings and numbers, Francesco Potortì, 2014/09/07
Re: Import large field-delimited file with strings and numbers, Ben Abbott, 2014/09/06
Re: Import large field-delimited file with strings and numbers, Philip Nienhuis, 2014/09/06
- Re: Import large field-delimited file with strings and numbers, João Rodrigues, 2014/09/06
- Re: Import large field-delimited file with strings and numbers,
Philip Nienhuis <=
- Re: Import large field-delimited file with strings and numbers, João Rodrigues, 2014/09/08
- Re: Import large field-delimited file with strings and numbers, Markus Bergholz, 2014/09/08
- Re: Import large field-delimited file with strings and numbers, Markus Bergholz, 2014/09/08
- Re: Import large field-delimited file with strings and numbers, Joao Rodrigues, 2014/09/08
- Re: Import large field-delimited file with strings and numbers, Markus Bergholz, 2014/09/08
- Re: Import large field-delimited file with strings and numbers, Markus Bergholz, 2014/09/08
Re: Import large field-delimited file with strings and numbers, Philip Nienhuis, 2014/09/11
Re: Import large field-delimited file with strings and numbers, Philip Nienhuis, 2014/09/10
Re: Import large field-delimited file with strings and numbers, CdeMills, 2014/09/08