Re: How to read integers from file faster?

[Daniel Llorens]

> Date: Sun, 01 Sep 2013 22:55:56 +0200
> From: "Pascal J. Bourguignon" <address@hidden>
> To: address@hidden
> 
> Darren Hoo <address@hidden> writes:
> 

>> It is way too slow to read numbers from a file simply by using `read' 
>> 
>> for example a txt file contains 10,000,000 line of numbers:
> 
> To get speed, I would:
> 
> 1- open a binary file,
> 
> 2- perform double-buffering I/O,  (ie. read a buffer while you process
>   another),
> 
> 3- assume the input format is as specified, ie. no check.
>   ?
>   (let ((byte (vector-ref buffer i)))
>     (if (= byte 10)
>        (begin (set! values (cons value values))
>               (set! value 0))
>        (set! value (+ (* value 10) (mod byte 16)))))
>   ?
> 
> 
> Actually, only using buffered I/O matters.  For big files, what slows
> down is I/O, not the handling of characters or checking of the syntax or
> anything that's done in memory.  If you want to read your data faster,
> keep it in RAM, or else in SSD.

I know this is not exactly a Scheme solution, but if you can choose your file 
format, consider using HDF5. You can read directly into a typed (srfi4) array, 
and a lot of languages have bindings (e.g. h5py), so it's a portable format. 
I've found it to be fast.

I haven't found bindings for Guile on the web (the ones I use depend on local 
stuff), but it shouldn't be a lot of work, and it's more flexible than a 
one-off integer reader. Of course, if that's all you need, then disregard this.

If you or someone else is interested in doing proper bindings using the ffi, I 
can lend a hand and test.

        Daniel