Re: Handling large files with emacs lisp?

[Jambunathan K]

May be you can steal some stuff from here.

        http://elpa.gnu.org/packages/vlf.html

It is a GNU ELPA package that you can install with M-x list-packages
RET.



Klaus-Dieter Bauer <bauer.klaus.dieter@gmail.com> writes:

> Hello!
>
> Is there a method in emacs lisp to handle large files (hundreds of MB)
> efficiently? I am looking specifically for a function that allows
> processing file contents either sequentially or (better) with random
> access.
>
> Looking through the code of `find-file' I found that
> `insert-file-contents' and `insert-file-contents-literally' seem to be
> pretty much the most low-level functions available to emacs-lisp. When
> files go towards GB size however, inserting file contents is
> undesirable even assuming 32bit emacs were able to handle such large
> buffers.
>
> Using the BEG and END parameters of `insert-file-contents' however has
> a linear time-dependence on BEG. So implementing buffered file
> processing for large files by keeping only parts of the file in a
> temporary buffer doesn't seem feasible either.
>
> I'd also be interested why there is this linear time dependence. Is
> this a limitation of how fseek works or of how `insert-file-contents'
> is implemented? I've read[1] that fseek "just updates pointers", so
> random reads in a large file, especially on an SSD, should be
> constant-time, but I couldn't find further verification.
>
> kind regards, Klaus
>
> PS: I'm well aware that I'm asking for something, that likely wasn't
>     within the design goals of emacs lisp. It is interesting to push
>     the limits though ;)
>
> ------------------------------------------------------------
>
> [1] https://groups.google.com/d/msg/comp.unix.aix/AXInTbcjsKo/qt-XnL12upgJ