Re: [External] : Passing buffers to function in elisp

[Petteri Hintsanen]

<tomas@tuxteam.de> writes:

> On Tue, Feb 21, 2023 at 11:21:47PM +0000, Drew Adams wrote:
>> What is it that you're really trying to do?
>
> That's exactly the point, yes.

Specifics, as usual, are somewhat messy.  But I try to summarize below.

I'm working with Ogg audio files, specifically Vorbis and Opus.  I need
to extract certain metadata from such files.  This code is part of EMMS
(see https://git.savannah.gnu.org/cgit/emms.git/tree/emms-info-native.el
if you're curious, but please note that it's not the same version I'm
working on.)

Ogg file is basically a sequence of logical "pages", and each page has
zero or more logical "packets".  I need to read and decode the first two
packets and the last page from a given file.  Page size is bounded by
65307 bytes, while packets can be of any size (they can span multiple
pages).

My code extracts the first two packets by repeatedly reading and
decoding a single page along its packet data ("payload"), until I have
assembled two complete packets.  These packets contain most of the
metadata I'm interested in.

Each page is read and decoded in somewhat wasteful manner by reading
65307 bytes worth of data from a certain offset (a page boundary) into a
temporary buffer.  So "func1" in my original posting is actually this:

  (defun emms-info-native--read-and-decode-ogg-page (filename offset)
    (with-temp-buffer
      (set-buffer-multibyte nil)
      (insert-file-contents-literally filename
                                      nil
                                      offset
                                      (+ offset
                                         emms-info-native--ogg-page-size))
      (emms-info-native--decode-ogg-page (buffer-string))))

The function emms-info-native--decode-ogg-page uses bindat to do the
actual decoding, and packs the results into a plist, which is then
returned to the caller.  I'm using separate function here because it is
easy to test -- just supply fixed byte vectors for it and check that you
get correct results.

Calling code looks like this:

  (defun emms-info-native--decode-ogg-packets (filename packets)
    (let ((num-packets 0)
          (offset 0)
          (stream (vector)))
      (while (< num-packets packets)
        (let ((page (emms-info-native--read-and-decode-ogg-page filename
                                                                offset)))
          (cl-incf num-packets (or (plist-get page :num-packets) 0))
          (cl-incf offset (plist-get page :num-bytes))
          (setq stream (vconcat stream (plist-get page :stream)))
      stream))

This function calls emms-info-native--read-and-decode-ogg-page in a loop
until the desired number of packets has been extracted.  So by
evaluating (emms-info-native--decode-ogg-packets filename 2) I get what
I need.

All data is read-only in the sense it is read from the disk and then
just copied around to alists, plists, vectors and so on.

-----

I added a counter for tracking the number of temp buffers and ran a
benchmark against some 3000+ Ogg files.  This was done on primed cache
so disk I/O should have had minimal effect.

There were 12538 temp buffers created (= 12538 pages decoded).
Benchmark function output was

  "Elapsed time: 23.806966s (18.743661s in 373 GCs)"

So this means that ~78% of the time was spent on garbage collection?  If
so, I think my design sucks.

-----

I am well aware that (preliminary) optimization is best avoided.
Also, "when in doubt, use brute force."
And even the current performance is good enough.

My problem here is of more fundamental sort: I don't know what are the
right data structures and calling conventions.  I am still learning
(emacs) lisp, and it shows.

In C or C++ it is "easier": just pass pointers or references and you're
good.  With Lisp and especially Emacs Lisp things are more convoluted --
at least until you learn the necessary idioms.

Thanks,
Petteri