Re: [Pika-dev] Guile FFI: resizable vector problem

[Tom Lord]

    > From: address@hidden

    > [spin-locking idea for vector implementation]


Sure... variations on that idea are probably possible to do within the
representations layer.

At this point during development, my main focus is on getting the APIs
right --- implementing them optimally can come later.  So I'm mostly
interested in making sure that the API doesn't _preclude_ an efficient
operation.

The vector API has two parts: one already written, and one yet to be
written.

The already written part is just, in essense, VECTOR-REF, VECTOR-SET!,
and VECTOR-RESIZE! --- which are promised to behave in a consistently
serializable way.

The yet to be written part involves a locking protocol.  You can
"lock" one or two vectors, peform VECTOR-FAST-REF, VECTOR-FAST-SET!,
and VECTOR-FAST-RESIZE! operations on those, then release the lock.
There are sharp limits on what other operations you an do while
holding the lock.  You're expected to hold the lock(s) only for a very
brief time.

Serializability applies to the locking API as well:  the code executed
while holding a lock must be consistently serializabile with all other
lock-holding code, and with VECTOR-REF, VECTOR-SET!, and
VECTOR-RESIZE!.

As an example of how I expect these to be used:

~ a function that fills a vector with information returned from
  the stat(2) system call should probably just use the slow
  but easy interface --- there's no reason to optimize the heck
  out of a few vector sets in a function whose run-time is going to be
  dominated by the system call.

~ a function implementing VECTOR-MOVE!, to copy a subvector to a 
  subvector, should use the locking interface.  (I don't even consider
  it critical that the initial implementation of VECTOR-MOVE! use the
  locking interface --- it can use the easy one to help us bootstrap
  quickly and we can optimize it later.)

As far as I know, the existing and planned API is consistent with
implementation techniques such as the spin-lock idea you've proposed.
The vector operations are provided (in essense, not quite literally)
by the "representations layer" -- how they provide the serializability
guarantees is entirely an internal matter for that layer.  That layer
has free reign to determine every detail of the representation of
vectors.

The current implementation of the representations layer is being
written mostly with bootstrapping in mind -- not for optimization and
not to be featureful.   In particular, it does _not_ provide support
for concurrent threads.   (However, we're trying to be careful to make
sure that thread support can be added by changing _only_ the
representations layer.)

So, the spin-lock idea seems like a good one.  It goes on my list of
things to think about when it's time to revise the representations
layer.  That time _probably_ isn't just yet, though -- at least for
me.

On the other hand -- if it's something you want to work on -- well, go
for it.  That's not premature.  For example, someone else has been
working on writing a new representations layer based on code from
Guile.

-t