Re: sqlite3

[Karl Fogel]

On 06 Dec 2021, Qiantan Hong wrote:
> As for key value store, if you really want a persistent
> kv store with O(1) insert, couldn’t we just use something like 
> the following super simple (skeleton) code?
>
> (defvar kv-store-table)
> (cl-defstruct (kv-store (:constructor make-kv-store-1)) path 
> table)
> (defun make-kv-store (path)
>   (let* ((kv-store (make-kv-store-1 :path path))
>          (kv-store-table (make-hash-table :test 'equal)))
>     (when (file-exists-p path)
>       (load-file path)
>       ;; and some error handling to ignore clear trailing 
>       un-balanced form
>       ;; just in case Emacs crash in the middle of a kv-put
>       )
>     (setf (kv-store-table kv-store) kv-store-table)
>     kv-store))
> (defun compact-kv-store (kv-store)
>   ;; dump the full content of kv-store-table at once
>   ;; to compress the log and speed up loading
>   )
> (defun kv-put (key value kv-store)
>   (with-temp-buffer
>     (print `(puthash ',key ',value kv-store-table) 
>     (current-buffer))
>     (append-to-file nil nil (kv-store-path kv-store)))
>   (puthash key value (kv-store-table kv-store)))
> (defun kv-get (key kv-store)
>   (gethash key (kv-store-table kv-store)))

As a data point:

I wrote a similar generic data-persistence mechanism a while ago, 
because I had exactly the kind of need Lars described in his OP. 
The code has just been living in my .emacs [1].  I never tried to 
package it up as an independent thing, and I'm sure that with more 
& wiser heads we'd easily come up with a better system.  I suspect 
that I'm not the only person who implemented a personal 
persistence mechanism :-).

Just having _one_ persistence mechanism (as opposed to everyone's 
personal improvisation) would be an improvement.  I liked Lars's 
sqlite3 proposal, but it doesn't have to be sqlite3 and I don't 
have a strong enough opinion to argue hard for that versus some 
more Lisp-y solution.

However, whatever the system is, it shouldn't require loading the 
entire data store into memory.  That's a key test that sqlite3 
passes; if we do something else instead, it should also pass that 
test.

Best regards,
-Karl

[1] https://svn.red-bean.com/repos/kfogel/trunk/.emacs, search for
   "A generic cross-session data persistence mechanism."