Re: sqlite3

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 06 Dec 2021 20:51:30 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >>From what I see there, you completely ignore the text encoding issue.
> > SQLite needs UTF-8, but our internal representation is a superset of
> > UTF-8, so we need to decide what to do with the stuff that's beyond
> > UTF-8.  Does SQLite check this and return an error indication if the
> > byte sequence we feed it is not a valid UTF-8?  If it does check that,
> > we can rely on it, but we need to signal an error with a clear
> > explanation when it detects something like that.  Alternatively, we
> > should check this ourselves before calling SQLite, and signal an error
> > on our own.
> 
> Good catch.  😀  I was scratching my head about that for about ten
> minutes -- did I forget to put a "POSSIBLY FIXME" in there?
> 
> SQLite does expect UTF-8, but as far as I can tell, it doesn't really
> check anything.  So the current code puts our private binary
> utf-8-private-plane-encoded things into SQLite -- and gets it back as
> is.  (See the sqlite-char test for this stuff.)

If we want to do better, we need to call encode_string_utf_8, it
already has all the infrastructure for this case, including fast
no-copy operation if everything checks out, and a telltale return
value if not (so we could signal an error).

> That is, 
> 
> "insert into test2 (col1, col2) values ('fó\x82o', 3)"
> 
> and
> 
> "insert into test2 (col1, col2) values ('f\x82o', 3)"
> 
> both round-trip without any problems as is.  Perhaps SQlite just sees it
> as a byte string?

As long as you don't ask it to search inside text, it probably doesn't
care.