[bug#33210] Cuirass: Use a SQLite in single-thread mode

[Clément Lassieur]

Hi Danny,

Danny Milosavljevic <address@hidden> writes:

> Yeah, right now users can query something using the web interface while
> a build is updating (or running) at the cost of the returned data being
> potentially very strange.

This is quite unlikely.

> The "one-line fix" would make it worse in that users cannot query while
> a build is running, making them wait until the build is done (approx.
> 30 min) before the query succeeds.  The upside is that the returned
> data is consistent at all times.  This is how DBMSes did it in the 90s,
> too.

No, there are no SQL requests during the build.  There are some before
the build, and some after the build.  But they are all short.

> What I'd like to eventually have is the proper fix where users can query
> while a build is running, *and* the build doesn't have to wait either.
> This works just fine using two transactions with WAL mode of sqlite,
> which means it uses MVCC in order to keep both "world views", one for the
> querier and one for the builder (easily extended to an arbitrary
> number of queriers and builders at once by just having more transactions)
> while they are both using "the world".

It's already the case, because all the queries are very short.

>> is an essential difference however: if we take care of the scheduling,
>> we won't have SQLITE_BUSY blocking the Fibers scheduler all the time.
>> And blocking the Fibers scheduler has an impact on all other possibly
>> unrelated Fibers clients.
>
> Right.  I just wanted to make sure we understand the possible implications 
> here.
>
> In the end I'm not sure we even need multithreading even for my scenario -
> maybe (probably) just having an extra sqlite_open would be enough, threads
> or not.  On the other hand there are shared caches etc and this change here
> could cause some very tricky problems then.

I don't understand this.  Could you explain why we would need an extra
sqlite_open, what change are you talking about, and why there could be
tricky problems with shared caches?

> I have to say I liked the external evaluator much more since then all
> this complexity would be contained in the external program and it would
> just magically work without special-casing any of this stuff.

The evaluator is still external, I'm not sure what you are talking
about.

>>  When guile-sqlite3 handles SQLITE_BUSY
>> correctly, I'll be happy to switch back to a multi-threading SQLite.
>> While waiting for it, I believe our users want a fast Cuirass, and I'd
>> like the time spent in the Fibers scheduler to be balanced by removing
>> the SQLite now useless mutexes.
>
> That makes sense.
>
> It's difficult for guile-sqlite3 to handle SQLITE_BUSY correctly since
> sqlite also uses SQLITE_BUSY to indicate errors that you are supposed to
> fail on.

[...]

> Hmmmmmmmm.  I think that can be done.

Cool!

Cheers,
Clément