[Monotone-devel] merged: sqlite 3

[graydon hoare]

hi,

I've merged in christof's sqlite 3 branch. thanks christof, this is very 
helpful! (sorry I haven't merged the ssh one yet, but it seems to be 
using stuff which I'm not sure will work on win32 just yet). anyways, I 
made the following small changes to your work:

  - I added a function which detects the sqlite3 file format and
    complains if you're trying to use a non-sqlite3 file, so that it
    is a little more obvious what to do to migrate.

  - I adjusted the default and maximum page sizes. these are only
    advisory in sqlite 3; the same binary can read and write databases
    with any such settings, since the page size is variable at the
    database level. I checked, it really does work.

this is really great to have working. in particular, it means we are in 
a much better state as far as size limits. as a test, I loaded a 107mb 
mp3 file into a database. it's slow and eats a lot of memory (> 500mb) 
but it works. the resulting database isn't even that big (144mb). the 
only remaining hard size limit is on the database file as a whole (2TB), 
the rest is just us being sloppy with memory use.

I've synced this change up to the net.venge.monotone head, so 0.17 will 
have it. you'll have to dump and reload if you update to it; the 2.x 
file format is now officially deprecated for monotone :)

now there are a bunch of follow-on tasks someone can do if they want to 
have a go at further optimization:

  - make a cache of parsed queries in database.cc and have the query
    execution reuse them, binding to parameters. this will mostly
    eliminate the sql parsing cost (though not the logging cost).

  - play with switching to blobs for files, or possibly for everything
    blobby. we're under no *real* obligation to keep files in base64,
    except that it makes them not screw up your xterm when you do
    monotone db dump. maybe that's not worth the price (?)

  - play with switching the file_data object to a lazy / streamy sort
    of thing, so that it keeps most data out of memory and can handle
    larger data better.

  - as a less intrusive step, figure out where the worst number of
    in-memory copies of a file are made, and see if we can get that
    down a bit.

next up: the restrictions branch...

-graydon