Re: [Monotone-devel] upgrade of included botan for mtn

[Markus Wanner]

Hello Jack,

thank you very much for commenting on this issue.

Jack Lloyd wrote:
> I had thought there was a branch that allowed external Botan installs
> to be used and also built Botan using it's normal built routine. Is
> this dead?

That's .. uhm .. not dead, but... not really alive either. It's 
certainly one of the high priority things I have in mind for monotone.

> It would be nice to avoid having me rename things (which in
> the case of source files at least seems harmless) and have it cause
> more work for you to upstream merge. (Especially since you are on dev
> branch, it would be bad for Monotone to get 'stranded' on some 1.7
> point release that never got any bug fixes...)

Yup.

> > The Global_RNG of botan has gone, so I've added a pointer to an RNG to the 
> > app_state, the key_store and the database. Most places using an RNG have 
> > access to a key_store object, so we could maybe even get rid of the pointer 
> > in the database object.
>
> Wait, 3 RNGs?

Uhm.. no, a single one pointed to from the above mentioned objects.

> This is OK (and might be more secure in some scenarios, for example if
> randomness seeding failed catastrophically 1/3 of the time) but it was
> particularly the intent for this change that single-threaded programs
> like Monotone would be able to use a single RNG instance across its
> entire execution.

We had exactly that as well with the Global_RNG before, didn't we?

I'm now calling make_rng() once when initializing the app_state and 
passing on a boost::shared_ptr. (Do I have to seed the RNG after 
make_rng()?)

> > Only mkstemp.cc was puzzling me: I've now changed it to assign its own RNG. 
> > Dunno if that can be optimized to use monotone's, but OTOH it maybe doesn't 
> > matter.
>
> The usual convention IMO would be to pass it a RNG reference.

That's what I intended to do. The problem is that I'm not sure how to 
pass it to that lua extension.

I don't think it's much of an issue, though, because the RNG there is 
only used to create a name for a temporary directory AFAICT.

> This
> make it clear that mkstemp.cc needs random numbers, and also avoids
> (if one were to just create a new RNG inside mkstemp) all the seeding
> overhead. Of course this only works if most of the times when mkstemp
> is called an RNG is available through some other part of the object
> reference graph (app_state or whatever).

> Not entirely gone, just only in mem_pool.cpp and now derived from
> bad_alloc.  I wasn't sure if you didn't realize that or if you did and
> were just eliding the detail for simplicity.

..the later.

> But yes catching
> bad_alloc is the correct response. This was actually motivated by a
> comment in Monotone's sources (I think by Graydon) that said something
> to the effect of "Why do people make up their own bad_alloc?", I
> thought about it and realized that made sense, especially as bad_alloc
> is the sort of failure that one handles way high up, and in a big
> system perhaps in code that predates the introduction of Botan into
> the codebase.

That's exactly the place. We had an extra catch for the old 
MemoryExhausted exception there. With the only purpose of raising a 
bad_alloc instead. I've stripped that extra code and the cited comment 
just above it in favor of relying on Memory_Exhausted being derived from 
bad_alloc.

> a) Is this limitation due to the current static build configuration in mtn?

This limitation is due to not using botan's configure logic but stuffing 
all the cc and hh files into our own Makefile.am. Monotone's included 
botan lacks modularization and only distinguishes between unix/win32 (by 
adding es_win32.cpp for the later). I'm not sure how well that works for 
Windozen at all, but so far we had no complaints...

> b) library-build is the name of the branch where Botan gets built from vanilla 
> sources?

Sorry, I've not been overly specific in all of this email, it seems.

n.v.m.library-build as a dev. branch of monotone which tries to allow 
linking monotone against external libraries instead of the current "we 
don't rely on external dependencies - we re-bundle them instead" 
approach. This would solve the issue by letting botan use its own 
configure script and build infrastructure, which would be a very good thing.

The current curse of action is to propagate from your botan repository 
into a staging branch, where we do necessary adjustments for integration 
into monotone.

To prevent pulling in the complete botan history we do only copy files 
from the staging branch and mtn add/remove manually.

> The asm is not very tuned; with good compilers the C++ is sometimes as
> fast or faster. I would recommend focusing on Dean Gaudet's SSE2 - it
> is significantly faster than the asm on every SSE2-enabled machine
> I've tried it on, it may work on Windows (Visual C++ 2008 Express
> would not compile it, but I think that is a limitation of Express; at
> least I would think VC++ would support Intel's intrinsics?), and it
> works on both all x86-64 CPUs and on most x86 chips made since 2003 or
> so. Don't get me wrong I'd love to make the SHA-1 asm much faster
> (perhaps copying Dean's SSE2 algorithm into inline asm?), but at the
> moment the SSE2 code is where to go for the fastest SHA-1 (my only
> thought was that OpenSSL's might be faster, but I checked and the SSE2
> is faster than OpenSSL 0.9.8g's SHA-1 on my Core2 at least, hard to
> say about 32-bit x86 tho). And the 32-bit x86 asm SHA-1 (asm_ia32
> module) should be significantly faster than the C++ on non-SSE2 x86,
> and maybe faster than the SSE2, but I haven't compared them yet.

Aha, good to know.

Thank you again.

Regards

Markus Wanner