optimized SHA1 (was Re: [Monotone-devel] net.venge.monotone.experiment.performance)

[Nathaniel Smith]

On Tue, Aug 01, 2006 at 12:39:05AM -0700, Eric Anderson wrote:
> Needs further examination:
>   997a677db676734acc0d098979d2a9cee8765ec9: 
>     Add support for OpenSSL's libcrypto.  Measuring Pull(net.venge.monotone)
>     on a 2.8GhZ Xeon, this was worth 1.3x cpu reduction on the server,
>     and 2.4x cpu reduction on the client. On a 1.7GhZ Pentium-M, 1.07
>     on the client, and almost nothing on the server.  If some other
>     people could try this out (configure using --with-crypto to enable
>     usage of the library), it would be nice to find out whether it matters
>     on other platforms.  

Ah, right, and those percentages will only go up as we remove fat,
since SHA1 is one of the only truly unavoidable CPU costs.
Unfortunately, OpenSSL's license is not GPL-compatible, except under
particular weird conditions (e.g., maybe if OpenSSL is "part of the
system" then it's okay to use it, or maybe it's okay if its use is
optional, etc.); nothing that makes me very comfortable.  (And nothing
that I can think of that would let us distribute OpenSSL-linked
windows binaries.)

It'd be nicer to just free the hand-written asm SHA1 cores that
OpenSSL uses from the advertising clause, and ideally fold them into
Botan.  I started a page on the wiki working out what would need to
happen to make this work:
  http://venge.net/monotone/wiki/SpeedySpeedySHA1

Richard, I'd appreciate any feedback or tips you have before going
forward on this -- you know the OpenSSL folks and I don't, and I don't
want to step on any toes or accidentally run into some history I don't
know about :-).  (Also, do you by chance understand the tangle of code
that calls into the asm cores?  I'm sure we could work it out, but it
sure isn't as easy to read as the clean self-contained stuff like the
git asm kernels, e.g:
  http://www.kernel.org/git/?p=git/git.git;a=tree;f=arm
)

We should also get Jack Lloyd involved in this discussion -- I think
he reads the list, but I've CC'ed him just in case.  In particular,
is it possible to add such cores to Botan externally, I suppose using
the "engine" support?  (I don't see any documentation on the web site
for what engines can do, or how to write one.)  And what are your
thoughts on adding such asm cores to Botan proper?

-- Nathaniel

-- 
The Universe may  /  Be as large as they say
But it wouldn't be missed  /  If it didn't exist.
  -- Piet Hein