Bootstrapping

[Luke A. Kanies]

Hi all,

I know there are many solutions to the bootstrapping problem, and I'm
curious to hear what other people are doing.  I'm going to describe a
couple of methods I've used for bootstrapping cfengine, but I've yet to
find a relatively fast, completely automated means of bootstrapping
cfengine.

I've got my first cfengine article submitted to O'Reilly, and I'm nearly
done with the second.  My third article is going to be on bootstrapping
cfengine, so I would like to get an idea of how other people do it.

Here's what I see as the problem:

I have two possible bootstrapping cases, an existing machine needs to have
cfengine installed and configured, and a new machine needs the same.  I
want to get this done with as few manual steps as possible, preferably in
a single script at the most, and I especially want to be able to do an
automated bootstrap from the server build process.

Here're what I see as the necessary components:

-The client having the cfengine binaries
-The server allowing the IP address of the client
-Client and server having each other's public key
-The client being granted access to the server
-The client having at least update.conf
-The client and server agreeing on the client's domain

Cfengine binaries:

This is a classic package management problem.  If you're not using
cfengine for package management, then use your normal package management
solution to get the package installed.  If you are using cfengine, then
you're in an interesting catch-22:  How do I install the package used to
install packages?

My current solution is to create a shar (self-extracting executable
archive) file with my cfengine binaries, a setup script, and the most
recent configuration.  This works very well to get the binaries into a
simple setup, and from there my package management configuration takes
over.

Server allowing IP access:

Unfortunately, cfservd cannot grant connection capabilities based on DNS,
only on IP addresses.  If you are lucky (or smart) enough to have all of
your clients on the same IP space without untrusted clients on that space,
then you can simply trust that space, which works well.  If you are
unfortunate to have clients scattered to the 9 winds (like my current
client does) you have to build a long and relatively nasty list of IP
addresses.

My current solution to this problem involves storing host information in
LDAP and using a script to turn that host list into an importable list of
IP addresses to allow, and then having cfservd import that list.

This is functionally sufficient, but involves waiting a bit when you add a
host.  Once you add a new host to LDAP, you have to wait for cfagent to
run on the server and generate the new list of allowed IP addresses, and
then you have to wait for cfservd to realize one of its configuration
files has changed and reimport the new list.  It's functional, but I've
found that this kind of lengthy delay can really, really confuse admins
who are used to being to able to do everything right away.  Obviously, it
can be forced, but that's non-ideal.

Mark recently accepted a patch that will allow me to add the IP addresses
using ExecResult within cfservd, but this will then require a restart of
cfservd so it requeries LDAP.

How are other people solving the 'random list of IP addresses' to import,
especially with long lists that change regularly?

Public Keys:

Trusting both ends of the connection is one mechanism for solving this
problem, and it might be that it's the best mechanism.  After all, once
the keys are set through trust on the first connection, trust is no longer
used, so it's probably fine.  Is that what most people are doing?

For a connection to work, each side needs to either trust the other side,
or have the other side's public key.  The server side is pretty easy here;
you can distribute the public key with your package (which is what I do),
and trusting the server side is usually pretty safe (because you're just
trusting one IP address).

The client side is much tougher:  You have many keys, and they can be a
bear to manage -- my current client has some admins who rebuild boxes two
or three times during their deployment process, and those keys need to be
refreshed after every build.  I have not found what I consider to be a
good process that does not use trust.

I currently have the clients make their public keys available through
cfservd.  Then I have my server connect and retrieve the key, trusting the
client.  Because my client already knows the server, the connection
succeeds and the client's public key gets stored on the server.  This only
works for a new key, though, not updating them.  It's possible that I
could have a script tailing the syslog output on my server, waiting for a
failed connection, and then kick this script off if I get a failed
connection.  Is anyone doing this?

Once I have a client built completely, I run a script that scrubs some
info from the client (mostly uname info and public keys) and stores it in
LDAP, and then I periodically update the on-disk public keys from LDAP.
This currently involves running the script manually (it involves multiple
SSH connections as a normal user), as I haven't found a good mechanism for
having the hosts update themselves in LDAP.

Granting Access:

My current client has an astounding 20 subdomains with Unix machines in
them.  They are trying to consolidate, but...  For granting, I currently
just list all of the granted subdomains, even though that often paints too
broad of a stroke.  This list fortunately doesn't change much, so it's
pretty easy to maintain, once you get it working.

Client having a copy of update.conf:

I generally just distribute this with my cfagent binaries.  The same shar
file that installs a bootstrap copy of cfagent also contains a recent copy
of the entire cfengine configuration.  It's convenient to have the entire
configuration, because my current setup (unfortunately) involves about
three runs on the client and a manual step on the server before the client
is fully functional.  This can take a while, so it's nice that the client
can do some stuff (like basic inetd securing) in the interim.

Domain agreement:

This is an often annoying requirement.  The domain doesn't have to reflect
any kind of "reality", but the server's reverse lookup of the client's IP
address must match what the client provides to the server as configured in
the 'domain' variable within cfengine.  It's important that it be phrased
this way, because there are often many ways of listing a domain.

This is obviously dependent on local host naming policies.  I find the
best way is to define an authoritative means of gathering the domain from
the client (DNS, /etc/resolv.conf, /bin/domainname, hard coded to
something).  If you only have one domain, this is a trivial problem for
you, but for those of us with many domains, we have to autodetect the
right domain or cfengine is essentially non-functional.

I've used a couple different methods.  I currently have a somewhat ugly
script that does an nslookup of the host but defaults to a certain domain:

domain = ( ExecResult(/bin/sh -c "domain=`${nslookup} \`${hostname}\`
2>/dev/null| grep Name | awk '{print $2}' | sed 's/[^.][^.]*\.//'`; echo
${domain:=unix.domain.com}") )

This mostly works, but not only is it not pretty, it also has to be in
both update.conf and cfagent.conf, since cfagent purges info collected in
update.conf.

At my last client, I made sure that /bin/domainname always returned
correct information (I had some somewhat elaborate mechanisms for doing
that on multiple platforms), but you always have to have a default,
assuming it's cfengine that making the information correct in the first
place.  If you have a script to pull the domain, but cfengine copies that
script from a central server, you're in a catch-22.  This is the same
reason why the above is a shell one-liner, rather than an external script.

-----------------------

These are what I see as the basic bootstrapping problems with cfengine.
The less complicated your setup, the easier these are to solve.  Also, the
fewer people you have, the easier it is to come up with a good system that
works.  I've found the best way is to automate as much as possible, and
then have a fixed checklist for the manual portions.

Anyone else have any stories of how they bootstrap their cfengine clients?
How do you solve the above problems?  I hope to incorporate some different
ideas and solutions into my article, so give me anything you think
cfengine newbies should know about.

Thanks,
Luke

-- 
Should I say "I believe in physics", or "I know that physics is true"?
                -- Ludwig Wittgenstein, On Certainty, 602.