SUMMARY: Package management with cfengine

[Luke A. Kanies]

I wanted to touch base again and let everyone know what I'm doing with
this and how well I think it's working.

I basically followed the example I emailed earlier.  I created a
packages.cf which I import in cfagent.conf, and all package management
starts from there.  Here is a snippet of how it looks:

control:
    packagefile = ( /var/cfengine/inputs/packageinstaller.cf )

    AllowRedefinitionOf = ( packages linkinstall )
    linkinstall = ( "" )
    packages = ( "" )

groups:
    deploy_packages = ( host1 host2 host3 )

    packagefileexists = ( IsPlain(${packagefile}) )

# do all the package installation work first, so it's clear what
# exactly is happening
import:
    # the InstallPackages class is set in groups.cf
    deploy_packages.packagefileexists.InstallPackages:: ${packagefile}

# now set up for creation of the file
control:
    packageroot = ( /software/master/${uname}/${osrev}/${processor} )
    packagedest = ( /usr/local/software )
    packageserver = ( masterserver.domain.com )

    # list the packages I want to install
    deploy_packages::
        packages = ( "cfengine--2.1.0.5 cvs--1.11.5 bash--2.05b
coreutils--5.0 less--381 vim--6.2")

    # and then create the file to install them
    any.deploy_packages::
        actionsequence = ( "module:packages ${packages}" )

---------------

This is definitely hackish, because on the first run of this config file,
packageinstaller.cf does not exist; it gets created by the packages
module.  In fact, that's the only purpose of the packages module:  Create
the installer file.  On the second run, the file exists and it gets
executed.  So, using this method, you're guaranteed to need at least two
runs for any changes to take affect.

I also created an inputs/packages subdirectory, and I put per-package
snippets and a stub file in there.  The stub looks like this:

control:
    actionsequence = ( copy )

    linkinstall = ( "${linkinstall} ${package}--${version}" )

copy:
    ${packageroot}/${package}/${version}
                            dest=${packagedest}/${package}/${version}
                            server=${packageserver}
                            r=inf
                            o=bin
                            g=bin
                            trustkey=true
                            purge=true
                            ignore=man
                            ignore=doc
                            ignore=info
                            backup=false

----------------

All this does is copy a specific revision of a file into
/usr/local/software, in a directory named for that file and revision.  I
could use this to install multiple revisions of the same package, although
my modules don't currently support that.

And module:packages basically iterates across the list of packages I give
it, so it looks a lot like this:

control:
  package = ( cvs )
  version = ( 1.11.5 )

import:
  /var/cfengine/inputs/packages/mainstub.cf

....

-------------------

And it does that for each package.  Although it doesn't actually import
the file, it includes the file directly.  Same idea, though.

In addition to the stub file, module:packages looks for a per-package or
a per-package-revision file and includes that, too.  My cfengine package
file looks like this, for instance:

control:
    actionsequence = ( editfiles.cfedit )

processes:
    "cfservd"           restart "/usr/local/sbin/cfservd"

editfiles:
    cfedit::
        { ${rootcron}
            AppendIfNoSuchLine "0,15,30,45 * * * * /usr/local/sbin/cfexecd
-F"
            Define cronrestart
        }

        { /etc/services
            SetLine "cfengine                5308/tcp        #
Configuration Engine"
            AppendIfNoLineMatching "cfengine.*"
        }

------------------

This actually needs to be modified a bit, because it doesn't deal with
making sure the keys are set up, and the cron editing isn't as good as it
could be, but the basic idea holds.  The theory is that this per-package
config file would do everything necessary to set up a given package, and
then would verify forever that that package was set up correctly.  As
another example, here's my sshd.cf:

control:
    actionsequence = ( editfiles.cfedit )

processes:
    any::
        "sshd" restart "/usr/local/sbin/sshd -D"


editfiles:
    cfedit::
        { /etc/services
            SetLine "ssh    22/tcp  # SSH Remote Login Protocol"
            AppendIfNoLineMatching "ssh .*tcp.*"
            SetLine "ssh    22/udp  # SSH Remote Login Protocol"
            AppendIfNoLineMatching "ssh .*udp.*"
            SetLine "x11-ssh-offset 6010/tcp    # SSH X11 forwarding
offset"
            AppendIfNoLineMatching "x11-ssh-offset.*tcp.*"
        }

-------------------

this is also incomplete, because it doesn't create /var/empty or the sshd
user/group, and it doesn't describe where to get the config files from,
which it should do.

Then, after all of the per-package stuff is done, I append this:

        actionsequence = ( shellcommands )

groups:
        linkinstalled = ( IsPlain(/usr/local/scripts/adm/linkinstall) )

shellcommands:
        linkinstalled::
                "/usr/local/scripts/adm/linkinstall -c ${linkinstall}"

-----------------------

linkinstall takes the list of packages and links their contents back to
/usr/local.  It's a bit more sophisticated version of the following:

for dir in bin sbin lib libexec include; do
  for file in $packagedir/$dir/*; do
    rm -f /usr/local/$dir/$(basename $file)
    ln -s $file /usr/local/$dir
  done
done

I've actually got a version of linkinstall that just generates cfengine
code and runs that, although I'm only using it at home.

--------------------

And how do I like this system?  Well, first I'll say that I only copy
packages down once a day, because it is too slow otherwise.  But other
than that...

I like it a lot.  It allows me to completely describe everything about a
package in one single place.  Where are you describing how to set up
cfengine or ssh in your configs?  Is that cleanly separated from the rest
of your configuration?  This setup allows you to create one script which
should completely describe a "valid" setup for a given package, including
creating users, starting processes, and where to get config files, and
then it should always bring an installed machine into compliance.

This is definitely the first "package manager" I've ever used that will
daily verify that every package installed is still configured correctly,
down to having the right config files (pulling them down using cfengine)
and all the right processes are running.  Currently the entire config only
runs once a day, but I am in the process of changing things so that the
copy gets run once a day but everything else runs every time cfengine
runs.

I really, really like this so far.  It's very simple to use, once it's all
set up, and it makes it trivial to install new packages.  It's also nice
because if you did a bad compile of a package or something, it's easy to
fix.  For instance, a coworker compiled perl with a default he later
wanted to change, but he'd already installed it in a number of places.
Rather than having to create a fake "5.8.2a" version, he just recompiled
it and replaced the version on the master server; within a day, everyone
had the updated version without complaining.  It was very cool, and very
easy.  I recently had to increase a buffer in cfservd, and rather than
having to do a pkgrm/pkgadd on every machine on my network, I just swapped
out the binary.

Obviously, you have the option of considering that binary change a new
version, but you aren't required to, which is pretty nice.

I guess I should further note:  This doesn't really "solve" the package
management problem, because you'll probably always have packages and
patches that get installed using the vendor tools.  And there will
probably be packages you don't want to install this way; I don't think I'd
want to install 1000 packages on a workstation this way, for instance.
But for the packages you create to do your job, I think most people would
find this very useful.  And you might just be able to get your apps people
to use it too, who knows.  And what I'd really like to see some day is
people sharing their package installation scripts.  If I've got a script
that completely describes a valid sshd installation, why should
someone else have to write one?

Obviously, this does make bootstrapping cfengine slightly more
interesting, since, um, you can't install cfengine until cfengine's
installed, but it's still surmountable.  I currently create a
self-extracting archive with the binaries and current configs, including
the package configs, and that works well.

I hope to write this up for ONlamp or something relatively soon.  I also
would like to release the code, but I'm going to have to reimplement it
from scratch to do so, as my current client does not allow me to keep
ownership of my code.

If you would find this writeup and/or code useful, you should let me know,
as I prioritize on things people actually want done.  And if you have any
questions, please ask.

Luke

-- 
The salesman asked me what size I wore, I told him extra-medium.
                -- Stephen Wright