Staging rollouts to production environment

[Brian Thomas]

So I've spent probably way too much time thinking about this, and it may
be that others have come up with more elegant, simpler solutions. Which
is part of why I'm posting this.

Anyway, the problem we're trying to solve is: How can one gracefully
manage rollouts to a large production environment (2000+ systems), such
that changes to the cfengine environment can be tested before making it
out to the entire world?

We started with the following assumptions:

1) Every possible change that could effect multiple systems had to be
staged prior to rollout -- That is, cfengine's own files (update.conf,
cfagent.conf, impoted .cf's, etc.), or any file copied/modified/etc.
through cfengine.

2) It's OK for the staging environment to consist of representative
samples of each sub-group of machines. i.e. one or two database systems,
one webserver, one dns server, etc. -- That way, worst case, only a
small slice of any of our farms could be taken out by a massive mistake.

3) Changes to the staging environment should be expected to rollout to
the production environment, unless something really bad is found in
testing. (This is just a philsophical difference, but an important one.
Staging is for changes that are presumably considered solild enough to
go live, staging is just their last stop before going there.)

The biggest problem for us to solve was #1 -- How can we test an
update.conf without hand-creating one on every staging system? We could
have just specified a different one to be copied from the policyhost if
it was a staging client versus a production client, but we wanted to try
and keep production and staging servers entirely separate. A staging
server could then have things like a staging cfservd configuration file,
or test an entirely new layout of the file depot if we needed it to
(where the policyhost keeps the files for the copy actions). So as long
as the update.conf needs to be unique, you have a problem unless you
want to hand-copy update.conf's to staging machines.

So the answer I initially came up with was to try and use envars, via a
file we already used for our custom-built cfengine RPM,
/etc/sysconfig/cfengine. Originally we didn't want to have to create a
new RPM for an initial update.conf every time we changed basic
information (such as default policyhost) in our cfengine environment.
But since RPM's don't allow user interaction [insert random grumbling
about evangelistic developers here] we seed the update.conf with the
mutable information based on values in the /etc/cfengine/sysconfig file,
like:

CFDOMAIN=[domainname]
CFDEFSERVER=[policyhost]
CFTRUSTKEY=[yes]

etc. So my original idea was to add something like:

CFCLIENTTYPE=[production|staging]

Or similar, and somehow have something parse this file (.
/etc/sysconfig/cfengine) before cfexecd ran.

Only, uh, it turns out accessing envars from cfexecd calls is really
hard, or at least, I couldn't figure out an easy way to do it.
Supposedly you can access system envars from within the update.conf, but
I never got it to work right, no matter how hard I tried. So instead of
driving myself crazy, I punted and decided instead to use the FileExists
test in the groups: section as a 'toggle', basically saying, "If the
file /etc/cfengine.stagehost exists, consider yourself a staging server,
otherwise, you're production."

Either way, this kind of tweak has the nice effect of being able to
manage it through a manual process (i.e. touch the file by hand, presto,
it's a staging client), or through cfengine (For this group of servers,
add the stagehost file, from now on it's a staging client).

The relevant portions of the update.conf are:

--
groups:

  cfstagehost          = ( FileExists(/etc/cfengine.stagehost) )

control:
     
  actionsequence  = ( copy tidy )    
  domain          = ( domain.com )   
  workdir         = ( /var/cfengine )
  cfdepotdir      = ( $(workdir)/depot/cfengine )
  inputdir        = ( $(workdir)/inputs )
  master_update   = ( $(cfdepotdir)/common/master.update.conf )
  master_cfagent  = ( $(cfdepotdir)/common/master.cfagent.conf )

  # Setup for staging/production:
  cfstagehost::  primaryph   = ( stagepph.usa.domain.com )
  !cfstagehost:: primaryph   = ( prodpph.usa.domain.com )
  any::          secondaryph = ( prodsph.usa.domain.com )
--

Note that we set it up so if a prod OR staging host can't access it's
primary policyhost, we specify a secondary policyhost that either one
fails over to. At some point in the future the environment will be big
enough to have primary and secondary policyhosts for both production and
staging.

So my question is, is there an easier way to take care of this? I
searched around for quite awhile, and never found anyone with a solution
that allowed for testing everything including update.conf.