Re: Staging rollouts to production environment

[Ed Brown]

I think that it would help to make a distinction between
'testing/development' and 'staging'.  A 'staging' environment, like you
say, is the last stop before rollout to production.  So it would NOT be
the place to be running a different cfservd configuration or a 'entirely
new layout of the file depot'.  You need a testing/development setup for
that kind of experimentation.  

If you want to maintain a true staging environment, it should not be
'entirely separate' in my opinion.  Our changes go from CVS, to a
staging environment, and when we're ready, (and this is the key thing)
from the staging environment(policyhost/file-depot) to production. 
There are no changes directly to, or only on, the production
policyhost/file-depot's.  That's an important criteria I think for a
truly staged rollout.  Though having said that, our staging servers are
in a class of their own, so a change in configuration can be constrained
to only staging servers, for added safety in testing a change.  Also,
makefile 'targets' in the CVS working directory of cfengine files allow
for instantly syncing changes to production, for when you can't wait,
but they still go to staging, and from there to production. 

The questions you raise are important, and I've also been hoping to see
more replies about different approaches to staging rollouts.  It's never
clearcut: the staging environment is also inescapably a testing
environment, and conflicts arise. 

I don't understand your dilemma about update.conf.  Seems like you
answered your own question: either you have a completely separate
testing system (policyhost / clients), or you don't. Using envars rather
than classes, (or to set classes - I'm really not sure where you were
headed with that), doesn't create true separation.  

-Ed  
   

    


On Mon, 2004-10-18 at 12:04, Brian Thomas wrote:
> So I've spent probably way too much time thinking about this, and it may
> be that others have come up with more elegant, simpler solutions. Which
> is part of why I'm posting this.
> 
> Anyway, the problem we're trying to solve is: How can one gracefully
> manage rollouts to a large production environment (2000+ systems), such
> that changes to the cfengine environment can be tested before making it
> out to the entire world?
> 
> We started with the following assumptions:
> 
> 1) Every possible change that could effect multiple systems had to be
> staged prior to rollout -- That is, cfengine's own files (update.conf,
> cfagent.conf, impoted .cf's, etc.), or any file copied/modified/etc.
> through cfengine.
> 
> 2) It's OK for the staging environment to consist of representative
> samples of each sub-group of machines. i.e. one or two database systems,
> one webserver, one dns server, etc. -- That way, worst case, only a
> small slice of any of our farms could be taken out by a massive mistake.
> 
> 3) Changes to the staging environment should be expected to rollout to
> the production environment, unless something really bad is found in
> testing. (This is just a philsophical difference, but an important one.
> Staging is for changes that are presumably considered solild enough to
> go live, staging is just their last stop before going there.)
> 
> The biggest problem for us to solve was #1 -- How can we test an
> update.conf without hand-creating one on every staging system? We could
> have just specified a different one to be copied from the policyhost if
> it was a staging client versus a production client, but we wanted to try
> and keep production and staging servers entirely separate. A staging
> server could then have things like a staging cfservd configuration file,
> or test an entirely new layout of the file depot if we needed it to
> (where the policyhost keeps the files for the copy actions). So as long
> as the update.conf needs to be unique, you have a problem unless you
> want to hand-copy update.conf's to staging machines.
> 
> So the answer I initially came up with was to try and use envars, via a
> file we already used for our custom-built cfengine RPM,
> /etc/sysconfig/cfengine. Originally we didn't want to have to create a
> new RPM for an initial update.conf every time we changed basic
> information (such as default policyhost) in our cfengine environment.
> But since RPM's don't allow user interaction [insert random grumbling
> about evangelistic developers here] we seed the update.conf with the
> mutable information based on values in the /etc/cfengine/sysconfig file,
> like:
> 
> CFDOMAIN=[domainname]
> CFDEFSERVER=[policyhost]
> CFTRUSTKEY=[yes]
> 
> etc. So my original idea was to add something like:
> 
> CFCLIENTTYPE=[production|staging]
> 
> Or similar, and somehow have something parse this file (.
> /etc/sysconfig/cfengine) before cfexecd ran.
> 
> Only, uh, it turns out accessing envars from cfexecd calls is really
> hard, or at least, I couldn't figure out an easy way to do it.
> Supposedly you can access system envars from within the update.conf, but
> I never got it to work right, no matter how hard I tried. So instead of
> driving myself crazy, I punted and decided instead to use the FileExists
> test in the groups: section as a 'toggle', basically saying, "If the
> file /etc/cfengine.stagehost exists, consider yourself a staging server,
> otherwise, you're production."
> 
> Either way, this kind of tweak has the nice effect of being able to
> manage it through a manual process (i.e. touch the file by hand, presto,
> it's a staging client), or through cfengine (For this group of servers,
> add the stagehost file, from now on it's a staging client).
> 
> The relevant portions of the update.conf are:
> 
> --
> groups:
> 
>   cfstagehost          = ( FileExists(/etc/cfengine.stagehost) )
> 
> control:
>      
>   actionsequence  = ( copy tidy )    
>   domain          = ( domain.com )   
>   workdir         = ( /var/cfengine )
>   cfdepotdir      = ( $(workdir)/depot/cfengine )
>   inputdir        = ( $(workdir)/inputs )
>   master_update   = ( $(cfdepotdir)/common/master.update.conf )
>   master_cfagent  = ( $(cfdepotdir)/common/master.cfagent.conf )
> 
>   # Setup for staging/production:
>   cfstagehost::  primaryph   = ( stagepph.usa.domain.com )
>   !cfstagehost:: primaryph   = ( prodpph.usa.domain.com )
>   any::          secondaryph = ( prodsph.usa.domain.com )
> --
> 
> Note that we set it up so if a prod OR staging host can't access it's
> primary policyhost, we specify a secondary policyhost that either one
> fails over to. At some point in the future the environment will be big
> enough to have primary and secondary policyhosts for both production and
> staging.
> 
> So my question is, is there an easier way to take care of this? I
> searched around for quite awhile, and never found anyone with a solution
> that allowed for testing everything including update.conf. 
> 
> 
> _______________________________________________
> Help-cfengine mailing list
> Help-cfengine@gnu.org
> http://lists.gnu.org/mailman/listinfo/help-cfengine