Re: [Infrastructures] FW: Version control checkout

[John Sechrest]

Luke, 

      This is an interesting path. We have been keeping
      system configs in CVS and using cfengine to check them
      out too. However, the system admins got a bit confused
      about how to represent the testing tree vs the production
      tree. So they could not roll in a test, try it and then
      roll it into production.

      How would you approach the branch for testing vs production,
      so that you can watch the checkout happen on the testing
      system, and then once it works all the way thru, roll it
      into service on the production instances?




"Luke A. Kanies" <luke@madstop.com> writes:

 % (I'm CC'ing this to the cfengine list because it's pertinent to both
 % lists.)
 % 
 % On Mon, 29 Mar 2004, Francis Liu wrote:
 % 
 % > >Hi,
 % > >  it's easy enough putting configuration files into CVS, but how do I
 % > >select the right files to pull out when a machine is built? I'm thinking
 % > >it's got something to do with tags and maybe a module for each type of
 % > >configuration, where all the changes are added together and somehow the
 % > >"correct" version is selected or created, but I'm not sure how.
 % > >
 % > >  Can someone offer me some clue as to:
 % > >1) how to structure the repository
 % > >2) how to checkout the right files
 % > >
 % > >I've read the paper, and I've checked the mail archives, but the answer is
 % > >not obviousŽâ€Ž¦ is it all achievable with cfengine? I can't believe that 
I
 % > >don't need other scripting to wrap it all up. I've got a unix environment,
 % > >it's mainly solaris, but there is some hp-ux.
 % 
 % I have a bit of a different way of solving this problem apparently.
 % 
 % The easy part is how to get the files out of CVS (or subversion or
 % whatever).  I really like this method and find it works quite well.  And
 % incidentally, I've written an article on this for onlamp.com, which
 % hopefully should be out next month.
 % 
 % This is the basic data flow:
 % 
 % Human's sandbox -> CVS repository -> cfengine server -> client
 % 
 % The easiest CVS example is the cfengine configuration itself:  I store it
 % in config/cfengine/inputs (and I usually have a data/ and modules/
 % subdirectory).  Check that out in your sandbox (I use ~/cvs), make your
 % changes, check them in.
 % 
 % Now check them out on your cfengine server.  I like to store it wherever
 % and then link it to /cfengine:
 % 
 % cd /export
 % sudo mkdir cfengine
 % cd cfengine
 % sudo cvs co config/cfengine
 % 
 % One of the things you need to add to your config is the following:
 % 
 % shellcommands:
 %   cfengine_server::
 %     "/bin/sh -c 'cd /export/cfengine/config; cvs up -d > /dev/null
 % 2>/dev/null'"
 % 
 % Then have your update.conf set up thus:
 % 
 % copy:
 %   /export/cfengine/config/cfengine dest=/var/cfengine
 %   ....
 % 
 % -------------
 % 
 % As I said, that's the easy part, but it gets the basic data flow down.
 % You make your changes in the copy of the config files in your sandbox,
 % and the next time cfagent runs on the server the updates get pulled down,
 % and then the clients pull the changes down when they next run.
 % 
 % I actually extend this by making a cvs module.  It's an incredibly simple
 % script.  I keep a hash of checked out cvs directories to be put into
 % /var/cfengine/data/cvs.pl:
 % 
 % $VAR1 = {
 %   configmaster => {
 %     module => 'config',
 %     start => '/cfengine/config',
 %   },
 % }
 % 
 % 
 % Then I have a very simple perl script that accepts a list of cvs modules,
 % sources this file, and cd's into each module and runs 'cvs up -d -P'.  As
 % you can see, the above module defines the root of the config tree, and I'd
 % call it like the following:
 % 
 % control:
 %   actionsequence = ( "module:cvs configmaster" )
 % 
 % You can define as many modules as you want in a variable and then just
 % pass that variable to the module.
 % 
 % ----------------
 % 
 % I don't currently have a copy of this code that I can release, but
 % hopefully I soon will.  As always, being prompted is much more likely to
 % result in faster response.
 % 
 % ----------------
 % 
 % Okay, now we've got our cfengine configs, including modules and data,
 % maintained out of CVS and automatically being updated by cfengine.  It's
 % especially important in this model that the cfengine server run cfagent
 % with no SplayTime, so that it does a 'cvs up' before all of the clients
 % connect.
 % 
 % Now we want to manage other configs out of CVS.
 % 
 % Some configs are easy.  You almost definitely have the exact same ssh
 % configurations on all of your servers, or at least all servers of the same
 % platform.  Add them to your CVS repository, at config/sshd, and then just
 % copy them to your server.
 % 
 % Maybe something like the following:
 % 
 % # sshd.cf
 % copy:
 %   ${filesource}/config/sshd/. dest=/usr/local/etc/ssh/.
 %   r=inf define=restart_sshd
 %   ...
 % 
 % shellcommands:
 %   restart_sshd::
 %     "/bin/pkill sshd"
 % 
 % processes:
 %   "sshd" restart "/usr/local/sbin/sshd"
 % 
 % --------
 % Just make your changes in your cvs sandbox, commit them, the cfengine
 % server automatically runs 'cvs up -d -P' in its checked out copy, the
 % clients all copy them down, and they restart the server if any of the
 % files have changed.  I don't know of a way to do this _only_ if
 % sshd_config changes, but that's fine.  I'm kind of using trickery here by
 % just killing sshd if it needs to be restarted and then depending on sshd
 % to restart the process automaticaly.
 % 
 % 
 % For configurations that are less consistent across platforms, things
 % obviously become more complicated.  There's a sliding scale of differences
 % here.  Easier examples are if you've got one sendmail.cf for your server
 % and another for all of your clients.  In that case, I commit them as
 % something like sendmail.server and sendmail.client and then just copy them
 % based on roles:
 % 
 % copy:
 %   mail_server::
 %     ${filesource}/config/sendmail/sendmail.server
 %        dest=/etc/mail/sendmail.conf define=restart_sendmail
 %        ...
 %   !mail_server::
 %     ${filesource}/config/sendmail/sendmail.client
 %        dest=/etc/mail/sendmail.conf define=restart_sendmail
 % 
 % It gets more complicated if you find that roles don't map one to one with
 % files.  inetd.conf is the best example of this because different roles
 % result in different daemons running, and you often end up with essentially
 % unique files.
 % 
 % My solution here closely resembles the above solution for the cvs module.
 % I make an inetd data file and store a hash of each of possible config
 % line:
 % 
 % $VAR = {
 %     'telnet_hp' => { # telnet stream tcp nowait root /usr/lbin/telnetd
 % telnetd -b /etc/issue
 %         'wait' => 'nowait',
 %         'endpoint' => 'stream',
 %         'command' => 'telnetd -b /etc/issue',
 %         'binary' => '/usr/lbin/telnetd',
 %         'protocol' => 'tcp',
 %         'uid' => 'root',
 %         'name' => 'telnet'
 %     },
 % }
 % 
 % Then collect the list of modules to enable via roles:
 % 
 % control:
 %   solaris::
 %     inetd = ( "${inetd} rstatd_sun telnet_sun" )
 %   hpux::
 %     inetd = ( "${inetd} rstatd_hp telnet_hp" )
 % 
 %   any::
 %     actionsequence = ( "module:inetd ${inetd}" )
 % 
 % Notice that I'm not able to just say "rstat" or "telnet" for this and have
 % it pick the correct one.  My module isn't very smart, but maybe you could
 % do better. :)
 % 
 % ------------------------
 % 
 % If each machine truly has a unique config file, then that's easy: just
 % make a config/<machine> module, check it out on the server, and then copy
 % it down.
 % 
 % --------------
 % 
 % That about covers the gradient of how I get config files to systems.
 % There are two things that maybe you should notice here:  First, I am not
 % checking files out directly on the systems.  This allows me to completely
 % abstract how files are stored on disk from how files are stored in the
 % repository.  This is very useful.
 % 
 % Second, because of this capability for abstraction, I don't store my files
 % in the CVS repository in much of a way resembling how they sit on disk.
 % I generally divide them instead by related app ("config/<app1>",
 % "config/<app2>"), sometimes creating a single common directory for all
 % files that don't deserve their own directory (the sudoers file is sudo's
 % only config, there's probably only one resolv.conf, etc.).
 % 
 % I plan on turning this whole thing into one or more articles in the series
 % I'm writing for onlamp.com, but I won't be writing them until at least
 % mid-April and it appears to take about a month for them to get published,
 % so if anyone has any questions in the meantime, feel free to ask.
 % 
 % Luke
 % 
 % -- 
 % The Number 1 Sign You Have Nothing to Do at Work...
 %    The 4th Division of Paperclips has overrun the Pushpin Infantry
 %    and General White-Out has called for a new skirmish.
 % 
 % 
 % _______________________________________________
 % Help-cfengine mailing list
 % Help-cfengine@gnu.org
 % http://mail.gnu.org/mailman/listinfo/help-cfengine

-----
John Sechrest          .         Helping people use
                        .           computers and the Internet
                          .            more effectively
                             .                      
                                 .       Internet: sechrest@peak.org
                                      .   
                                              . http://www.peak.org/~sechrest