[Fab-user] Experiences working with Fabric

[Daniel Pope]

I've been working with Fabric for deployments of perhaps a dozen apps 
across our 70-odd server network. As I'm now leaving this contract I 
wanted to submit the experiences I had using Fabric in case the 
community would find that beneficial.

- Our servers are configured using Puppet (http://www.puppetlabs.com/), 
and I quickly realised that Puppet classes of the servers are tied very 
closely to the apps I was deploying to them with Fabric. This allowed me 
to write sanity checks for our servers such as

assert hasclass('apache::nso')

that have saved embarrassing misadventures when Fabric is misconfigured 
and head-scratching when Puppet is misconfigured. Implementation:

class_cache = {}

def hasclass(classname):
    """Test whether the current host has the puppet class classname."""
    if env.host_string in class_cache:
        return classname in class_cache[env.host_string]
    else:
        with hide('stdout'):
            classes = run("cat /var/lib/puppet/classes.txt")
        class_set = set([l.strip() for l in classes.splitlines()])
        class_cache[env.host_string] = class_set

        return classname in class_set


- Since I was using puppet it was also convenient to use facter, which 
brings a kind of convenient symmetry:

import yaml

fact_cache = {}

def fact(name):
    try:
        facts = fact_cache[env.host_string]
    except KeyError:
        with hide('running', 'stdout'):
            out = run('facter -y')
        facts = yaml.safe_load(out)
        fact_cache[env.host_string] = facts
    return facts[name]

(Going through YAML was merely convenient as it saved me writing a 
parser for facter's output, and is not a requirement of the approach).

- The fabric sudo() operation had to have pty=True often enough that it 
might merit a global switch. In the end I reconfigured the sudoers files 
on all of the machines, but I could only do this because I had 
sufficient access to the machines.

- Something I wanted time and time again was a remote analogue to the 
GNU install command, ideally something a little like this (which I 
suspect is Debian-specific):

def install(local_file, remote_path, owner='root', group='root', mode=0644):
    """Install local_file to remote path setting its ownership and mode."""
    tmp = run('tempfile')
    put(local_file, tmp)
    sudo('install -D -m %04o -o %s -g %s %s %s' % (
        mode, owner, group, tmp, remote_path
        )
    )
    run('rm -f ' + tmp)

- I very quickly found that I wanted meta-tasks to name a list of hosts 
to run on. Though I didn't like the global side-effect nature of it I 
found the syntax I liked best to be

for host in each_host(['web1', 'web2']):
    run('hostname')

implemented as

def each_host(hostlist):
    for h in hostlist:
        with settings(host_string=h):
            yield h

However the implicit side-effects of the above approach meant I avoided 
it where possible.

- Instead I wrote a @default_role(role) decorator which supplies a host 
list if no hosts are in the env when the task is called.

Thus I could equally do

$ fab deploy

to use the default role for deployment, or

$ fab -H foo deploy

to deploy to a specific server, perhaps a new one.

This allows me describe meta-tasks that take no hosts (ie decorated with 
@hosts() or @runs_once) and call those component tasks, eg.

@runs_once
def switch(tag):
    """Switch to a different release."""
    take_site_offline()
    switchreleasetag(tag)
    purgememcached()
    purgevarnish()
    clear_db_caches()
    put_site_online()

with component tasks like

@default_role('varnish')
def purgevarnish():
    """Purge the varnish servers"""
    assert hasclass('varnish::nso')
    run('varnishadm -T localhost:6082 url.purge .')

- One of the biggest things was that we have multiple environments: 
development, staging, production and something ill-defined called 
"preview". These each have different hosts, different settings, etc.

I wrote a load() task that would populate a global environment. I also 
wrote a decorator that ensures an environment has been loaded and errors 
out otherwise. This leads to command lines such as

fab load:nso-staging switch:nso_5.10.00.rc2

I don't know if this could be made general enough for inclusion in 
Fabric but I have to report I fought with this for quite a while to get 
everything I needed out of Fabric with a commandline syntax I liked.

-

I hope the above is useful. I've found Fabric to be an excellent tool; 
the developers were writing deployment scripts in ant. I'm pleased to 
report that having converted them to Fabric they are easier to read and 
maintain, more flexible, and faster.

Dan
--
Mauve Internet
t: 01243 888187
w: www.mauveinternet.co.uk