monitoring (was: [Savannah-hackers-public] admin help)

[Dave Love]

Sylvain Beucler <address@hidden> writes:

>> OK, I can doubtless do some of that.  I can also do sysadmin jobs if
>> that's any use.  [Do you have a monitoring system in place for the
>> services?  I have some experience of that if it's of any interest.]
>
> That's of interest :)

OK, I assume you're not doing specific monitoring at present.  Sorry
if this is the sort of thing you do know.  I haven't tried to look at
the system yet.

What I've done before is used Nagios (nagios2 is in Debian now, I
think).  It runs on a separate server (preferably) to monitor whatever
services you're providing and alerts groups of people by email (or SMS
etc.) in a flexible way if something goes down.

You can write `plugins' (e.g. in sh) to run specific tests if there
isn't one available for what yo need.  As far as I know there isn't a
standard CVS pserver plugin, for instance, but you can write something
to test checking out an arbitrary file, for instance.  Previously I
just used the built-in port check to test that pserver was running.
Other things are canned, like testing an https URL (including
validating the certificate).

As well as monitoring things remotely, you can test locally on the
server you're monitoring and report them either via a separate agent
or SNMP, for instance.  (Nearly-full filesystems is a useful test in
my experience!)

You can get statistics on the services, which can sometimes be useful.
You can expose status information to users from the web interface, if
appropriate; it might avoid storms of reports when something fails if
users can check that someone will have been alerted.

It would make most sense to have Nagios (or something similar) set up
to monitor all GNU services, though, and perhaps there's already
something running which could be used.  I don't know how the GNU
systems are actually adminned overall these days.

I could look into setting up such a system if you think it would be
sufficiently useful.  Maybe it isn't so useful if savannah is fairly
reliable and problems always get spotted quickly these days, and it
would be better to work on other things.

Something like Cfengine might also be useful for helping with some of
the issues in the task list, though I got rather disillusioned by
Cfengine itself.  It can do things like tidy up lock files and manage
daemon processes.