Re: [rdiff-backup-users] Simple nightly script?

On 10/19/06, David Kempe <address@hidden> wrote:

Hi Greg,
After much experimentation with various scripts, I am happy with this
latest rdiff-backup parser we have written. Script is written by
Christian Marie, who also wrote the rdiff-backup nagios plugin on
nagiosexchange.org.

The script is designed to be passed on stdin the output of an
rdiff-backup job or multiple jobs with -v5 or greater and
--print-statistics both turned on. It can deal with many jobs passed in
ie: backup.sh | parse-rdiff-backup
where backup.sh might run three jobs.
check it out - you get a nicely formatted email that says things like
Backup(s) OK 3/3 in the subject, and a summary of the job like so:

Status of backup(s) follow

Backup completed successfully
        Direction: /home/sol1backup/cookie-data to /var/backupdisk/cookieman
        Backup started: Tue Oct 17 23:00:06 2006
        Source files: 61248
        Source files size: 22290479056 (20.8 GB)
        New files size: 6224482 (5.94 MB)
        Deleted files size: 308229 (301 KB)
        Destination size change: 18.4 MB
        Errors: 79

Anway, Here is the script and config file. Read the source if you want
more info.... :)

Script:
--------------------------------

#!/usr/bin/env ruby
require 'yaml'
require 'tempfile'

class Parser
   attr_reader :status
   DEFAULT_CONFIG = File::SEPARATOR +
                    File.join('etc', 'parse-rdiff-backup.conf')
   BACKUPOKCORRECT = /TotalDestinationSizeChange/

   def initialize data, config = DEFAULT_CONFIG
     begin
       @log = data
       @config = YAML::load_file(config)['parser']
       @statistics_output = []
       @statistics = []
       @config['statistics'].each do |key,value|
         @statistics << SessionStatistic.new(key, value)
       end
       parse!
     rescue Exception => exception
       @status = BackupStatus::ERROR
       @error = "Error: #{exception.class} (#{exception.message})\n"
       @error += "Backtrace:\n"
       @error += exception.backtrace.join "\n"
     end
   end

   def parse!
     @status = BackupStatus::BROKEN
     @log.each do |line|
       @ statistics.each do |statistic|
         if line =~ /#{statistic.regex}/
           if $1
             @statistics_output << statistic.message + $1
           end
         end
       end
       if line =~ BACKUPOKCORRECT
         @status = BackupStatus::OK
       end
     end
   end

   def statistics_output
       return @statistics_output.join("\n").gsub(/^/, "\t")
   end

   def error
     return @error.gsub(/^/, "\t")
   end

   private
   class SessionStatistic
     attr_accessor :message, :regex
     def initialize message, regex
       @regex = regex
       @message = message + ': '
     end
   end
end

class BackupStatus
   OK = 0
   BROKEN = 1
   ERROR = 2
end

class Message
   attr_accessor :subject, :message, :to, :cc, :log
   DEFAULT_CONFIG = File::SEPARATOR +
     File.join('etc', 'parse-rdiff-backup.conf')

   def initialize config=DEFAULT_CONFIG
     @config = YAML::load_file(config)['message']
     @subject = 'Unknown status'
     @message = "Status of backup(s) follow\n\n"
     @to = @config['main_contact']
     @cc = @config['cc_contact']
   end

   def send
     temp = Tempfile.new "rdiff_backup_log"
     temp.puts @log
     temp.close

     zipped_log = "#{ temp.path}.zip"

     `zip #{zipped_log} #{temp.path}`

     IO.popen(
       'nail ' +
       "-a '#{zipped_log}' " +
       "-c 'address@hidden' " +
       "-s 'address@hidden' 'address@hidden'",
       'w'
     ) do |nail|
       nail.puts @message
     end

     File.unlink zipped_log
   end
end

def split_log log
   marks = []
   logs = []

   log.each_with_index do |line, index|
     if line =~ /Starting mirror|increment\ operation (.*) to (.*)/
       marks << index
     end
   end

   marks.size.times do |index|
     first =  marks[index]
     last = marks[index+1] || log.size

     logs << log[first..last].to_s
   end
   return logs
end

if __FILE__ == $0
   parsers = []

   input = ARGF.readlines

   split_log(input).each do |log|
     parsers << Parser.new(log)
   end

   ok_backups = 0
   message = Message.new

   parsers.each do |parser|
     case parser.status
     when BackupStatus::OK
       ok_backups += 1
       message.message += "Backup completed successfully\n"
       message.message += parser.statistics_output
       message.message += "\n\n"
     when BackupStatus::BROKEN
       message.message +="Backup did not complete\n"
       message.message += parser.statistics_output
       message.message += "\n\n"
     when BackupStatus::ERROR
       message.message += "Error running parsing script\n"
       message.message += parser.error
       message.message += "\n\n"
     end
   end

   if parsers.size == 0
     message.subject = 'Backup(s) unknown'
     message.message = 'Status unknown, did not get a valid log.'
   elsif ok_backups == parsers.size
     message.subject = 'Backup(s) OK'
   else
     message.subject = 'Backup(s) failed'
   end
   message.subject += " (#{ok_backups}/#{parsers.size})"
   message.log = input

   message.send
end

--------------------------
Config file:
-------------------------
cat /etc/parse-rdiff-backup.conf
message:
   main_contact: address@hidden
   cc_contact: address@hidden

parser:
   statistics:
     Direction: Starting mirror|increment\ operation (.*)
     Backup started: ^StartTime [\d\.]+ $(.*)$$
     #EndTime: ^EndTime (.*)$
     #ElapsedTime: ^ElapsedTime (.*)$
     Source files: ^SourceFiles (.*)$
     Source files size: ^SourceFileSize (.*)$
     #MirrorFiles: ^MirrorFiles (.*)$
     #MirrorFileSize: ^MirrorFileSize (.*)$
     #NewFiles: ^NewFiles (.*)$
     New files size: ^NewFileSize (.*)$
     #DeletedFiles: ^DeletedFiles (.*)$
     Deleted files size: ^DeletedFileSize (.*)$
     #ChangedFiles: ^ChangedFiles (.*)$
     #Changed files size: ^ChangedFileSize (.*)$
     #ChangedMirrorSize: ^ChangedMirrorSize (.*)$
     #IncrementFiles: ^IncrementFiles (.*)$
     #IncrementFileSize: ^IncrementFileSize (.*)$
     Destination size change: ^TotalDestinationSizeChange \d+ $(.*)$$
     Errors: ^Errors (.*)$

Script is licensed under the GNU GPLv2 and Author is Christian Marie
while working for Solutions First.

thanks

dave

Greg Freemyer wrote:
> All,
>
> I've looked at the examples at
> http://www.nongnu.org/rdiff-backup/examples.html , but none of them seem
> to address automated nightly scripts and error handling.
>
> Are there some more complex examples available?
>
> === Details
> I haven't used rdiff-backup for a couple of years and I never did have
> it integrated into a nightly cron script for production use.
>
> My needs have changed and I want to give it another shot.
>
> I'm backing up to a local directory on a dedicated backup disk so it is
> easy enough to add "rdiff-backup /src /backup" to my backup script.
>
> But what about catching errors?
>
> Seems like I should be sending output to a log file, grepping thru it
> and e-mailing it to myself if anything goes wrong.
>
> I've looked at the examples at
> http://www.nongnu.org/rdiff-backup/examples.html, but none of them seem
> to address this common need.
>
> Thanks
> Greg
> --
> Greg Freemyer
> The Norcross Group
> Forensics for the 21st Century
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> rdiff-backup-users mailing list at address@hidden
> http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

From:	Greg Freemyer
Subject:	Re: [rdiff-backup-users] Simple nightly script?
Date:	Fri, 20 Oct 2006 15:01:39 -0400