cron woes – file timsetamp monitoring plugin for munin

I don’t like being late. It runs in the family. My dad is so obsessed with being late that he’s always early. How embarrassing. Unfortunately I seemed to have inherited it from him. I do however try to compensate. I am deliberately late on-time. I usually plan to be a bit late.

Anyway, enough about me.

cron cron cron, the mighty scheduler. Making sure things are running on time, but does it? Well, sort of. My cron (or more precisely anacron) has failed me. It appears like a logrotate cron.daily job was stuck due to some random bug in the lighttpd postrotate script (see here). Causing anacron to lock, giving messages like:
Job `cron.weekly' locked by another anacron - skipping

I don’t get it why if one script in the /etc/cron.* directory fails, or hangs, anacron doesn’t somehow detect it and moves on. I tried searching a bit, but couldn’t find anything. Any ideas or suggestions are welcome.

I then decided to be a little more careful, and monitor those cron jobs myself. The most important jobs are those running rsnapshot to run regular backups. I was looking for a simple way to check a file/folder modified timestamp and alert me if it reaches certain threshold, i.e. it’s older than X days.

I am already using munin, for monitoring cpu, network, diskspace etc and it has a CRITICAL/WARNING email notification built-in, so I thought it might be a good tool to use for this purpose too. The graphs aren’t going to be particularly useful, but the alerting functionality is.

It’s my first munin plugin, and one of a few python scripts I’ve written, so hope you can be forgiving for lack of style or technique

#!/usr/bin/env python

import os
import time
import datetime
import sys


# a python dictionary of file/folders and the threshold to
# issue a warning for each. You can add a cron job
# to 'touch' a file and check it, or use any file you expect to be up-to-date.
# this list covers only the rsnapshot backups.
# You can see the different threshold for each folder based on the i
# expected update frequency.
# 1 day for daily, 7 days for weekly, 31 for monthly

fileList = {"/var/cache/rsnapshot/daily.0": 1,
            "/var/cache/rsnapshot/weekly.0": 7,
            "/var/cache/rsnapshot/monthly.0": 31}


if len(sys.argv) == 2 and sys.argv[1] == "autoconf":
    print "yes"
elif len(sys.argv) == 2 and sys.argv[1] == "config":
    print   'graph_title File Modified Date'
    print   'graph_vlabel Days Since Modified'
    print   'graph_category disk'
    print   'graph_scale no'
    print   'graph_info Monitors file/folder last modified date and shows ' \
            'how many days since modified. Also supports a warning if a ' \
            'specified number of days exceeds the threshold and a ' \
            'critical warning if file/folder does not exist'

    for item in fileList:
        muninItem = item.replace(".", "_")
        muninItem2 = muninItem.replace("/", "_")
        print '%s.label %s' % (muninItem2, muninItem)
        print '%s.warning %s' % (muninItem2, fileList[item])
        print '%s.critical 99998' % (muninItem2)
else:

    for item in fileList:
        muninItem = item.replace(".", "_")
        muninItem2 = muninItem.replace("/", "_")
        try:
            itemTime = time.localtime(os.path.getmtime(item))
            itemDTime = datetime.datetime(*itemTime[0:5])
            now = time.localtime()
            nowDTime = datetime.datetime(*now[0:5])
            print "%s.value %s" % (muninItem2, (nowDTime - itemDTime).days)
        except OSError:
            # if the file doesn't exist it returns 99999
            print "%s.value 99999" % (muninItem2)