I have been having issues I can’t narrow down that have been causing Apache to go into a conniption and the websites on my server to go down. The quick duct tape fix has been to login ot the server and restart Apache. This is a manual process though and the site could be down for a while before I get in to do the restart.

There are tons of website monitoring services out there, some free, that can alert you to such a failure. The “free” ones I have found though don’t allow you to do more frequent checking without upgrading to a paid plan. Furthermore, most are only “alert” services and cannot actually intervene and do anything.

I thought there must be a way to do this via a script and a cron job. After some digging around I wasn’t able to find a simple script that would do exactly what I wanted. However I found enough bits and pieces to cobble this together and after some testing it seems to work well.

#!/bin/bash
DATE=$(date)
/usr/bin/curl -s --head  --request GET www.kiloroot.com | if ! grep "200 OK"; then
service apache2 restart
echo "$DATE - NOT OKAY, apache restarted" >> /var/log/apache2/custom-restarts.log
fi

Basically the script runs the curl program against one of my sites. If part of the response to curl doesn’t include a “200 OK” message (which means that Apache has served the page successfully) then the script writes a log message with the date and restarts Apache. I then setup a root cron job to run this script every 5 minutes.

This is just a fancier form of automated duct tape… I realize that. Ultimately I need to get to the root of the problem and stabilize Apache. However it will provide me some peace of mind in the mean time. It should also give me a bit of forensic data as heretofore I really wasn’t sure how often my site was going down. After a week or so I should have a log of every time this script has had to restart Apache due to the site not responding.

Hopefully other folks will find this simple solution useful 🙂

BTW – Suggestions for improvement on this are always welcome! I think one could do a fair bit of automated monitoring and intervention with a more involved script.

References:

http://stackoverflow.com/questions/17522360/how-do-i-write-a-shell-script-for-checking-website-live-or-not
http://stackoverflow.com/questions/10552711/how-to-make-if-not-true-condition
http://stackoverflow.com/questions/6207573/how-to-append-output-to-the-end-of-text-file-in-shell-script
http://www.cyberciti.biz/faq/unix-linux-getting-current-date-in-bash-ksh-shell-script/
http://www.thegeekstuff.com/2010/06/bash-if-statement-examples/
http://www.thegeekstuff.com/2011/07/cron-every-5-minutes/
https://help.ubuntu.com/community/CronHowto

1 of 1

One comment on: Simple Script + Cron Job for Website Health Check + Apache Restart

  1. Lyubo
    Reply

    Nice topic!
    Here is my version. Suspecting that apache service may not be able to log any 500 messages while down, I am checking if it is active. I had to add multiple ‘state’ variables, because once invoked at the first line it remains “inactive” during the whole script execution, wherever I call it in the code. If inactive it will try to start it first, if still inactive ( second state variable ) then it will try restarting it.

    STATE=`systemctl status apache2 | grep Active | awk ‘{print $2}’`
    DATE=`date | awk ‘{print $2″ “$3” “$4}’`
    DAEM=”[web server health check]”
    if [[ $STATE != “active” ]]; then
    echo “apache state is $STATE, trying to start it” | mail -s “apache_state alert” lyubomir_minchev.bg;
    echo “$DATE $DAEM Apache service is $STATE attempting to start it” >> /var/log/apache2/healthcheck.log
    systemctl start apache2
    sleep 6
    STATE1=`systemctl status apache2 | grep Active | awk ‘{print $2}’`
    if [[ “$STATE1” != “active” ]]; then
    echo “apache state is still $STATE1 after starting, trying to restart it” | mail -s “apache_state alert” lyubomir_minchev.bg;
    echo “$DATE $DAEM apache state is still $STATE1 after starting, trying to restart it” >> /var/log/apache2/healthcheck.log
    systemctl restart apache2
    sleep 6
    STATE2=`systemctl status apache2 | grep Active | awk ‘{print $2}’`
    if [[ $STATE2 == “acive” ]];then
    echo “apache state is $STATE2 and it was successfully restarted” | mail -s “apache_state alert” lyubomir_minchev.bg;
    echo “$DATE $DAEM apache state is $STATE2 and it was successfully restarted” >> /var/log/apache2/healthcheck.log
    fi
    else
    echo “$DATE $DAEM apache state is now $STATE1, and was successfully started” >> /var/log/apache2/healthcheck.log
    fi
    else
    echo “$DATE $DAEM apache service is $STATE and it looks good” >> /var/log/apache2/healthcheck.log
    fi

Leave a Reply