cPanel + logster + graphite

cPanel servers use one accesslog per virtualhost, which they call domlogs.

This can be problematic if you want to grab metrics from your logs. Tools like logster (metrics), beaver (log collection), … are usually designed to tap one single log file.

Some months ago I rewrote beaver, implemented daemon mode, multi-file and threading. At the time it seemed a good idea, but multi-file and daemon mode is just a resource hog when you have a server with ~300 vhosts.
From now on I follow Etsy advice which is, stay with simple and cron based log processing tools.

If you just want metrics, logster is the way to go you just have to find a way to parse all domlogs.

Below is a small bash script, that finds all the access logs and push their metrics to graphite using logster.

#!/bin/bash

for file in `find /usr/local/apache/domlogs/ -type f ! -name *.gz ! -name *-ftp_log ! -name *bytes_log ! -name *.offset* ! -name *-ssl*`
do
    aux=${file//./_}
    domain=${aux##*/}
    hostname=`/bin/hostname`
    hostname=${hostname//./_}

    logster -p servers.$hostname.$domain --output=graphite --graphite-host=XXX.XXX.XXX.XXX:2003 SampleLogster $file
done

Graphite

You just have to run this script in a cronjob, cron interval will depend on how many vhosts you have and how much delay you consider acceptable in your metrics.
In graphite this will create a folder for each server, then there will be a sub-folder for each virtualhost. All this is automatically created.

I will not cover graphite in this post, installing logster in a cPanel however may require some work.

The easiest way is to install logcheck from the epel repo, do not enable this repo globally.

The problem is, to install logcheck you need some perl dependencies, which cPanel will not allow you to install them.

exclude=apache* bind-chroot courier* dovecot* exim* filesystem httpd* mod_ssl* mydns* mysql* nsd* perl* php* proftpd* pure-ftpd* ruby* spamassassin* squirrelmail*

To install them you need to edit /etc/yum.conf and remove the “perl*” exclusion, install logtail and then enable the exclusion again.