Rebuilding Outkept

Outkept DashboardLast November I did a talk at Codebits about a private project that ended to be Outkept, it was basically a project that was build for a  specific use case and then I generalized it.

At the time I developed it in Java, but after the talk I decided to rebuild it for more generalist use cases and start everything entirely from scratch in node.js.

Outkept allows you to control and monitor a large server cluster, without needing to manually specify which and what servers to monitor/control. Instead you just define which subnets you want to control and then, using SSH, Outkept‘s crawlers will look for servers and what do they support.

Rebuilding it in node.js was awesome, allowing me to tackle my node.js skills and dig more into node’s scene while using a lot ‘earlier adopter’ tech.

Right now Outkept v2 supports everything the old version supported and even more, things are quicker and more fluid. New dashboard connects using shoe (sockjs) and the new system relies entirely in multiple node.js processes.

I will talk more about this project later, but right now I would love some feedback. The codebase is big and mostly uncommented, in the next posts will fix this and talk a bit about it.

If you want to give it a try just follow these instructions. If you need help you can reach me at petermdias@gmail.com

Advertisements

cPanel + logster + graphite

cPanel servers use one accesslog per virtualhost, which they call domlogs.

This can be problematic if you want to grab metrics from your logs. Tools like logster (metrics), beaver (log collection), … are usually designed to tap one single log file.

Some months ago I rewrote beaver, implemented daemon mode, multi-file and threading. At the time it seemed a good idea, but multi-file and daemon mode is just a resource hog when you have a server with ~300 vhosts.
From now on I follow Etsy advice which is, stay with simple and cron based log processing tools.

If you just want metrics, logster is the way to go you just have to find a way to parse all domlogs.

Below is a small bash script, that finds all the access logs and push their metrics to graphite using logster.

#!/bin/bash

for file in `find /usr/local/apache/domlogs/ -type f ! -name *.gz ! -name *-ftp_log ! -name *bytes_log ! -name *.offset* ! -name *-ssl*`
do
    aux=${file//./_}
    domain=${aux##*/}
    hostname=`/bin/hostname`
    hostname=${hostname//./_}

    logster -p servers.$hostname.$domain --output=graphite --graphite-host=XXX.XXX.XXX.XXX:2003 SampleLogster $file
done

Graphite

You just have to run this script in a cronjob, cron interval will depend on how many vhosts you have and how much delay you consider acceptable in your metrics.
In graphite this will create a folder for each server, then there will be a sub-folder for each virtualhost. All this is automatically created.

I will not cover graphite in this post, installing logster in a cPanel however may require some work.

The easiest way is to install logcheck from the epel repo, do not enable this repo globally.

The problem is, to install logcheck you need some perl dependencies, which cPanel will not allow you to install them.

exclude=apache* bind-chroot courier* dovecot* exim* filesystem httpd* mod_ssl* mydns* mysql* nsd* perl* php* proftpd* pure-ftpd* ruby* spamassassin* squirrelmail*

To install them you need to edit /etc/yum.conf and remove the “perl*” exclusion, install logtail and then enable the exclusion again.