Monitoring log events, part II (the sec way)

Monitoring log events, part II (the sec way)

On part I, I used swatch but I had to document this too for the fun of it :-)
sec is well documented here, here and here but I just wanted to show a really quick and simple setup for anyone interested in this stuff.

I will show this only for CentOS (applies of course to RHEL/OEL/SuSe/etc) at this time.

Install the 'sec' package

sudo yum install epel-release && yum install sec

This will automatically install the binaries (/usr/bin/sec), create a startup script (/etc/init.d/sec), create a sec script (.sec) specific directory /etc/sec/ with nice README with more details:

cat /etc/sec/README
This is the SEC configuration directory. Because SEC usage varies so widely
from user to user, this package is configured by default to not run.
The commented-out default settings in /etc/sysconfig/sec will load any file in
this directory with a .sec suffix. Please look through the example files
included in /usr/share/doc/sec-<version>/examples/ and install the ones you
want here (taking into account that the examples are generic and some of them
may need to be tweaked to work with your setup). You should also read the SEC
man page so you have at least a basic understanding of the SEC configuration

Quick'n'dirty setup

Setting up the first log event
sudo nano /etc/sysconfig/sec
Just to load all .sec configurations from /etc/sec directory and to use /var/log/test.log as input on all configurations (please see the /etc/sysconfig/sec how to setup multiple configurations and inputs - please be aware that the array SEC_ARGS[0]... SEC_ARGS[3] needs to be in numeric order!)
SEC_ARGS="-detach -conf=/etc/sec/*.sec -input=/var/log/test.log -log=/var/log/sec -intevents -pid=/var/run/"
Lets do the first .sec configuration
sudo nano /etc/sec/log.sec
Quick'n'dirty pattern matching
# Teemu Otala
# Recognize a string foo anywhere in a row (case insensitive)
action=shellcmd /etc/sec/ '$0'
Please notice few very important things here:
  • full path to shell script ( - relative path won't do
  • the full row is stored in $0
  • the $0 needs to be wrapped in apostrophe ' - if there are any spaces in the variable/row then it would be chopped to $1$2$3 and so on on the receiving end (BTW $0 would be the script name)
  • There can't be any comments in-between the configuration rows... This is really strange behaviour. A feature, not a bug - right :-) 
The executable
sudo nano /etc/sec/
Just a quick test
# Print the row to another file or whatever you want to do with it
echo "bar: $1" >> /var/log/test2.log
Make the script executable
chmod +x /etc/sec/
And light up the grill
sudo service sec restart

Testing it

On terminal #1
sudo tailf /var/log/test2.log
On terminal #2
sudo echo "lorem ipsum foo bar me" >> /var/log/test.log
(and maybe on terminal #3 to see the insights of sec)
sudo tailf /var/log/sec

Additional stuff

You can use whatever language on the executable.
This is just a plain example how to utilize PHP for this (sudo yum install php-cli). No flames please, PHP is a very handy language that works like charm for simple tasks like this.

The PHP script version of the .sec configuration file
# Teemu Otala
# Recognize a string foo anywhere in a row (case insensitive)
action=shellcmd /etc/sec/log.php '$0'

The PHP executable (sudo nano /etc/sec/log.php). This is file_append so the file needs to exist.
    // $argv[0] is '/path/to/this/script.php'
    $argument1 = $argv[1];
    file_put_contents("/var/log/test2.log", "bar: ".$argument1."\n", FILE_APPEND);
Make the script executable and restart sec (sudo service sec restart) 
chmod +x /etc/sec/log.php

... Rememeber to restart/reload the sec daemon after each modification on /etc/sec/*.sec files or /etc/sysconfig/sec for the changes to take effect