Friday, June 19, 2009

syslog-ng pipelines

The other day someone wanted a special syslog-ng macro that would expand into digit changing every 5 seconds (e.g. R_UNIXTIME % 5) and although I couldn't give an exact solution to his problem, I've came up with this configuration snippet:

rewrite p_date_to_values {
set("$R_DATE", value("rdate"));
};

filter f_get_second_chunk {
match('^... .. [0-9]+:[0-9]+:(?<rdate.second_tens>[0-9])[0-9]$'
type(pcre) value('rdate'));
};

The way it works is as follows:
  • the rewrite statement sets the name-value pair named "rdate" to $R_DATE (the macro)
  • the filter statement uses Perl Compatible Regular Expressions to parse the value of the "rdate" value and uses a named subpattern on the tens of seconds position to store that character in a value named "rdate.second_tens"
  • Later on in the configuration you can use "rdate.second_tens" just like any other macro/value.
This proves that the current rewrite/parser/filter subsystems are really powerful, however even though this proved to be possible, there are some lessons learned from this example:
  • the macro and name-value space should really converge to each, this would mean that the match() filter could directly match against the macro value $R_DATE without the need for the separate rewrite statement
  • when you are after a given goal, you don't really want to differentiate rewrite/parser/filter rules at all. The current syntax of using separate blocks for separate type of log processing elements is a pain.
So I'm thinking about inventing yet another block, which simply wouldn't care what kind of processing element is added to it, something along the lines:

pipeline rdateseconds {
set("$R_DATE", value("rdate"));
match('^... .. [0-9]+:[0-9]+:(?[0-9])[0-9]$'
type(pcre) value('rdate'));

};

And then:

log {
source(src);
pipeline(rdateseconds);
destination(dst);
};


Maybe I should even allow the creation of rewrite/parser/filter elements right there in the log statement:

log {
source(src);
filter(facility(mail));
destination(dst);
};


What do you think?

Wednesday, June 03, 2009

Nordic Meet on Nagios 2009

I'm sitting at NMN 2009 right now, and although the event title says it is a Nagios meet, I'm going to give a presentation on syslog-ng and the new features that 3.0 brings and an example on how to integrate syslog-ng and Nagios.

If you are here and have a question just feel free to find me in the "BalaBit" T-Shirt. :) There's also live streaming on the conference website, so you can catch me at 15:50 Central European Time.