Friday, June 19, 2009

syslog-ng pipelines

The other day someone wanted a special syslog-ng macro that would expand into digit changing every 5 seconds (e.g. R_UNIXTIME % 5) and although I couldn't give an exact solution to his problem, I've came up with this configuration snippet:

rewrite p_date_to_values {
set("$R_DATE", value("rdate"));
};

filter f_get_second_chunk {
match('^... .. [0-9]+:[0-9]+:(?<rdate.second_tens>[0-9])[0-9]$'
type(pcre) value('rdate'));
};

The way it works is as follows:
  • the rewrite statement sets the name-value pair named "rdate" to $R_DATE (the macro)
  • the filter statement uses Perl Compatible Regular Expressions to parse the value of the "rdate" value and uses a named subpattern on the tens of seconds position to store that character in a value named "rdate.second_tens"
  • Later on in the configuration you can use "rdate.second_tens" just like any other macro/value.
This proves that the current rewrite/parser/filter subsystems are really powerful, however even though this proved to be possible, there are some lessons learned from this example:
  • the macro and name-value space should really converge to each, this would mean that the match() filter could directly match against the macro value $R_DATE without the need for the separate rewrite statement
  • when you are after a given goal, you don't really want to differentiate rewrite/parser/filter rules at all. The current syntax of using separate blocks for separate type of log processing elements is a pain.
So I'm thinking about inventing yet another block, which simply wouldn't care what kind of processing element is added to it, something along the lines:

pipeline rdateseconds {
set("$R_DATE", value("rdate"));
match('^... .. [0-9]+:[0-9]+:(?[0-9])[0-9]$'
type(pcre) value('rdate'));

};

And then:

log {
source(src);
pipeline(rdateseconds);
destination(dst);
};


Maybe I should even allow the creation of rewrite/parser/filter elements right there in the log statement:

log {
source(src);
filter(facility(mail));
destination(dst);
};


What do you think?

2 comments:

Gyöngyösi Péter said...

The part in the regexp where you assign the name "second_tens" did not get HTML-escaped properly so it's not visible right now, which made it a bit hard to understand at first :)

On the other hand, I like the idea. But if you let filter rules be written inline, you should also let source and destination definitions be specified the same way to keep things consistent.

Bazsi said...

Fixed the HTML escaping, thanks for noticing that.