Bazsi's Home Automation blog

The other day someone wanted a special syslog-ng macro that would expand into digit changing every 5 seconds (e.g. R_UNIXTIME % 5) and although I couldn't give an exact solution to his problem, I've came up with this configuration snippet: rewrite p_date_to_values { set("$R_DATE", value("rdate")); }; filter f_get_second_chunk { match('^... .. [0-9]+:[0-9]+:(?<rdate.second_tens>[0-9])[0-9]$' type(pcre) value('rdate')); }; The way it works is as follows: the rewrite statement sets the name-value pair named "rdate" to $R_DATE (the macro) the filter statement uses Perl Compatible Regular Expressions to parse the value of the "rdate" value and uses a named subpattern on the tens of seconds position to store that character in a value named "rdate.second_tens" Later on in the configuration you can use "rdate.second_tens" just like any other macro/value. This proves t...

I'm sitting at NMN 2009 right now, and although the event title says it is a Nagios meet, I'm going to give a presentation on syslog-ng and the new features that 3.0 brings and an example on how to integrate syslog-ng and Nagios. If you are here and have a question just feel free to find me in the "BalaBit" T-Shirt. :) There's also live streaming on the conference website , so you can catch me at 15:50 Central European Time.

I've updated the syslog-ng OSE roadmap on the syslog-ng webpage to include information about the upcoming syslog-ng version: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/roadmap/ Also, I'd like to bring the changed release/support policy to your attention, that you can read at the same location above. I'd like to introduce stable track and feature track releases, the first being supported for a long time, whereas feature track releases are only supported until the next feature/stable release is published. When a sufficient number of features were published via feature track releases, the last one becomes stable and the cycle continues. Note that feature releases are NOT development snapshots, they are releases just like the major versions previously, the only difference is that instead of a large feature list like with syslog-ng 3.0 , only a smaller set of changes are included. This makes it possible to publish features more often, always c...

After a long time and a lot of accumulated bugfixes, I've pressed the "release" button and syslog-ng OSE 3.0.2 was published on our website. The first official version to feature binary packages for Linux and BSD platforms. Since there was a long time between 3.0.1 and 3.0.2 the changelog is quite large, however most of it are bugfixes, only some minor enhancements here and there. Hopefully I didn't miss any important bugs and problems. It must be much better stability/functionality wise than 3.0.1 was. The diffstat since 3.0.1: 150 files changed, 4332 insertions(+), 3000 deletions(-) You can also check the patches in our git repository . If you are using the 3.0 branch you are really recommended to check out this release. If you are using anything earlier than 3.0 you are also recommended upgrade, syslog-ng 3.0 is revolutionary to previous versions in many ways , especially if you want to do more to your logs than merely store them in a plain text file.

I've uploaded my OSDC 2009 presentation slides to http://people.balabit.hu/bazsi/slides/osdc-2009-syslog-ng-3.0.odp Which has an example for processing iptables logs with db-parser() and putting the results in a customized SQL table.

I'm going to give a talk on syslog-ng on the upcoming Nordic Nagios Meet 2009. I expect the event to be great fun, just like last year . If you are in the Nordic region and use Nagios, rrdtools or syslog-ng, I recommend to pay a visit as you can meet the primary authors and some active contributors to these projects. If you are there and have anything to ask/talk about syslog-ng, just feel free to approach me, I'm probably going to wear a badge, so you can recognize me :)

I've spent the last week in the nice city of Nuremberg where Open Source Data Center Conference took place, organized by Netways AG . I really liked the talks about Puppet , DRBD and the description of the booking.com infrastructure which runs MySQL. Although I really enjoyed the conference I also had some free time to improve the automatic test program for syslog-ng, which now also covers TLS encrypted source and SQL destinations. I've also implemented a small script to collect coverage data of the testcases, thus right now I know that about 63% of syslog-ng is covered by automatic tests. (initially it was 55% but there were some low hanging fruits). I expect to raise this number easily to around 80%, then it'll probably become much more difficult to increase it further as the rest is error processing paths, and unless I come up with something to inject errors from the testcases those are difficult to test. Of course having a test suite is not a replacement for real-lif...

From Dani-aprilis-25 The reason I was absent from this blog in the couple of last weeks is my now 7 weeks old son, Dániel. You can find a picture of him right here in the post, but some additional ones in my Picasa albums .

I was long pondering with the problem that it is quite tricky to enter regexps into syslog-ng configuration file, since if you enclose the string in double quotes (e.g. in ""), the backslash character needs to be escaped. Since backslash is used in regexps quite often, it can become cumbersome to enter regexps like: match("[a-z\\-]+"); Note that the backslash is doubled because otherwise the syslog-ng string parser would pass the sequence to the regexps compiler as: "[a-z-]+" which is certainly different in meaning what the above expression says. I always remembered that syslog-ng also supports single quotes (aka apostrophes), but I remembered they behaved just as if you used normal quotation marks. Therefore I was thinking about a 3rd string format, one that would not require escaping. However I was reading the related code the other day, and found that apostrophes work exactly the way I planned this 3rd string syntax to behave: not to get in the way when...

After about two weeks being late, my son was born yesterday evening at 22:45CET. He weights 3270g and 56cm. Both the mother and the child are fine and I'm a proud new father. I guess this starts a section in my life, hopefully for the better.

I' happy to announce that BalaBit has decided to make the binary packages for syslog-ng OSE available for free. As you may know, BalaBit has various syslog-ng support packages and as a part of this service it prepared binary installation packages for different platforms. The access to these packages either required a support contract but could also be purchased separately for a yearly fee. With syslog-ng 3.0, the binary packages for syslog-ng OSE will become freely accessible. Since syslog-ng is an open source project, BalaBit planned to finish this task in the Open Source spirit: open and visible to all community members. This also means that the set of packages published with this e-mail is NOT yet release grade, rather it is more of a development snapshot of the current state of affairs. So please don't ruin your production systems with this package, it is more advisable to try them in a test environment (chroot or a dedicated test machine). With all these said, here is the ...

I'm happy as I've received the first question about the new IETF specified syslog-protocol support. There's a need for that after all :)

I didn't realize it is already that time of the year, but I was reminded that I'm going to give a talk on syslog-ng 3.0 on Open Source Data Center conference in Nürnberg, Germany at the end of April. I'm going to talk about the nifty new features of syslog-ng 3.0. It would be very nice to meet syslog-ng users there. :)

As promised on the mailing list here comes a short description of the new db-parser functionality of syslog-ng. For an introduction to parsers in general see my previous blog post here . The aim for db-parser is two-fold: extract interesting information from a log message attach tags to a log message for later classification. For instance here's a log sample (lines broken for readability): Feb 24 11:55:22 bzorp sshd[4376]: Accepted password for bazsi \ from 10.50.0.247 port 42156 ssh2 This message states that a user named "bazsi" has logged into the host named "bzorp" using SSH2 from the quoted IP and port. When you read this message as a human, the event that happened is perfectly clear. However if it is not a human, but a piece of software that has to make out the meaning of the message, you need to identify the event (e.g. that a user login has happened) and the additional information associated with the event (e.g. that he used 10.50.0.247 as the cl...

If you use GLib on non-Linux platforms such as AIX and think that G_STATIC_MUTEX_INIT does nothing but zero-initialize the mutex, think twice. Although quite clearly stated in the documentation , I thought I was smarter and used a GStaticMutex embedded in a structure that was zero initialized. If you look at the definition of G_STATIC_MUTEX_INIT on most platforms (Linux, Solaris, BSDs), it contains nothing but zeroes. This lead me to the impression that zero filling a GStaticMutex instance is enough to initialize it. In reality it isn't. On AIX this renders the mutex to be entirely useless without warnings or aborts. The results are of course bugs that are difficult to track down and fix. This took me an entire day to figure out, as the SQL driver in syslog-ng had this problem. This was fixed since, but if you are running syslog-ng on AIX with the SQL driver, be sure to have this patch applied .

Finally I could take the time to actually announce the freshly released syslog-ng OSE 3.0 branch. It was uploaded to our website during the winter holidays, but I had to integrate syslog-ng OSE to our new release infrastructure, which among others has a much nicer web interface . Here is a summary on what is new in syslog-ng 3.0: http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch01s04.html Enjoy!

I've implemented basic include file/directory functionality in syslog-ng, using the format numbered second in my previous post . I've now pushed an expermental implementation of include files in the syslog-ng OSE 3.0 repository, in a separate branch called 'include'. E.g. in order to test the code, please clone the syslog-ng 3.0 repository: $ git clone git://git.balabit.hu/bazsi/syslog-ng-3.0.git Then check out the 'include' branch: $ git checkout --track -b include origin/include Then compile as usual. I didn't want to integrate it right into syslog-ng OSE 3.0 tree as I'd like to release that first as 3.0.1.

I'm about to implement configuration file includes, and although the implementation is quite straightforward, the syntax to be used is something to give a thought or two. Currently the syslog-ng configuration file consists of statements, each with the following basic format: stmt [ ] { ... }; The "id" gives a unique identifier of the statement, and the braces enclose the contents. Currently only the ID part is optional, the braces are always there. To make the include statement consistent with that, it'd have to look something like: include { "filename" }; Obviously I don't like this too much, as it is way different from all other applications permitting the use of include statements. What about this: include "filename"; E.g. use the ID part the name of the file to be included. I like this better. A third option might be the use of 'pragma' directives, currently only used to specify the file format compatibility in the case of syslog-ng...

Last time I've written about how syslog-ng is able to change message contents. I thought it'd be useful to give you a more practical example, instead of a generic description. It is quite common to convert SNMP traps to syslog messages. The easiest implementation is to run snmptrapd and have it create a log message based on the trap. There's a small issue though: snmptrapd uses the UNIX syslog() API, and as such it is not able to propagate the originating host of the SNMP trap to the hostname portion of the syslog message. This means that all traps are logged as messages coming from the host running snmptrapd, and the hostname information is part of the message payload. Of course it'd be much easier to process syslog messages, if this were not the case. A solution would be to patch snmptrapd to send complete syslog frames, but that would require changing snmptrapd source. The alternative is to use the new parse and rewrite features of syslog-ng 3.0. First, you need to f...

For a long time I meant to give the "log statistics" feature of syslog-ng an overhaul, and finally with the advent of syslog-ng 3.0, this was done. I'm not sure all of you know, but even earlier syslog-ng versions (2.1 and 2.0) did collect some per-source and per-destination statistics. These were reported periodically in the system log. The problem with this approach that it didn't really scale: with a large configuration the statistics message could become kilobytes long, and parsing this information from a file possibly several gigabytes in size is daunting. syslog-ng 3.0 has two important changes in this area: it adds several new kinds of counters (like per-host counters), and a UNIX domain socket where you can query the current status of these counters. As counters certainly have an overhead, you can now control how much statistics you want to gather. The new stats_level() option has three levels for now: stats_level(0) is basically the same as earlier syslog-ng ...