Bazsi's Home Automation blog

By now probably most of you know about patterndb, a powerful framework in syslog-ng that lets you extract structured information from log messages and perform classification at a high speed: http://www.balabit.com/dl/html/syslog-ng-ose-v3.1-guide-admin-en.html/concepts_pattern_databases.html Until now, syslog-ng offered the feature, but no release-quality patterns were produced by the syslog-ng developers. Some samples based on the logcheck database were created, but otherwise every syslog-ng user had to create her samples manually, possibly repeating work performed by others. Since this calls out to be a community project, I'm hereby starting one. Goals Create release-quality pattern databases that can simply be deployed to an existing syslog-ng installation. The goal of the patterns is to extract structured information from the free-form syslog messages, e.g. create name-value pairs based on the syslog message. Since the key factor when doing something like this is the naming of ...

I've just commited a small incompatible change for syslog-ng 3.1, even though theoreticaly I shouldn't have. The change is not big, simply the 'store-legacy-msghdr' flag became default for all sources, whereas earlier you had to specify that explicitly. In order to understand why I did that, a short description of the flag follows below. syslog-ng processes all incoming messages into fields (things like $PROGRAM and $DATE) and then reconstructs the message based on this parsed information when it has to write the message to a file. Before syslog-ng 3.0 a message was split into the macros: "$DATE $HOST $MSG", which expanded to the actual log message. "$MSG" above was expanded to a line like: "program[pid]: message" With syslog-ng 3.0 and the integrated handling of RFC5424 and RFC3164 this format was changed and an $MSGHDR macro was created for the "program[pid]: " part and I got rid of this part from $MSG. (of course if you are run...

I've just pushed a round of updates to the syslog-ng 3.2 repository, featuring some interesting stuff, such as: SQL reorganization: Patrick Hemmer sent in a patch to implement explicit transaction support instead of the previous auto-commit mode used by syslog-ng. I threw in some fixes and refactored the code somewhat. Configuration parser changes: the syntax errors produced by syslog-ng became much more user-friendly: not only the column is displayed, but also the erroneous line is printed and the error location is also highlighted. Additional plugin modules were created: afsql for the SQL destination, and afstreams for Solaris STREAMS devices. Creating a new plugin from core code takes about 15 minutes. I'm quite satisfied. With the addition of these two modules, it is now possible to use syslog-ng without any kind of runtime dependency except libc. The already existing afsocket module (providing tcp/udp sources & destinations) is compiled twice: once with and once withou...

The SQL destination in syslog-ng so far assumed that databases automatically start a new transaction for each INSERT statement that syslog-ng issues. This works fine, however there is a significant overhead of starting new transactions, with sqlite I've measured about 20 times performance increase on my development notebook and my debug build. With explicit-commits: bazsi@bzorp:~/.zwa/install/syslog-ng-ose-3.2$ loggen -x -r 1000000 -I 10 -S log average rate = 9377.28 msg/sec, count=93776, time=10.003, msg size=256, bandwidth=2344.32 kB/sec With per-statement (automatic) commits: bazsi@bzorp:~/.zwa/install/syslog-ng-ose-3.2$ loggen -x -r 1000000 -I 10 -S log average rate = 529.46 msg/sec, count=5299, time=10.083, msg size=256, bandwidth=132.36 kB/sec So this really seem to matter. In order to configure it you can use the following options in an SQL destination: flush_lines/flush_timeout controls how much messages get into the same transaction, similar to what these parameters mean f...

After last the stable syslog-ng 3.1.0 release last week, I've opened the 3.2 branch to receive the new stuff. The first bits are already in the repository: the basic plugin framework and the conversion of the socket related stuff (tcp, udp, unix-dgram, unix-stream, syslog drivers) into a separate plugin. The reason of the afsocket plugin conversion is to help moving the OpenSSL dependency to a separate package in distributions where this dependency cannot be associated with core packages like syslog-ng. But I see a lot of potential behind the plugin framework, and I still have a lot of ideas, just check my last blog post in the topic . Also, there's an even more experimental feature in the "blocks" branch right now, it is still incomplete, the naming and the syntax is still vague. The aim is there to provide syslog-ng.conf C++ template-like functionality in order to make the configuration easier by using pre-configured config snippets. For example: block source tomca...

I'm proud to announce that both the Open Source and the Premium editions of syslog-ng 3.1 was published and are available on our website. This is an important milestone in multiple ways: the new feature/stable release schema is making its debut the patterndb got significant improvements: new parsers, pdbtool, tagging support the ability to change/add RFC5424 style structured data to messages even more supported platforms (Tru64 on alpha, HP-UX 11iv2 on Itanium and older Linux versions) the diverging developments of syslog-ng Open Source Edition, Premium Edition and syslog-ng Store Box was merged into a new base, Some interesting (ok, for us developers :) statistics follow: Premium Edition: 586 commits 200 files changed, 23479 insertions(+), 5513 deletions(-) Open Source Edition: 189 commits 115 files changed, 9020 insertions(+), 3225 deletions(-) The reason for the big difference is the merger of the currently propriatery log indexer engine used in SSB into the current Premium Edi...

Since the last post, I could hack a couple of hours on the plugins branch, which now compiles. The plugin framework is capable for supporting a quite important core functionality: all socket like sources/destinations are now found in an external plugin called "afsocket". The reason I've started with afsocket is to make syslog-ng a bit less dependant on OpenSSL. A couple of distributions didn't include syslog-ng 3.0 in their current releases, because it uses OpenSSL from /usr, while syslog-ng should remain in the root directory. By separating afsocket from the syslog-ng core, I can compile afsocket with and without TLS support, which can be put into separate packages. Thus syslog-ng can operate without OpenSSL. And the same plugin framework will enable me to create a wide variety of plugins. My ideas: Plugins for all syslog-ng components (source, destination, filter, rewrite, parser) Python scriptability (a simple correllation engine in Python?) macro transformation fu...

Things have been a little rough last couple of months, that's why I haven't posted here. I'm in a rush right now as well, but I just wanted to let you know that I have started working on modularizing syslog-ng. It is only a preliminary prototype, and as of now it doesn't compile, but the way it's going to work is already visible: each plugin will have its own plugin and with some trickery the large syslog-ng.conf parser will call out to the plugin parser. The user will recognize such a plugin as an integral part of syslog-ng. E.g. this is a sample configuration file: @version: 3.0 @module: dummy ... destination d_dummy { dummy(dummy_opt(yes)); }; ... See the dummy plugin code in my git repository, in the "plugins" branch. Please note that that branch is going to be rebased a couple of times yet, I've released it in the spirit of "release early, release often". I hope to get some of the recent contributions into plugins, instead of bloating th...

I've mentioned shortly in my previous post, but here's a more official announcement: I've released syslog-ng OSE 3.1beta2, containing some important bugfixes. The list of changes: http://www.balabit.com/downloads/syslog-ng/open-source-edition/3.1beta2/changelog-en.txt Thanks to Martin Holste for the feedback he provided, hopefully we can forget about the "beta" part soon.

You may probably know that starting with syslog-ng 3.0, we started poking into the message payload by being able to extract information from the log messages and use that information in structured form for message routing, filtering and storing them as separate fields in a database table. You may have read about patterndb on this blog or on Marci's blog and we have also given talks about it on different conferences: NNM 2009 and LSM/RMLL 2009 . The reason I'm raising the topic here again is that we have now released about 8000 patterns covering about 200 applications for patterndb and are now in the process of creating a community site to maintain this database. You can download the database from www.balabit.com . Also an important thing to know that syslog-ng OSE 3.1 features enhanced performance with regard to handling information extracted from the message payload and it also has support for the latest patterndb database format. So if you want to try the new database, fetc...

I'm proud to announce that syslog-ng OSE 3.1 has been released and uploaded to our webserver. This version is new in two ways: 1) of course it has new features, see below for the most interesting bits 2) it is a "feature release", which means that once syslog-ng 3.2 or syslog-ng 4.0 is released, the support for this release will be ceased. See our new version policy at this link: https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/roadmap.bbx Since the documentation is not yet up to date with this beta release, I'll try to include the most crucial information about the new features right here in this announcement. For those who hurry, here's a link for the source code: https://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.1beta1/source/syslog-ng_3.1beta1.tar.gz And here are the binaries for Linux/FreeBSD systems: https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/ Select the Downloads tab, and i...

One of my collegues probably felt like crazy and implemented a twitter() destination driver for syslog-ng. Although the value is dubious, I think it is the neatest contribution to syslog-ng so far. :)

Like I announced in one of my previous posts, towards the syslog-ng OSE 4.0 release I'm going to make smaller, short-term supported releases. The first of these, called syslog-ng 3.1 is nearing completion, and thus a status report is due. Here's the original plan (quoting the roadmap page here ): support tags for syslog messages: each message can be marked with one or more tags, then apply filtering based on tags patterndb: add tag support patterndb: v2 database format support patterndb: add parsers for IPv6 addresses and hex numbers converge macros in templates and name-value pairs even more (right now it is not possible to use any macro in match()) I've just pushed out another set of updates to our git repository , which: adds tag support: a new tags() filter and a tags() option for all sources and a builtin logic to assign the syslog-ng source name as a tag (in the format: .source. ) adds support for patterndb v2 and a newly introduced but compatible v3 format ...

BalaBit has grown quite a lot in the last 9 years since it was founded, these days there are about 60 employees and more than 50% of that is working in the development field (give or take a couple, I've lost count some time ago). As we currently work on 4 products, support 5-6 CPU architectures and a host of different Operating Systems, automation in development is a must. We try to automate everything and that means a lot. Some examples: preparing the development workstation for development/testing work in 15 minutes for any of our products building source code for tens of CPU/OS combinations by issuing a single command creating bundles of intermediate components when generating setup packages doing releases test automation and a host of other things Some of these solutions are completely our own development, others are derived from public projects, and as BalaBit tries hard to be a good friend of Free, Libre and Open Source Software (FLOSS) we try to contribute back to projects t...

According to the plan of my recently published syslog-ng OSE roadmap , I've worked on integrating the various patterndb related fixes/enhancements in the syslog-ng OSE 3.1 tree. This now means that OSE 3.1 is now capable of working with all the version2 style pattern databases that syslog-ng Store Box is using. Here is a link for the SSB patterns: http://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/ I still need to work on integrating the new tags framework and the integration between tags and patterndb. Once that is done, I only have one item left for the 3.1 feature release. So with some luck, we can have a new shiny syslog-ng OSE release this summer. Please note that this is not released code yet and is only available via git, however if there's demand, I'm willing to create an alpha release (with binaries) if you want to try it. Just drop me an email, or simply write a comment to this post, and I'm going to create one for you. Stay tuned.

One of my collegues (Péter Höltzl, he does all our trainings) has created a nice detailed example on how to use the parser/rewrite framework to pull in yet another application into syslog: dpkg, the Debian package manager. If you are interested in what rewrite/parser can do for you, but didn't have the time to find out, the blog post is worth a read .

The other day someone wanted a special syslog-ng macro that would expand into digit changing every 5 seconds (e.g. R_UNIXTIME % 5) and although I couldn't give an exact solution to his problem, I've came up with this configuration snippet: rewrite p_date_to_values { set("$R_DATE", value("rdate")); }; filter f_get_second_chunk { match('^... .. [0-9]+:[0-9]+:(?<rdate.second_tens>[0-9])[0-9]$' type(pcre) value('rdate')); }; The way it works is as follows: the rewrite statement sets the name-value pair named "rdate" to $R_DATE (the macro) the filter statement uses Perl Compatible Regular Expressions to parse the value of the "rdate" value and uses a named subpattern on the tens of seconds position to store that character in a value named "rdate.second_tens" Later on in the configuration you can use "rdate.second_tens" just like any other macro/value. This proves t...

I'm sitting at NMN 2009 right now, and although the event title says it is a Nagios meet, I'm going to give a presentation on syslog-ng and the new features that 3.0 brings and an example on how to integrate syslog-ng and Nagios. If you are here and have a question just feel free to find me in the "BalaBit" T-Shirt. :) There's also live streaming on the conference website , so you can catch me at 15:50 Central European Time.

I've updated the syslog-ng OSE roadmap on the syslog-ng webpage to include information about the upcoming syslog-ng version: http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/roadmap/ Also, I'd like to bring the changed release/support policy to your attention, that you can read at the same location above. I'd like to introduce stable track and feature track releases, the first being supported for a long time, whereas feature track releases are only supported until the next feature/stable release is published. When a sufficient number of features were published via feature track releases, the last one becomes stable and the cycle continues. Note that feature releases are NOT development snapshots, they are releases just like the major versions previously, the only difference is that instead of a large feature list like with syslog-ng 3.0 , only a smaller set of changes are included. This makes it possible to publish features more often, always c...

After a long time and a lot of accumulated bugfixes, I've pressed the "release" button and syslog-ng OSE 3.0.2 was published on our website. The first official version to feature binary packages for Linux and BSD platforms. Since there was a long time between 3.0.1 and 3.0.2 the changelog is quite large, however most of it are bugfixes, only some minor enhancements here and there. Hopefully I didn't miss any important bugs and problems. It must be much better stability/functionality wise than 3.0.1 was. The diffstat since 3.0.1: 150 files changed, 4332 insertions(+), 3000 deletions(-) You can also check the patches in our git repository . If you are using the 3.0 branch you are really recommended to check out this release. If you are using anything earlier than 3.0 you are also recommended upgrade, syslog-ng 3.0 is revolutionary to previous versions in many ways , especially if you want to do more to your logs than merely store them in a plain text file.