Bazsi's Home Automation blog

In one of my previous posts , I've mentioned that syslog-ng is not for syslog anymore, we aim to support other log formats too, preferably those that have some kind of structure. In fact syslog-ng is trying to convert all incoming messages (be them unstructured syslog messages, process accounting messages or SNMP traps) into the same, common format: name-value pairs tag or tags that connects the event to one of the patterndb schemas This information coming in from different sources can be stored and processed with the same infrastructure. Correllation between SNMP traps and syslog messages or netflow records should be possible. I probably don't need to mention, that we use patterndb to extract information from syslog messages. But structured information sources contain name-value pairs in the first place, so why not use them natively? This is what the experimental process accounting feature of syslog-ng demonstrates. With this module, syslog-ng is able to read the process accou...

I thought I'd post a quick update on the patterndb project status. Our first aim was to draft a basic policy which governs how patterns should be created. This is available in the patterndb git repository as a README.txt file. Although not completely finished, I feel the current description is enough for some basic work to start, to gather more experience. Here is the current version: http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=README.txt;hb=HEAD Also, after discussing the policy we've set a target to cover login/logout events from all parts of a generic Linux system. Currently sshd is quite nicely covered, su is coming along and I still have some submitted log samples that need marking up. With the sshd/su patterns a quite nice percentage of my "auth.log" file is covered and using pdbtool " grep on steroids " feature, the marked up patterns are already quite useful. Further log samples and a hand in helping me out to mark up the patterns...

You may have heard of my last project to collect log samples from various applications, in order to convert log data from free-form human readable strings into structured information. The first round to collect login/logout messages from sshd is now complete. You could ask: ok, but what is the immediate benefit? You supposedly have a lot of unprocessed log files, and syslog-ng's db-parser() has not been used to process them, thus they are stored as good-old plain text files. I spent a couple of hours to add a "grep"-like functionality to pdbtool which makes it easy to process already existing log files, giving you immediate benefit for each and every sample added to patterndb. For example, if you are interested in login failure events, you could say: zcat logfile.gz | pdbtool match -p access/sshd.pdb \ --file - \ --filter 'tags("usracct") and match('REJECT' type(string) value("secevt.verdict"));' \ --template '${usracct...

Starting with syslog-ng OSE 3.2, syslog-ng became plugin based, which has some consequences that even experienced syslog-ng users may stumble into. The most obvious one, is that syslog-ng now produces a series of .so files loaded at runtime, instead of being a monolithic executable. If a given .so is not not or not loaded, some of the functionality may be missing. This usually manifests itself by a syntax error when parsing the configuration file. Second: if you compile syslog-ng from source, the unit/functional test programs also want to load plugins, and they try to do that from the install directory. This means that you first have to install syslog-ng using "make install" before running the testsuite. This is not an ideal solution, but should work for now. Plugins are loaded from $prefix/lib/syslog-ng by default, however this can be changed using the `module-path` global, which contains the list of directories where syslog-ng should look for modules. You can change this us...

syslog-ng has been around for about 12 years now, but I think the biggest change in the project's life is imminent: with the upcoming release of syslog-ng OSE 3.2, syslog-ng will become an independent entity. Until now, syslog-ng was primarily maintained & developed by BalaBit, copyrights needed to be reassigned in order to grant BalaBit special privileges. BalaBit used her privileges to create a dual-licensed fork of syslog-ng, named "syslog-ng Premium Edition". The value we offer over the Open Source Edition of syslog-ng are things that larger enterprises require: support on a large number of UNIX platforms (27 as of 3.1), smaller and larger feature differences (like the encrypted/digitally signed logfile feature) better test coverage and release management longer term support Although perfectly legal, this business model was not welcome in various Free Software communities, and has caused friction and harm, because BalaBit has enjoyed a privilege that no others cou...