Thursday, July 29, 2010

syslog-ng and process accounting

In one of my previous posts, I've mentioned that syslog-ng is not for syslog anymore, we aim to support other log formats too, preferably those that have some kind of structure.

In fact syslog-ng is trying to convert all incoming messages (be them unstructured syslog messages, process accounting messages or SNMP traps) into the same, common format:
This information coming in from different sources can be stored and processed with the same infrastructure. Correllation between SNMP traps and syslog messages or netflow records should be possible.

I probably don't need to mention, that we use patterndb to extract information from syslog messages. But structured information sources contain name-value pairs in the first place, so why not use them natively?

This is what the experimental process accounting feature of syslog-ng demonstrates. With this module, syslog-ng is able to read the process accounting file produced by the Linux kernel directly (this is currently Linux-only, but should be easy to port to other platforms) and produce a set of name-value pairs mimicing the structure of the accounting record.

This is how it works:
  • the Linux kernel writes an accounting record to /var/log/account/pacct file (distro dependant though) whenever a process terminates and writes process related information to this record (exit code, execution time, etc)
  • syslog-ng uses the file() source driver, and periodically polls this file for changes (once per second by default)
  • instead of processing this as a plain text file, the "pacctformat" plugin tells syslog-ng to fetch binary records
  • the pacctformat plugin then transforms account record members into syslog-ng name-value pairs
Each name-value pair produced by the pacct plugin has a prefix of "pacct", and the members are described in the header file or in acct(5) manual page.

In order to try this feature, you need to tell syslog-ng to compile the "pacctformat" plugin by passing the --enable-pacct command line option to configure.

Also, there's support for the pacct module in the SCL, so in order to fetch process accounting records, you only need a very small configuration file:

@version: 3.2
@include "scl.conf"

source s_pacct {
pacct();
};

log { source(s_pacct); destination(...); };

After that, you only need to enable Linux accounting by issuing an accton command.

That's all.

Monday, July 26, 2010

patterndb status update

I thought I'd post a quick update on the patterndb project status. Our first aim was to draft a basic policy which governs how patterns should be created. This is available in the patterndb git repository as a README.txt file.

Although not completely finished, I feel the current description is enough for some basic work to start, to gather more experience. Here is the current version:

http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=README.txt;hb=HEAD

Also, after discussing the policy we've set a target to cover login/logout events from all parts of a generic Linux system. Currently sshd is quite nicely covered, su is coming along and I still have some submitted log samples that need marking up.

With the sshd/su patterns a quite nice percentage of my "auth.log" file is covered and using pdbtool "grep on steroids" feature, the marked up patterns are already quite useful.

Further log samples and a hand in helping me out to mark up the patterns would be appreciated.

Tuesday, July 20, 2010

patterndb: grep on steroids

You may have heard of my last project to collect log samples from various applications, in order to convert log data from free-form human readable strings into structured information.

The first round to collect login/logout messages from sshd is now complete.

You could ask: ok, but what is the immediate benefit? You supposedly have a lot of unprocessed log files, and syslog-ng's db-parser() has not been used to process them, thus they are stored as good-old plain text files.

I spent a couple of hours to add a "grep"-like functionality to pdbtool which makes it easy to process already existing log files, giving you immediate benefit for each and every sample added to patterndb.

For example, if you are interested in login failure events, you could say:

zcat logfile.gz | pdbtool match -p access/sshd.pdb \
--file - \
--filter 'tags("usracct") and match('REJECT' type(string) value("secevt.verdict"));' \
--template '${usracct.type},${secevt.verdict},${usracct.username}\n'


What the command above does is the following:
  • reads a compressed logfile from logfile.gz
  • tells pdbtool to use access/sshd.pdb (in the patterndb git repo) as its pattern database file
  • tells pdbtool to read its stdin as a logfile, and
  • apply the db-parser() for each log message
  • apply the syslog-ng filter specified above
  • and print matching messages using the template also specified above
As a combination, it results in a CSV file, containing login failure records found in the logfile. Also please note that as long there's a pattern in the pdb file, it doesn't really matter how that originally looked like, the fact that ssh can use 3-5 different messages for the same meaning is hidden nicely under the hood.

And imagine we'd have patterns for all common applications running on our computers: this would mean that the same command above would produce login-failure reports independently from the application/OS combination being used.

Try that with grep. :)

This pdbtool is in the OSE 3.2 tree, clone the tree from: git://git.balabit.hu/bazsi/syslog-ng-3.2.git

syslog-ng OSE 3.2 caveats

Starting with syslog-ng OSE 3.2, syslog-ng became plugin based, which has some consequences that even experienced syslog-ng users may stumble into.

The most obvious one, is that syslog-ng now produces a series of .so files loaded at runtime, instead of being a monolithic executable. If a given .so is not not or not loaded, some of the functionality may be missing. This usually manifests itself by a syntax error when parsing the configuration file.

Second: if you compile syslog-ng from source, the unit/functional test programs also want to load plugins, and they try to do that from the install directory. This means that you first have to install syslog-ng using "make install" before running the testsuite. This is not an ideal solution, but should work for now.

Plugins are loaded from $prefix/lib/syslog-ng by default, however this can be changed using the `module-path` global, which contains the list of directories where syslog-ng should look for modules. You can change this using the syntax:

@define module-path `module-path`:/usr/local/lib/syslog-ng-plugins

I think that's it for now.

Wednesday, July 14, 2010

syslog-ng contributions redefined

syslog-ng has been around for about 12 years now, but I think the biggest change in the project's life is imminent: with the upcoming release of syslog-ng OSE 3.2, syslog-ng will become an independent entity.

Until now, syslog-ng was primarily maintained & developed by BalaBit, copyrights needed to be reassigned in order to grant BalaBit special privileges. BalaBit used her privileges to create a dual-licensed fork of syslog-ng, named "syslog-ng Premium Edition". The value we offer over the Open Source Edition of syslog-ng are things that larger enterprises require:
  • support on a large number of UNIX platforms (27 as of 3.1),
  • smaller and larger feature differences (like the encrypted/digitally signed logfile feature)
  • better test coverage and release management
  • longer term support
Although perfectly legal, this business model was not welcome in various Free Software communities, and has caused friction and harm, because BalaBit has enjoyed a privilege that no others could get. We plan to fix this situation and we've worked hard in the past months to make this possible.

We're letting syslog-ng go: no signed Contributory License Agreements will be required in order to contribute to syslog-ng in the future.

This is not just a matter of policy: while BalaBit wants to be a true citizen of the Free Software world, we also need to ensure the continued revenue stream that the Premium Edition provides. The adjusted business model allows us to deliver what our customers need, and at the same time make it possible for anyone else in the community to have the same privileges as BalaBit has.

The syslog-ng changes in 3.2 that I feel are most important are detailed below.

Plugins

syslog-ng was transformed from being a monolithic executable, to a core and a set of plugins.

Plugins range from source and destination drivers, filters, parsers, rewrite objects, anything can be extended with the use of a relatively simple plugin.

And this change didn't transform the configuration file format, it remained the same as before: readable and flexible. If you don't want to look under the hood, no functionality has changed, except plugins are loaded at runtime, and as a side effect, the syntax error reports are way better.

The core is licensed under the LGPL and plugins are licensed under the GPL. This legal framework allows BalaBit to deliver value to its customers in non-free plugins and syslog-ng related services.

syslog-ng Configuration Library

syslog-ng is an infrastructure element: very flexible, but sometimes intimidating. For example, having to specify the pad_size() option for an HP-UX /dev/log device explicitly makes syslog-ng flexible, but not very user-friendly.

It is common to share working configuration snippets on the mailing list or elsewhere in the syslog-ng community. We try to concentrate this effort with the creation of the syslog-ng Configuration Library (aka SCL). This library is a set of config snippets that can be included into a syslog-ng configuration file, using proper defaults but still allowing customization.

For example: we've created a source driver named "system()" which automatically expands to the local log devices as needed by the current Operating System syslog-ng runs on, but it is also possible to create application specific configuration snippets (for example apache source?) and ship it as a ready to use config block. See this post for more information.

This makes it possible to create a configuration file that will run on each of your platforms. In fact, we did that too: syslog-ng now comes with a default configuration file.

Support for non-syslog messages

Since syslog-ng is now plugin based and even the "syslog" message format as such is a plugin, it is now quite easy to add support for non-syslog message sources. As an example, we've added a plugin to parse Linux process accounting records, which makes it trivial to collect this data as well and possibly use it as a source of information when correllating data.

Future plugins like creating a MIB aware SNMP listener, or possibly processing netflow data, but I'd like to create a generic SQL source as well.

Support for these formats doesn't mean that syslog-ng would transform them into textual messages: proper name-value structure is kept and it is possible to put them into nicely structured SQL tables. Of course if all you want to transform them to text for readability, that's possible too with the use of a template().

patterndb project

Like I announced before, we're starting a parallel project to create a set of message patterns directly usable with syslog-ng's db-parser().

Current state

The changes are happening in the syslog-ng 3.2 repository at:

http://git.balabit.hu/

We have created a source-only 3.2alpha1 release of the current state. It runs our automated test harness on Linux, but of course it is not yet recommended in production environments. The release can be downloaded from www.balabit.com.

There were significant changes in how syslog-ng is compiled, thus it is expected that we have build issues on non-Linux systems. We expect to address these issues as they are found and create a 3.2beta1 release, for a longer beta testing.

Future

Parallel to fixing up the remaining issues on syslog-ng 3.2, we're going to open new ways to improve syslog-ng in the 3.3 branch. The most important item on our product backlog is to address the scalability problems and improve performance of syslog-ng.

Summary

With the upcoming changes in syslog-ng OSE 3.2 possible contributions will be greatly expanded:
  • write a pattern for your favourite application and process log data faster + easier
  • write an SCL configuration snippet, make your application easier to integrate with syslog-ng
  • write a plugin for your favourite NoSQL database,
  • write a plugin for a transformation that syslog-ng is not capable of doing right now (what about facility / severity rewrite rules? easy as a piece of cake)
  • write a plugin for things that you do with an external script
  • or contribute to the core of syslog-ng.
And all this without having to sign paperwork.