Bazsi's blog

blog moved

2010-08-12T15:23:00.003+02:00

I'm moving this blog to a wordpress instance deployed on our company webserver. The URL should be unchanged but the RSS/Atom feeds have changed, so please update your bookmarks.

Thanks and sorry for the confusion.

LWN: syslog-ng rotten to the (Open) Core?

2010-08-08T13:23:00.004+02:00

This was first posted as a comment under an article on lwn.net, but I thought it was important enough to post it here for others not reading lwn. Please go ahead and read the original article which is about the "Open Core" business model and its problems from the Free Software community point of view.

A commenter thought that syslog-ng was an example, which only exists as a marketing tool for the company's commercial offering. Anyway, here's my post:

First of all, I want to make it clear that I'm biased on the syslog-ng case, but still wanted to express my opinion here. I'm biased as I'm the primary author of syslog-ng.

I think syslog-ng is a completely different case from the one described by Neary. The GPL version is not crippleware, it was never published for marketing purposes only and for the majority of syslog-ng's existence only the Open Source stuff existed. The Premium Edition is only about 3 years old and syslog-ng started in 1998.

We never removed features from the OSE version, the Premium Edition only included _additional_ features, and a lot of those are already available in the OSE.

Some examples:
* TLS support (became available in 3.0, almost 2 years ago)
* SQL destination (became available in 2.1, 2.5 years ago)
* performance improvements (3.0)
* etc.

In the other direction, we usually receive bugfixes and it is a pure technical reason that we used to require copyright assignment: I wanted to keep the two branches as close as possible (which if not done is the reason #1 why Open Core products become crippleware fast). _And_ since we heavily invested in automatic testing and our customers report bugs directly to us, we fix way more bugs in the OSE version than the community.

But anyway, I didn't think that the dual license model was so problematic at the time we made this decision 3 years ago. Our efforts have never been "Rotten to the Open Core". If you don't believe that, check out the git repository or read the mailing list archive and see it yourself.

And this whole mess is the past, OSE 3.2 has been relicensed, and it is true that we're going to publish non-free plugins, but anyone else is welcome to join and do the same.

syslog-ng 3.2alpha2 released

2010-08-07T17:58:00.004+02:00

I've just uploaded syslog-ng 3.2alpha2 to the release directory. The last alpha release didn't compile on all supported platforms and the automatic test-suite was disabled, because it only worked if syslog-ng got installed first.

These obstacles have been overcome and together with some fixes and a couple of new features, 3.2alpha2 is now available. I've also forward ported all bugfixes from syslog-ng 3.1.2.

For those who are starting to experiment with the 3.2 branch, here's the list of new features compared to 3.1. Those who tried 3.2alpha1, the list of changes compared to 3.2alpha1 is at the end of this post.

Since the documentation of syslog-ng is not yet up-to-date with the new features introduced, I've tried to also include URLs for the best known descriptions. The references may not be 100% accurate, but should give anyone interested an idea how to start experimenting.

Also, please note that although this is an alpha release, the bulk of the changes are in the configuration parser, so once your configuration was parsed properly and syslog-ng starts up, an almost unchanged code is processing it. This means that this release should be good enough to start playing with. And feedback about what kind of syslog-ng.conf parsing errors you encounter on real-life configuration files is more than welcome.

Code quality & functionality wise, this could be a beta release, I only expect "procedural" changes, like cleaning up the plugin names, which wouldn't be nice to do in a beta release (though not unheard of :)

New features in 3.2:

Plugins: the new architecture replaces the old monolithic one, all syslog-ng functionality is loaded from external plugins when needed. It is possible to write plugins to extend syslog-ng functionality in the following areas: sources, destinations, filter expression, parsers, rewrite ops, message format.

http://bazsi.blogs.balabit.com/2010/04/syslog-ng-32-changes.html
http://bazsi.blogs.balabit.com/2010/07/syslog-ng-contributions-redefined.html

The framework for a "syslog-ng configuration library" (aka SCL) a collection of configuration snippets installed along syslog-ng, simplifying the authoring of syslog-ng configuration files.

http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=287993339599deac0442e26355c600b5aee63583
http://bazsi.blogs.balabit.com/2010/07/syslog-ng-contributions-redefined.html

pdbtool match is now able to read a file containing syslog messages and apply patterndb and a filter expression on the contents.

http://bazsi.blogs.balabit.com/2010/07/patterndb-grep-on-steroids.html

pdbtool test is now able to perform pattern testing automatically based on the supplied example log message.

http://marci.blogs.balabit.com/2010/07/pdbtool-test-and-pattern-database.html

Persistent state containing the current file position for file sources is now continously updated during runtime, instead of updating it only at exit, which makes it much more reliable in case syslog-ng doesn't terminate normally.

Better syntax error reporting in the configuration file.

http://bazsi.blogs.balabit.com/2010/04/syslog-ng-32-changes.html

Support for reusable configuration snippets, similar to macros with parameters, named "blocks".

http://bazsi.blogs.balabit.com/2010/04/syslog-ng-32-opened-experimental-blocks.html

Added a confgen plugin that includes the output of a program into the configuration file, making it possible to generate configuration file snippets dynamically.

http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=5248ef6c49ff3af0b3c896448360073606c9c7d7

Support for BSD-style process accounting logs via the pacct() source driver defined in by SCL and the underlying pacctformat plugin.

http://bazsi.blogs.balabit.com/2010/07/syslog-ng-and-process-accounting.html

Support for explicit COMMITs in the SQL driver, this speeds up SQL INSERT rate significantly if flush_lines() is non-zero.

http://bazsi.blogs.balabit.com/2010/04/explicit-transaction-support-in-sql.html

It is now possible to supply a filter to rewrite expressions and only apply the rewrite rule in case the filter matches.

https://lists.balabit.hu/pipermail/syslog-ng/2010-July/014565.html

It is now possible to use multiple parser expressions in a single parser object, similar to rewrite rules.
Added support for using the include statement from anywhere in the configuration file, instead of only at top-level. Also introduced syslog-ng "global values" that can be defined and the substituted anywhere in the configuration file.

http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=1203267c465256c99e622edf11e226301170f1c7

http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=52098762f27cde059e8b8ecda67691df85364e6d

Default configuration file supplied as part of SCL.

Incompatible changes:

syslog-ng traditionally expected an optional hostname field even when a syslog message is received on a local transport (e.g. /dev/log). However no UNIX version is known to include this field. This caused problems when the application creating the log message has a space in its program name field. This behaviour has been changed for the unix-stream/unix-dgram/pipe drivers if the config version is 3.2 and can be restored by using an explicit 'expect-hostname' flag for the specific source.

Changes since 3.2alpha1:

Now compiles on all platforms and the unit/functional tests also run. (tested: AIX, HP-UX, Solaris, FreeBSD, Linux, Tru64)
Fixed pdbtool match --debug-pattern output for ESTRING parsers.
Fixed a possible memory leak in the lexer, which would accumulate in case SIGHUPs.
Fixed Solaris STREAMS device support.
Forward ported all bugfixes from syslog-ng OSE 3.0 & 3.1
Disable process accounting module by default as it doesn't compile on non-Linux platforms.
Added "pdbtool match --file" option to read and parse an existing logfile.
Added "pdbtool test" to check the log samples in the patterndb file.
Added "dont-create-tables" flag for the SQL destination to inhibit automatic table creation.
Added "condition()" support for rewrite expressions, which makes it possible to skip rewrite rules that do not match a filter expression.
Added "--module-path" command line option to control where modules are loaded from from the command line.

Happy logging!

syslog-ng name-value pair naming

2010-08-06T20:26:00.002+02:00

I was giving a lot of thought recently to the topic of naming name-value pairs in syslog-ng. Until now the only documented rule is stating somewhat vaguely that whenever you use a parser you should choose a name that has at least one dot in it, and this dot must not be the initial character. This means that names like MSG or .SDATA.meta.sequenceId are reserved for syslog-ng, and APACHE.CLIENT_IP is reserved for users.

However things became more complex with syslog-ng OSE 3.2. Let's see what sources generate name-value pairs:

traditional macros (e.g. $DATE); these are not name-value pairs per-se, but behave much like them, except that they are read-only
syslog message fields (e.g. $MSG) if the message is coming from a syslog source
filters whenever the 'store-matches' flag is set and the regexp contains groups
rewrite rules, whenever the rewrite rule specifies a thus far unknown name-value pair, e.g. set("something" value("name-value.pair"));
and of course parsers when you tell syslog-ng to parse an input as a CSV, or use db-parser together with the patterns produced by the patterndb project

The latest stuff generating name-value pairs is the support for process accounting logs, in this case even the syslog related fields are missing and only things like "pacct.ac_comm" (to contain the program name) are defined.

So I was thinking whether it should be "pacct.ac_comm" or ".pacct.ac_comm". With the quoted rule it should be simple: it is generated by syslog-ng itself, thus it should be in the syslog-ng namespace and should start with a dot. However in the era of syslog-ng plugins, what consists of syslog-ng at all?

First, I wanted to use "pacct.ac_comm" (e.g. without a dot), because I liked this name better. I was trying to explain myself why it would not violate the rule above. The explanation I had for myself was: I'm going to "register" names such as this in the patterndb SCHEMAS.txt file. With this - not yet published - explanation, I've committed a patch to convert the pacctformat plugin to use a dotless prefix.

Next, I was figuring that it is true that process accounting creates name-value pairs without going through patternization, but I've felt, that nothing ensures that these name-value pairs would be directly usable, when trying to analyse the logs. The patterndb concept uses tags and schemas to convert the incoming unstructured data into a consistent structure. However, pacct may not completely match what the user needs. And, in the future, when SNMP traps or SQL table polling are going to be supported, it is going to be even more true: these name-value pairs may need a conversion: from the SNMP/pacct structure to the patterndb schema described structure in order to handle these message sources consistently with regular syslog (and to make it easy to correllate these).

So at the end, I've committed another patch, this time going back to ".pacct" as a prefix and leaving the original naming rule intact. The "pacct" prefix is up to the users to use, they may want the same information in a "pacct" schema, but that may come from data not directly tied from process accounting (e.g. from syslog messages).

So this post is about doing nothing with regards to the naming policy, but I thought it'd be important to shed a light behind the scenes. Giving such decisions enough thought and coming up a with a long-term plan makes our lives much easier in the future.

This post may be a bit more involved than the others, but feel free to ask me to elaborate, if you are interested.

syslog-ng & distributions

2010-08-02T16:57:00.002+02:00

syslog-ng 1.6.x and 2.0.x versions had lived quite long. A lot of distributions used these versions and never upgraded to the newer ones.

This has changed recently, Peter Czanik was busy to help maintainers get to the latest versions.

Already available in the latest release:

openSUSE
FreeBSD ports
Mandriva
Gentoo portage
OpenBSD ports

In development branches:

Debian
Ubuntu
Fedora

These all carry 3.1.1, which is quite recent (and a successful release too). There are some fixes accumulated in the git tree though, so I hope to get 3.1.2 out of the door soon.

syslog-ng and process accounting

2010-07-29T10:20:00.003+02:00

In one of my previous posts, I've mentioned that syslog-ng is not for syslog anymore, we aim to support other log formats too, preferably those that have some kind of structure.

In fact syslog-ng is trying to convert all incoming messages (be them unstructured syslog messages, process accounting messages or SNMP traps) into the same, common format:

name-value pairs
tag or tags that connects the event to one of the patterndb schemas

This information coming in from different sources can be stored and processed with the same infrastructure. Correllation between SNMP traps and syslog messages or netflow records should be possible.

I probably don't need to mention, that we use patterndb to extract information from syslog messages. But structured information sources contain name-value pairs in the first place, so why not use them natively?

This is what the experimental process accounting feature of syslog-ng demonstrates. With this module, syslog-ng is able to read the process accounting file produced by the Linux kernel directly (this is currently Linux-only, but should be easy to port to other platforms) and produce a set of name-value pairs mimicing the structure of the accounting record.

This is how it works:

the Linux kernel writes an accounting record to /var/log/account/pacct file (distro dependant though) whenever a process terminates and writes process related information to this record (exit code, execution time, etc)
syslog-ng uses the file() source driver, and periodically polls this file for changes (once per second by default)
instead of processing this as a plain text file, the "pacctformat" plugin tells syslog-ng to fetch binary records
the pacctformat plugin then transforms account record members into syslog-ng name-value pairs

Each name-value pair produced by the pacct plugin has a prefix of "pacct", and the members are described in the header file or in acct(5) manual page.

In order to try this feature, you need to tell syslog-ng to compile the "pacctformat" plugin by passing the --enable-pacct command line option to configure.

Also, there's support for the pacct module in the SCL, so in order to fetch process accounting records, you only need a very small configuration file:

@version: 3.2
@include "scl.conf"

source s_pacct {
pacct();
};

log { source(s_pacct); destination(...); };

After that, you only need to enable Linux accounting by issuing an accton command.

That's all.

patterndb status update

2010-07-26T18:13:00.002+02:00

I thought I'd post a quick update on the patterndb project status. Our first aim was to draft a basic policy which governs how patterns should be created. This is available in the patterndb git repository as a README.txt file.

Although not completely finished, I feel the current description is enough for some basic work to start, to gather more experience. Here is the current version:

http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=README.txt;hb=HEAD

Also, after discussing the policy we've set a target to cover login/logout events from all parts of a generic Linux system. Currently sshd is quite nicely covered, su is coming along and I still have some submitted log samples that need marking up.

With the sshd/su patterns a quite nice percentage of my "auth.log" file is covered and using pdbtool "grep on steroids" feature, the marked up patterns are already quite useful.

Further log samples and a hand in helping me out to mark up the patterns would be appreciated.

patterndb: grep on steroids

2010-07-20T11:45:00.004+02:00

You may have heard of my last project to collect log samples from various applications, in order to convert log data from free-form human readable strings into structured information.

The first round to collect login/logout messages from sshd is now complete.

You could ask: ok, but what is the immediate benefit? You supposedly have a lot of unprocessed log files, and syslog-ng's db-parser() has not been used to process them, thus they are stored as good-old plain text files.

I spent a couple of hours to add a "grep"-like functionality to pdbtool which makes it easy to process already existing log files, giving you immediate benefit for each and every sample added to patterndb.

For example, if you are interested in login failure events, you could say:

zcat logfile.gz | pdbtool match -p access/sshd.pdb \
--file - \
--filter 'tags("usracct") and match('REJECT' type(string) value("secevt.verdict"));' \
--template '${usracct.type},${secevt.verdict},${usracct.username}\n'

What the command above does is the following:

reads a compressed logfile from logfile.gz
tells pdbtool to use access/sshd.pdb (in the patterndb git repo) as its pattern database file
tells pdbtool to read its stdin as a logfile, and
apply the db-parser() for each log message
apply the syslog-ng filter specified above
and print matching messages using the template also specified above

As a combination, it results in a CSV file, containing login failure records found in the logfile. Also please note that as long there's a pattern in the pdb file, it doesn't really matter how that originally looked like, the fact that ssh can use 3-5 different messages for the same meaning is hidden nicely under the hood.

And imagine we'd have patterns for all common applications running on our computers: this would mean that the same command above would produce login-failure reports independently from the application/OS combination being used.

Try that with grep. :)

This pdbtool is in the OSE 3.2 tree, clone the tree from: git://git.balabit.hu/bazsi/syslog-ng-3.2.git

syslog-ng OSE 3.2 caveats

2010-07-20T11:27:00.002+02:00

Starting with syslog-ng OSE 3.2, syslog-ng became plugin based, which has some consequences that even experienced syslog-ng users may stumble into.

The most obvious one, is that syslog-ng now produces a series of .so files loaded at runtime, instead of being a monolithic executable. If a given .so is not not or not loaded, some of the functionality may be missing. This usually manifests itself by a syntax error when parsing the configuration file.

Second: if you compile syslog-ng from source, the unit/functional test programs also want to load plugins, and they try to do that from the install directory. This means that you first have to install syslog-ng using "make install" before running the testsuite. This is not an ideal solution, but should work for now.

Plugins are loaded from $prefix/lib/syslog-ng by default, however this can be changed using the `module-path` global, which contains the list of directories where syslog-ng should look for modules. You can change this using the syntax:

@define module-path `module-path`:/usr/local/lib/syslog-ng-plugins

I think that's it for now.

syslog-ng contributions redefined

2010-07-14T20:30:00.004+02:00

syslog-ng has been around for about 12 years now, but I think the biggest change in the project's life is imminent: with the upcoming release of syslog-ng OSE 3.2, syslog-ng will become an independent entity.

Until now, syslog-ng was primarily maintained & developed by BalaBit, copyrights needed to be reassigned in order to grant BalaBit special privileges. BalaBit used her privileges to create a dual-licensed fork of syslog-ng, named "syslog-ng Premium Edition". The value we offer over the Open Source Edition of syslog-ng are things that larger enterprises require:

support on a large number of UNIX platforms (27 as of 3.1),
smaller and larger feature differences (like the encrypted/digitally signed logfile feature)
better test coverage and release management
longer term support

Although perfectly legal, this business model was not welcome in various Free Software communities, and has caused friction and harm, because BalaBit has enjoyed a privilege that no others could get. We plan to fix this situation and we've worked hard in the past months to make this possible.

We're letting syslog-ng go: no signed Contributory License Agreements will be required in order to contribute to syslog-ng in the future.

This is not just a matter of policy: while BalaBit wants to be a true citizen of the Free Software world, we also need to ensure the continued revenue stream that the Premium Edition provides. The adjusted business model allows us to deliver what our customers need, and at the same time make it possible for anyone else in the community to have the same privileges as BalaBit has.

The syslog-ng changes in 3.2 that I feel are most important are detailed below.

Plugins

syslog-ng was transformed from being a monolithic executable, to a core and a set of plugins.

Plugins range from source and destination drivers, filters, parsers, rewrite objects, anything can be extended with the use of a relatively simple plugin.

And this change didn't transform the configuration file format, it remained the same as before: readable and flexible. If you don't want to look under the hood, no functionality has changed, except plugins are loaded at runtime, and as a side effect, the syntax error reports are way better.

The core is licensed under the LGPL and plugins are licensed under the GPL. This legal framework allows BalaBit to deliver value to its customers in non-free plugins and syslog-ng related services.

syslog-ng Configuration Library

syslog-ng is an infrastructure element: very flexible, but sometimes intimidating. For example, having to specify the pad_size() option for an HP-UX /dev/log device explicitly makes syslog-ng flexible, but not very user-friendly.

It is common to share working configuration snippets on the mailing list or elsewhere in the syslog-ng community. We try to concentrate this effort with the creation of the syslog-ng Configuration Library (aka SCL). This library is a set of config snippets that can be included into a syslog-ng configuration file, using proper defaults but still allowing customization.

For example: we've created a source driver named "system()" which automatically expands to the local log devices as needed by the current Operating System syslog-ng runs on, but it is also possible to create application specific configuration snippets (for example apache source?) and ship it as a ready to use config block. See this post for more information.

This makes it possible to create a configuration file that will run on each of your platforms. In fact, we did that too: syslog-ng now comes with a default configuration file.

Support for non-syslog messages

Since syslog-ng is now plugin based and even the "syslog" message format as such is a plugin, it is now quite easy to add support for non-syslog message sources. As an example, we've added a plugin to parse Linux process accounting records, which makes it trivial to collect this data as well and possibly use it as a source of information when correllating data.

Future plugins like creating a MIB aware SNMP listener, or possibly processing netflow data, but I'd like to create a generic SQL source as well.

Support for these formats doesn't mean that syslog-ng would transform them into textual messages: proper name-value structure is kept and it is possible to put them into nicely structured SQL tables. Of course if all you want to transform them to text for readability, that's possible too with the use of a template().

patterndb project

Like I announced before, we're starting a parallel project to create a set of message patterns directly usable with syslog-ng's db-parser().

Current state

The changes are happening in the syslog-ng 3.2 repository at:

http://git.balabit.hu/

We have created a source-only 3.2alpha1 release of the current state. It runs our automated test harness on Linux, but of course it is not yet recommended in production environments. The release can be downloaded from www.balabit.com.

There were significant changes in how syslog-ng is compiled, thus it is expected that we have build issues on non-Linux systems. We expect to address these issues as they are found and create a 3.2beta1 release, for a longer beta testing.

Future

Parallel to fixing up the remaining issues on syslog-ng 3.2, we're going to open new ways to improve syslog-ng in the 3.3 branch. The most important item on our product backlog is to address the scalability problems and improve performance of syslog-ng.

Summary

With the upcoming changes in syslog-ng OSE 3.2 possible contributions will be greatly expanded:

write a pattern for your favourite application and process log data faster + easier
write an SCL configuration snippet, make your application easier to integrate with syslog-ng
write a plugin for your favourite NoSQL database,
write a plugin for a transformation that syslog-ng is not capable of doing right now (what about facility / severity rewrite rules? easy as a piece of cake)
write a plugin for things that you do with an external script
or contribute to the core of syslog-ng.

And all this without having to sign paperwork.

patterndb project

2010-06-25T19:42:00.002+02:00

By now probably most of you know about patterndb, a powerful framework in syslog-ng that lets you extract structured information from log messages and perform classification at a high speed:

http://www.balabit.com/dl/html/syslog-ng-ose-v3.1-guide-admin-en.html/concepts_pattern_databases.html

Until now, syslog-ng offered the feature, but no release-quality patterns were produced by the syslog-ng developers. Some samples based on the logcheck database were created, but otherwise every syslog-ng user had to create her samples manually, possibly repeating work performed by others.

Since this calls out to be a community project, I'm hereby starting one.

Goals

Create release-quality pattern databases that can simply be deployed to an existing syslog-ng installation. The goal of the patterns is to extract structured information from the free-form syslog messages, e.g. create name-value pairs based on the syslog message.

Since the key factor when doing something like this is the naming of fields, we're going to create our generic naming guidelines that can be applied to any application in the industry.

It is not our goal to implement correllation or any other advanced form of analysis, although we feel that with the results of this project, event correllation and analysis can be performed much easier than without it.

Related projects

I know there are other efforts in the field, why not simply join them?

CEF - is the log message format for a proprietary log analysis engine, primarily meant to be used to hold IP security device logs (firewalls, IPSs, virus gateways etc). The patterndb project aims to create patterns for a wider range of device logs and be more generic in the approach. On the other hand we feel that it might be useful to create a solution for converting db-parser output to the CEF format.

CEE - Common Event Expression project by Mitre has a focus on creating a nv pair dictionary for all kinds of devices/log messages out there. Although I might be missing something, but I didn't find the concrete results so far, apart from a nicely looking white paper. If the CEE delivers something, then patterndb would probably adapt the naming/taxonomy structure. But I guess not all devices will start logging in the new shiny format, thus the existing devices would need
their logs converted, so the patterndb work wouldn't be wasted.

Infrastructure

Our original patterndb related plans were to create an easy to use web based interface for editing patterns, but since that project is progressing slowly, I'm calling for a minimalist approach: git based version control of simple plain text files. Of course once the nice web based interface is finished, we're going to be ready to use it.

First steps

I have created a git repository at:

git://git.balabit.hu/bazsi/syslog-ng-patterndb.git

This contains the initial version of the naming policy document and a simple schema for SIEM-style and a user login-logout naming schema.

If you are interested please read the file README.txt in the git archive, or if you prefer a web browser, use this link:

http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=README.txt;h=9bbfeaead0c21dcf6171e12e311ae8612f572bfc;hb=6061e22221a72d35238b35f82b04afd436341b5c

Licensing

I do not have a decision yet, but for sure this is going to use one of the open source licenses or Creative Commons. Let me know if you have a preference in this area.

Getting involved

Join the syslog-ng mailing list, a start discussing! If you have existing patterns, great. If you don't, it is not late to join.

http://lists.balabit.hu/mailman/listinfo/syslog-ng

The posting address of the mailing list (to subscribers only) is:

syslog-ng@lists.balabit.hu

small incompatible change for 3.1

2010-05-03T16:57:00.003+02:00

I've just commited a small incompatible change for syslog-ng 3.1, even though theoreticaly I shouldn't have.

The change is not big, simply the 'store-legacy-msghdr' flag became default for all sources, whereas earlier you had to specify that explicitly.

In order to understand why I did that, a short description of the flag follows below.

syslog-ng processes all incoming messages into fields (things like $PROGRAM and $DATE) and then reconstructs the message based on this parsed information when it has to write the message to a file.

Before syslog-ng 3.0 a message was split into the macros: "$DATE $HOST $MSG", which expanded to the actual log message. "$MSG" above was expanded to a line like:

"program[pid]: message"

With syslog-ng 3.0 and the integrated handling of RFC5424 and RFC3164 this format was changed and an $MSGHDR macro was created for the "program[pid]: " part and I got rid of this part from $MSG. (of course if you are running syslog-ng in compatibility mode, you get the old behaviour). The reason is simple: RFC5424 has separate fields for program/pid.

The contents of $MSGHDR is constructed programmatically, e.g. the punctuation characters '[' and ']' around the pid and the colon, is added to the format by syslog-ng, based on the available information in $PROGRAM and $PID.

However (and here comes the magic) there are programs that do not adhere to this format and omit the space after the colon character. E.g. if syslog-ng received:

"program:value"

as the syslog message, it added an explicit space character, and you'd get this in your log file:

"program: value"

NOTE the added space. This resulted in the workaround called "store-legacy-msghdr", which made syslog-ng remember the original formatting of the MSGHDR macro. However this proved to be a performance issue, thus it didn't become default, and I let my users discover this problem and add the flag explicitly if they cared about the extra space.

syslog-ng 3.1 however solves the performance issue (with the NVTable refactorization), and more and more people run into the very same issue, who are migrating from 2.1 or earlier.

Therefore I've decided to make 'store-legacy-msghdr' the default, and added a 'dont-store-legacy-msghdr' flag. My hope is that

people who cared: they already had the store-legacy-msghdr, for them, nothing is changed
people who didn't notice: they don't have the flag, but should be better of with the original formatting
people who changed their parsing scripts: well, those are who I address this message to as a HEADS up.

I hope this post makes things clearer.

syslog-ng 3.2 changes

2010-04-15T21:35:00.006+02:00

I've just pushed a round of updates to the syslog-ng 3.2 repository, featuring some interesting stuff, such as:

SQL reorganization: Patrick Hemmer sent in a patch to implement explicit transaction support instead of the previous auto-commit mode used by syslog-ng. I threw in some fixes and refactored the code somewhat.
Configuration parser changes: the syntax errors produced by syslog-ng became much more user-friendly: not only the column is displayed, but also the erroneous line is printed and the error location is also highlighted.
Additional plugin modules were created: afsql for the SQL destination, and afstreams for Solaris STREAMS devices. Creating a new plugin from core code takes about 15 minutes. I'm quite satisfied. With the addition of these two modules, it is now possible to use syslog-ng without any kind of runtime dependency except libc.
The already existing afsocket module (providing tcp/udp sources & destinations) is compiled twice: once with and once without SSL support, so it is now possible to choose which one to use at runtime.

And since a blog post is not complete without a "screenshot", here's how the new error message looks like:

Error parsing plugin unix-stream, syntax error in etc/syslog-ng-null.conf at line 3, column 34:

source s_log { unix-stream("log" default-facilitz(syslog)); };
^^^^^^^^^^^^^^^^

Neat, eh?

One more thing I need to think about is how to configure module loading. I guess it'd be less than user friendly if I'd rely on the user to load the modules that so far were the core functionality of syslog-ng.

E.g. right now if you want udp() support, you'd need something like this in the syslog-ng.conf header:

@module: afsocket

Having to remember all the module names is cumbersome and irritating. On the other hand I'd like to make it possible to run a bare-bones syslog-ng without socket support, so autoloading modules without any configurability is also out of the question.

In the current implementation syslog-ng automatically loads core modules, if the configuration version is below 3.2, and does nothing if it is 3.2 (e.g. you need an explicit @module directive for the current configuration format).

If you have an idea about how you think configuring modules should look like, just drop me an email/comment.

Explicit transaction support in SQL

2010-04-12T20:43:00.005+02:00

The SQL destination in syslog-ng so far assumed that databases automatically start a new transaction for each INSERT statement that syslog-ng issues. This works fine, however there is a significant overhead of starting new transactions, with sqlite I've measured about 20 times performance increase on my development notebook and my debug build.

With explicit-commits:
bazsi@bzorp:~/.zwa/install/syslog-ng-ose-3.2$ loggen -x -r 1000000 -I 10 -S log
average rate = 9377.28 msg/sec, count=93776, time=10.003, msg size=256, bandwidth=2344.32 kB/sec

With per-statement (automatic) commits:
bazsi@bzorp:~/.zwa/install/syslog-ng-ose-3.2$ loggen -x -r 1000000 -I 10 -S log
average rate = 529.46 msg/sec, count=5299, time=10.083, msg size=256, bandwidth=132.36 kB/sec

So this really seem to matter.

In order to configure it you can use the following options in an SQL destination:

flush_lines/flush_timeout controls how much messages get into the same transaction, similar to what these parameters mean for standard log files
flags("explicit-commits") enables explicit transaction handling

Also an option named "session_statements" was added where you can list initial SQL commands, issued right after the connection is established.

This work has been started by Patrick Hemmer (thanks again Patrick). I had to do some work on it though, since in order to avoid races the timer code of the main loop couldn't be used.

You can find all this in the syslog-ng OSE 3.2 branch. I'd love to hear success/failure stories and performance numbers you can measure with your favourite database.

syslog-ng 3.2 opened, experimental "blocks" branch opened

2010-04-05T21:23:00.006+02:00

After last the stable syslog-ng 3.1.0 release last week, I've opened the 3.2 branch to receive the new stuff. The first bits are already in the repository: the basic plugin framework and the conversion of the socket related stuff (tcp, udp, unix-dgram, unix-stream, syslog drivers) into a separate plugin.

The reason of the afsocket plugin conversion is to help moving the OpenSSL dependency to a separate package in distributions where this dependency cannot be associated with core packages like syslog-ng.

But I see a lot of potential behind the plugin framework, and I still have a lot of ideas, just check my last blog post in the topic.

Also, there's an even more experimental feature in the "blocks" branch right now, it is still incomplete, the naming and the syntax is still vague. The aim is there to provide syslog-ng.conf C++ template-like functionality in order to make the configuration easier by using pre-configured config snippets.

For example:

block source tomcat<root_dir="/opt/tomcat"> {
   file("<<root_dir>>/var/tomcat.log" follow_freq(1));
};

source s_local {
   tomcat(root_dir("/usr/local/tomcat"));
};

The idea is that if "tomcat" is referenced in a source statement, it'll be expanded to the content defined before that, all with proper argument passing and default values. This example actually works fine with the "blocks" branch right now.

Combining this feature with include files I foresee something like a "syslog-ng configuration library", which would contain pre-configured syslog-ng sources containing settings needed in order for a specific application to log properly, this way avoiding the hassle of integrating each and every application to syslog separately.

Another (not-yet-working) example would be to get rid off platform specific /dev/log declarations:

block source system<> {
@if (OS == "solaris")
@  if (OS_VERSION >= "10")
    sun-streams("/dev/log" door("/var/run/syslog_door"));
@  else
    sun-streams("/dev/log" door("/etc/.syslog_door"));
@  endif
@elif (OS == "linux")
    unix-dgram("/dev/log");
    file("/proc/kmsg" program_override("kernel"));
@elif (OS == ...)
    ...
@endif
};

source s_local { system(); };

This would make the initial configuration of syslog-ng much easier, and would allow you to use the same configuration file everywhere.

As I said this is still very experimental, I'm not yet sure about:

the name (I would have called these templates, but that is already taken in syslog-ng, just like "macro"). so if you have an idea of a better name I'd love hear about it
argument passing, whether to allow positional arguments
argument expansion syntax, "<<arg>>", and "$$arg" both occurred to me, the latter seems to be used more universally. I'd want to include both in strings (like in the example above) and in non-string arguments, e.g. follow-freq($$arg)

If you have comments, ideas for names, please post them as comments, or send an email to the syslog-ng mailing list.

Also, while I was there changing the parser/lexer framework for pluginization, I added more details to the error message to make it easier to locate the syntax error in the configuration file.

As for the next things, I'm going to start integrating the external contributions I've received so far, so stay tuned.

syslog-ng 3.1 final release

2010-03-22T15:56:00.003+01:00

I'm proud to announce that both the Open Source and the Premium editions of syslog-ng 3.1 was published and are available on our website.

This is an important milestone in multiple ways:

the new feature/stable release schema is making its debut
the patterndb got significant improvements: new parsers, pdbtool, tagging support
the ability to change/add RFC5424 style structured data to messages
even more supported platforms (Tru64 on alpha, HP-UX 11iv2 on Itanium and older Linux versions)
the diverging developments of syslog-ng Open Source Edition, Premium Edition and syslog-ng Store Box was merged into a new base,

Some interesting (ok, for us developers :) statistics follow:

Premium Edition:

586 commits
200 files changed, 23479 insertions(+), 5513 deletions(-)

Open Source Edition:

189 commits
115 files changed, 9020 insertions(+), 3225 deletions(-)

The reason for the big difference is the merger of the currently propriatery log indexer engine used in SSB into the current Premium Edition tree, otherwise the two should be in sync.

The binaries/source packages can be downloaded via the usual URL:

http://www.balabit.com/downloads/upgrades/

Changelogs for the two releases:

Premium Edition:
http://www.balabit.com/downloads/files/syslog-ng/premium-edition/3.1.0/changelog-en.txt

Open Source Edition:
http://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.1.0/changelog-en.txt

And of course the OSE source is also available in our public git repository:

http://git.balabit.hu/

Happy logging!

plugins branch updated

2010-03-06T15:45:00.003+01:00

Since the last post, I could hack a couple of hours on the plugins branch, which now compiles. The plugin framework is capable for supporting a quite important core functionality: all socket like sources/destinations are now found in an external plugin called "afsocket".

The reason I've started with afsocket is to make syslog-ng a bit less dependant on OpenSSL. A couple of distributions didn't include syslog-ng 3.0 in their current releases, because it uses OpenSSL from /usr, while syslog-ng should remain in the root directory.

By separating afsocket from the syslog-ng core, I can compile afsocket with and without TLS support, which can be put into separate packages. Thus syslog-ng can operate without OpenSSL.

And the same plugin framework will enable me to create a wide variety of plugins. My ideas:

Plugins for all syslog-ng components (source, destination, filter, rewrite, parser)
Python scriptability (a simple correllation engine in Python?)
macro transformation functions, for example: $(stripslashes $macro), usable anywhere in templates and stripslashes a plugin that is invoked whenever such an expansion occurs
Hooks for transforming the log message as it enters syslog-ng (to fix parsing errors for example),

Do you have other ideas? Please post them as comments or as emails to the mailing list.

Again, this functionality is experimental, and I'm still going to rebase the current code and will probably be integrated to syslog-ng 3.2. I got to release 3.1 final first though. :)

plugins preview

2010-03-06T07:32:00.003+01:00

Things have been a little rough last couple of months, that's why I haven't posted here. I'm in a rush right now as well, but I just wanted to let you know that I have started working on modularizing syslog-ng.

It is only a preliminary prototype, and as of now it doesn't compile, but the way it's going to work is already visible: each plugin will have its own plugin and with some trickery the large syslog-ng.conf parser will call out to the plugin parser. The user will recognize such a plugin as an integral part of syslog-ng.

E.g. this is a sample configuration file:

@version: 3.0
@module: dummy

...

destination d_dummy { dummy(dummy_opt(yes)); };

...

See the dummy plugin code in my git repository, in the "plugins" branch. Please note that that branch is going to be rebased a couple of times yet, I've released it in the spirit of "release early, release often".

I hope to get some of the recent contributions into plugins, instead of bloating the core syslog-ng code. For example output colorization. I'm also thinking about adding built-in scripting support via Python.

syslog-ng OSE 3.1beta2 release

2009-12-18T17:47:00.003+01:00

I've mentioned shortly in my previous post, but here's a more official announcement: I've released syslog-ng OSE 3.1beta2, containing some important bugfixes.

The list of changes: http://www.balabit.com/downloads/syslog-ng/open-source-edition/3.1beta2/changelog-en.txt

Thanks to Martin Holste for the feedback he provided, hopefully we can forget about the "beta" part soon.

Patterndb release for syslog-ng 3.1

2009-12-18T09:08:00.004+01:00

You may probably know that starting with syslog-ng 3.0, we started poking into the message payload by being able to extract information from the log messages and use that information in structured form for message routing, filtering and storing them as separate fields in a database table.

You may have read about patterndb on this blog or on Marci's blog and we have also given talks about it on different conferences: NNM 2009 and LSM/RMLL 2009.

The reason I'm raising the topic here again is that we have now released about 8000 patterns covering about 200 applications for patterndb and are now in the process of creating a community site to maintain this database.

You can download the database from www.balabit.com.

Also an important thing to know that syslog-ng OSE 3.1 features enhanced performance with regard to handling information extracted from the message payload and it also has support for the latest patterndb database format. So if you want to try the new database, fetch a copy of the latest 3.1beta2 release.

syslog-ng OSE 3.1beta1 released

2009-12-03T09:56:00.004+01:00

I'm proud to announce that syslog-ng OSE 3.1 has been released and uploaded to our webserver. This version is new in two ways:

1) of course it has new features, see below for the most interesting bits

2) it is a "feature release", which means that once syslog-ng 3.2 or syslog-ng 4.0 is released, the support for this release will be ceased. See our new version policy at this link:

https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/roadmap.bbx

Since the documentation is not yet up to date with this beta release, I'll try to include the most crucial information about the new features right here in this announcement.

For those who hurry, here's a link for the source code:

https://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.1beta1/source/syslog-ng_3.1beta1.tar.gz

And here are the binaries for Linux/FreeBSD systems:

https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/

Select the Downloads tab, and in the Version selector select 3.1beta1.

Please try this beta version. Any feedback, positive or negative is appreciated, if you have comments, please post them on the mailing list at: syslog-ng@lists.balabit.hu

And now the new features in this release:

Support for patterndb v3

syslog-ng 3.1 now supports the patterndb v3 format, along with a bunch of new parsers: ANYSTRING, IPv6, IPvANY and FLOAT.Patterndb (more exactly the db-parser()) is a high performance message classifier and information extraction tool, that makes it easy to get away from the unstructured nature of syslog.

Patterndb has evolved since it was first introduced in syslog-ng 3.0. It is at the 3rd iteration, hopefully slowly reaching its final form. syslog-ng OSE 3.0 supported v1, our SSB product supports v2 and now syslog-ng OSE is the first version supporting v3.

Patterndb in general and the v1 format database is described in the syslog-ng manual at:

http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch02s12.html

The XML schemas that describe the different patterndb versions are available in the syslog-ng source tree:

http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=tree;f=doc/xsd;hb=HEAD

The changes in the patterndb format as they evolved were described in Marton Illes's blog at

http://marci.blogs.balabit.com/2009/06/new-db-parser-format-and-other.html

But see the other related posts as well.

Old patterndb databases can be converted to the new format by putting them in the /opt/syslog-ng/etc/patterns.d directory and using the pdbtool utility using the command:

$ pdbtool merge -p /opt/syslog-ng/var/patterndb.xml \
-D /opt/syslog-ng/etc/patterns.d

Assuming the installation prefix of syslog-ng is /opt/syslog-ng. The above filenames are also the recommended and default names for patterndb related files.

Some v2 format patterns are distributed by BalaBit itself for its SSB product, download location:

https://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/

You can convert these db files using pdbtool as described above.

Work is ongoing to publish a more comprehensive patterndb, but more on that in a separate post.

pdbtool

Added a new "pdbtool" utility to manage patterndb files: convert them from v1 or v2 format, merge mulitple patterndb files into one and look up matching patterns given a specific message.

See the manpage (by adding /opt/syslog-ng/share/man in the MANPATH) and Marci's post:

http://marci.blogs.balabit.com/2009/08/db-parser-new-utility-pdbtool.html

Message tags

Support for message tags was added: tags can be assigned to log messages as they enter syslog-ng: either by the source driver or via patterndb. Later it these tags can be used for efficient filtering.

http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html

Rewrite structured data

Earlier structured data fields in the new RFC5424 style syslog protocol were only read-only values that could be referenced in a template, but they couldn't be changed, and neither was it possible to add new fields in an already existing syslog message.

Now all these became possible by using the same syntax that didn't work earlier, e.g.

rewrite r_sd { set("55555" value(".SDATA.meta.sequenceId")); };

Macro and name-value integration

Macros and name-value pairs got a little tighter integration. syslog-ng 3.0 was limited in the use of macros in the value() option of the match() filter: it could only use name-value pairs, although intiutively it should have supported macros as well. This was changed, starting with 3.1 it is now possible to use macros as well.

The following now works:

match("regexp" value("R_DATE"));

syslog-ng is now warning you in case you are using '
Name-value pair performance improvements

With the advent of patterndb and the spreading use of name-value pairs in syslog-ng, a strong limitation was the performance penalty of using dynamically created name-value pairs. This was now solved, 3.1 features a new data structure to store message payload and name-value pairs in, which results in a 3 times better performance when looking up a name-value pair.

Patterndb parser enhancements

Some parsers got additional features: NUMBER is now able to parse hexadecimal numbers, ESTRING is now able to search for a sequence of characters as the end of the string. These changes make it easier to describe log messages in patterns.

Information about non-portable facilities

Added non-standard and non-portable facility codes (range 10-15), and decoupled syslog-ng facility name database from the system used to compile syslog-ng on.

Until this time the facility codes as understood by syslog-ng were dependant on the platform syslog-ng was compiled on. This is not true anymore, syslog-ng comes with its own "facility" code assignments, based on the RFC, and adding some non-standard values found on various UNIX systems. prefix in the value syntax, because you can't use the full template syntax when you specify a value to match against.

Name-value pair performance improvements

With the advent of patterndb and the spreading use of name-value pairs in syslog-ng, a strong limitation was the performance penalty of using dynamically created name-value pairs. This was now solved, 3.1 features a new data structure to store message payload and name-value pairs in, which results in a 3 times better performance when looking up a name-value pair.

Patterndb parser enhancements

Some parsers got additional features: NUMBER is now able to parse hexadecimal numbers, ESTRING is now able to search for a sequence of characters as the end of the string. These changes make it easier to describe log messages in patterns.

Information about non-portable facilities

Added non-standard and non-portable facility codes (range 10-15), and decoupled syslog-ng facility name database from the system used to compile syslog-ng on.

Until this time the facility codes as understood by syslog-ng were dependant on the platform syslog-ng was compiled on. This is not true anymore, syslog-ng comes with its own "facility" code assignments, based on the RFC, and adding some non-standard values found on various UNIX systems.

The neatest syslog-ng hack ever

2009-09-21T17:57:00.004+02:00

One of my collegues probably felt like crazy and implemented a twitter() destination driver for syslog-ng. Although the value is dubious, I think it is the neatest contribution to syslog-ng so far. :)

syslog-ng 3.1 status

2009-08-09T11:33:00.002+02:00

Like I announced in one of my previous posts, towards the syslog-ng OSE 4.0 release I'm going to make smaller, short-term supported releases. The first of these, called syslog-ng 3.1 is nearing completion, and thus a status report is due.

Here's the original plan (quoting the roadmap page here):

support tags for syslog messages: each message can be marked with one or more tags, then apply filtering based on tags
patterndb: add tag support
patterndb: v2 database format support
patterndb: add parsers for IPv6 addresses and hex numbers
converge macros in templates and name-value pairs even more (right now it is not possible to use any macro in match())

I've just pushed out another set of updates to our git repository, which:

adds tag support: a new tags() filter and a tags() option for all sources and a builtin logic to assign the syslog-ng source name as a tag (in the format: .source.)
adds support for patterndb v2 and a newly introduced but compatible v3 format
adds "pdbtool" a new utility for managing patterndb files (not yet complete)
a couple of new parsers (IPv6, ANYSTRING, FLOAT)

The last item in the roadmap is not yet addressed, in fact I haven't even started it yet. I'm thinking about leaving that out altogether in order to have 3.1 released as soon as possible. If you have an opinion about that please don't hesitate to post it here on the mailing list.

If you are experimenting with patterndb you are advised to use the 3.1 branch as development happens here. Of course if we find something that affects our current stable 3.0, I'm backporting the fix, but since 3.0 is stable, I'm only backporting bugfixes and not new functionality.

If you are interested you can get the sources via git, or if you prefer a tarball, just drop me an email.

Developer tools

2009-08-04T15:41:00.004+02:00

BalaBit has grown quite a lot in the last 9 years since it was founded, these days there are about 60 employees and more than 50% of that is working in the development field (give or take a couple, I've lost count some time ago). As we currently work on 4 products, support 5-6 CPU architectures and a host of different Operating Systems, automation in development is a must.

We try to automate everything and that means a lot. Some examples:

preparing the development workstation for development/testing work in 15 minutes for any of our products
building source code for tens of CPU/OS combinations by issuing a single command
creating bundles of intermediate components when generating setup packages
doing releases
test automation
and a host of other things

Some of these solutions are completely our own development, others are derived from public projects, and as BalaBit tries hard to be a good friend of Free, Libre and Open Source Software (FLOSS) we try to contribute back to projects that we use.

A couple of weeks ago, I published our modified version of dogtail, a test automation framework for AT-SPI based applications. We maintain our own dogtail in-house and since our patches were not accepted, we published our changes in a public git repository.

Earlier, one of our developers contributed to WAF to support building with Microsoft Visual C++, we've been using his work in two of our internal projects.

And this time, we published cccl a wrapper for MSVC to make it compatible with the gcc command line, in order to compile autoconf based projects under MSVC.

As you could guess, BalaBit is primarily a UNIX/Linux shop, but we need to support products aimed at Microsoft Windows, however with some heavylifting combining the best of both worlds is possible. And we've never been afraid of challenges. :)

Hopefully you can use some of these results, maybe even contribute back.

patterndb updates pushed in syslog-ng OSE 3.1

2009-07-16T22:43:00.002+02:00

According to the plan of my recently published syslog-ng OSE roadmap, I've worked on integrating the various patterndb related fixes/enhancements in the syslog-ng OSE 3.1 tree.

This now means that OSE 3.1 is now capable of working with all the version2 style pattern databases that syslog-ng Store Box is using. Here is a link for the SSB patterns: http://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/

I still need to work on integrating the new tags framework and the integration between tags and patterndb. Once that is done, I only have one item left for the 3.1 feature release.

So with some luck, we can have a new shiny syslog-ng OSE release this summer.

Please note that this is not released code yet and is only available via git, however if there's demand, I'm willing to create an alpha release (with binaries) if you want to try it. Just drop me an email, or simply write a comment to this post, and I'm going to create one for you.

Stay tuned.

syslog-ng rewrite use case: dpkg logs

2009-07-08T09:43:00.002+02:00

One of my collegues (Péter Höltzl, he does all our trainings) has created a nice detailed example on how to use the parser/rewrite framework to pull in yet another application into syslog: dpkg, the Debian package manager.

If you are interested in what rewrite/parser can do for you, but didn't have the time to find out, the blog post is worth a read.

syslog-ng pipelines

2009-06-19T06:54:00.003+02:00

The other day someone wanted a special syslog-ng macro that would expand into digit changing every 5 seconds (e.g. R_UNIXTIME % 5) and although I couldn't give an exact solution to his problem, I've came up with this configuration snippet:

rewrite p_date_to_values {
set("$R_DATE", value("rdate"));
};

filter f_get_second_chunk {
match('^... .. [0-9]+:[0-9]+:(?<rdate.second_tens>[0-9])[0-9]$'
type(pcre) value('rdate'));
};

The way it works is as follows:

the rewrite statement sets the name-value pair named "rdate" to $R_DATE (the macro)
the filter statement uses Perl Compatible Regular Expressions to parse the value of the "rdate" value and uses a named subpattern on the tens of seconds position to store that character in a value named "rdate.second_tens"
Later on in the configuration you can use "rdate.second_tens" just like any other macro/value.

This proves that the current rewrite/parser/filter subsystems are really powerful, however even though this proved to be possible, there are some lessons learned from this example:

the macro and name-value space should really converge to each, this would mean that the match() filter could directly match against the macro value $R_DATE without the need for the separate rewrite statement
when you are after a given goal, you don't really want to differentiate rewrite/parser/filter rules at all. The current syntax of using separate blocks for separate type of log processing elements is a pain.

So I'm thinking about inventing yet another block, which simply wouldn't care what kind of processing element is added to it, something along the lines:

pipeline rdateseconds {
set("$R_DATE", value("rdate"));
match('^... .. [0-9]+:[0-9]+:(?[0-9])[0-9]$'
type(pcre) value('rdate'));

};

And then:

log {
source(src);
pipeline(rdateseconds);
destination(dst);
};

Maybe I should even allow the creation of rewrite/parser/filter elements right there in the log statement:

log {
source(src);
filter(facility(mail));
destination(dst);
};

What do you think?

Nordic Meet on Nagios 2009

2009-06-03T10:34:00.003+02:00

I'm sitting at NMN 2009 right now, and although the event title says it is a Nagios meet, I'm going to give a presentation on syslog-ng and the new features that 3.0 brings and an example on how to integrate syslog-ng and Nagios.

If you are here and have a question just feel free to find me in the "BalaBit" T-Shirt. :) There's also live streaming on the conference website, so you can catch me at 15:50 Central European Time.

syslog-ng 4.0 roadmap plus release policy changes

2009-05-30T19:20:00.004+02:00

I've updated the syslog-ng OSE roadmap on the syslog-ng webpage to include information about the upcoming syslog-ng version:

http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/roadmap/

Also, I'd like to bring the changed release/support policy to your attention, that you can read at the same location above. I'd like to introduce stable track and feature track releases, the first being supported for a long time, whereas feature track releases are only supported until the next feature/stable release is published. When a sufficient number of features were published via feature track releases, the last one becomes stable and the cycle continues. Note that feature releases are NOT development snapshots, they are releases just like the major versions previously, the only difference is that instead of a large feature list like with syslog-ng 3.0, only a smaller set of changes are included.

This makes it possible to publish features more often, always concentrating on a few of them at a time, instead of doing development for a long time and come out with a feature packed release. I hope to increase the pace of syslog-ng development with this change and also to cause less problem for users who prefer stability over features. Please read the details on the roadmap page.

I've also opened the syslog-ng 3.1 repository and pushed it to our git server. Right now there are no differences (except for the version number) between 3.0 and 3.1, I'm planning to integrate Marton's message tagging and patterndb changes as soon as possible (his git tree is here). Hopefully the 3.1 cycle will be quite short as most of the things on the roadmap are already implemented, although scattered around in various public and private trees.

With the opening of the 3.1 branch, I'm also obsoleting 2.0 (in the new support model two stable track versions are supported at any given time and we have 2.0, 2.1 and 3.0 right now), but that'll go in a separate post/announcement.

syslog-ng OSE 3.0.2 released

2009-05-08T22:23:00.003+02:00

After a long time and a lot of accumulated bugfixes, I've pressed the "release" button and syslog-ng OSE 3.0.2 was published on our website. The first official version to feature binary packages for Linux and BSD platforms. Since there was a long time between 3.0.1 and 3.0.2 the changelog is quite large, however most of it are bugfixes, only some minor enhancements here and there.

Hopefully I didn't miss any important bugs and problems. It must be much better stability/functionality wise than 3.0.1 was.

The diffstat since 3.0.1:
150 files changed, 4332 insertions(+), 3000 deletions(-)

You can also check the patches in our git repository.

If you are using the 3.0 branch you are really recommended to check out this release. If you are using anything earlier than 3.0 you are also recommended upgrade, syslog-ng 3.0 is revolutionary to previous versions in many ways, especially if you want to do more to your logs than merely store them in a plain text file.

OSDC 2009 slides

2009-05-08T15:30:00.003+02:00

I've uploaded my OSDC 2009 presentation slides to
http://people.balabit.hu/bazsi/slides/osdc-2009-syslog-ng-3.0.odp Which has an example for processing iptables logs with db-parser() and putting the results in a customized SQL table.

Nordic Nagios Meet 2009

2009-05-03T20:36:00.004+02:00

I'm going to give a talk on syslog-ng on the upcoming Nordic Nagios Meet 2009. I expect the event to be great fun, just like last year. If you are in the Nordic region and use Nagios, rrdtools or syslog-ng, I recommend to pay a visit as you can meet the primary authors and some active contributors to these projects.

If you are there and have anything to ask/talk about syslog-ng, just feel free to approach me, I'm probably going to wear a badge, so you can recognize me :)

OSDC 2009 and syslog-ng automatic testing

2009-05-03T20:22:00.006+02:00

I've spent the last week in the nice city of Nuremberg where Open Source Data Center Conference took place, organized by Netways AG. I really liked the talks about Puppet, DRBD and the description of the booking.com infrastructure which runs MySQL.

Although I really enjoyed the conference I also had some free time to improve the automatic test program for syslog-ng, which now also covers TLS encrypted source and SQL destinations. I've also implemented a small script to collect coverage data of the testcases, thus right now I know that about 63% of syslog-ng is covered by automatic tests. (initially it was 55% but there were some low hanging fruits). I expect to raise this number easily to around 80%, then it'll probably become much more difficult to increase it further as the rest is error processing paths, and unless I come up with something to inject errors from the testcases those are difficult to test.

Of course having a test suite is not a replacement for real-life, field testing, but nevertheless it makes it much easier to do releases as it ensures that no important functionality is broken completely.

Based on this test infrastructure I'm going to release 3.0.2, after which I'll probably change the way I manage releases for syslog-ng, but I'll talk about that in a forthcoming post.

My son is 7 weeks old

2009-05-03T08:55:00.004+02:00

From Dani-aprilis-25

The reason I was absent from this blog in the couple of last weeks is my now 7 weeks old son, Dániel. You can find a picture of him right here in the post, but some additional ones in my Picasa albums.

Features that fell off the radar

2009-03-23T22:15:00.004+01:00

I was long pondering with the problem that it is quite tricky to enter regexps into syslog-ng configuration file, since if you enclose the string in double quotes (e.g. in ""), the backslash character needs to be escaped.

Since backslash is used in regexps quite often, it can become cumbersome to enter regexps like:

match("[a-z\\-]+");

Note that the backslash is doubled because otherwise the syslog-ng string parser would pass the sequence to the regexps compiler as: "[a-z-]+" which is certainly different in meaning what the above expression says.

I always remembered that syslog-ng also supports single quotes (aka apostrophes), but I remembered they behaved just as if you used normal quotation marks. Therefore I was thinking about a 3rd string format, one that would not require escaping.

However I was reading the related code the other day, and found that apostrophes work exactly the way I planned this 3rd string syntax to behave: not to get in the way when entering regexps. In fact it behaves just like apostrophes in the UNIX shells. It does not care about escaping, it only cares about the terminating apostrophe.

I was dealing with regexp related questions on the mailing list a lot, and the root cause of the problems was most times this escaping stuff, and I never knew the proper answer and behaviour is already in syslog-ng, I've just forgotten about it completely.

And now as I check the documentation for syslog-ng, it does not mention this syntax either, even though it had been present even in the 1.6.x times.

So if you had trouble writing lots of regexps in syslog-ng configuration, and I told you to properly escape your regexps, please forgive me. syslog-ng is better than I've thought :)

Newborn baby

2009-03-16T17:59:00.002+01:00

After about two weeks being late, my son was born yesterday evening at 22:45CET. He weights 3270g and 56cm. Both the mother and the child are fine and I'm a proud new father.

I guess this starts a section in my life, hopefully for the better.

syslog-ng OSE binary packages

2009-03-14T11:06:00.003+01:00

I' happy to announce that BalaBit has decided to make the binary packages for syslog-ng OSE available for free.

As you may know, BalaBit has various syslog-ng support packages and as a part of this service it prepared binary installation packages for different platforms. The access to these packages either required a support contract but could also be purchased separately for a yearly fee.

With syslog-ng 3.0, the binary packages for syslog-ng OSE will become freely accessible.

Since syslog-ng is an open source project, BalaBit planned to finish this task in the Open Source spirit: open and visible to all community members. This also means that the set of packages published with this e-mail is NOT yet release grade, rather it is more of a development snapshot of the current state of affairs. So please don't ruin your production systems with this package, it is more advisable to try them in a test environment (chroot or a dedicated test machine).

With all these said, here is the link:

https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/upgrades/

Please pick the release named "3.0HEAD". This contains a source snapshot (effectively git from two days ago), and a set of packages for SUSE 10, RHEL4/5, FreeBSD 6.x, Debian etch, and Linux generic.

The binary packages contain all runtime dependencies needed to run syslog-ng, thus no further packages are required, it is an all-in-one package. The rpm/deb packages are prepared the same, they install syslog-ng in /opt/syslog-ng in order to avoid clashes with a system supplied syslog-ng daemon.

There are two install kits for each platform:

one that includes database drivers (dubbed as "server")
one that does not include database drivers (dubbed as "client")

Currently there are no other differences between the packages, but later on there might be.

With the current infrastructure in place, I'm confident that with each syslog-ng OSE release, I can publish the source AND binary packages at the same time.

I'd really appreciate success/failure reports and also any kind of comment you may have.

I'd like to release 3.0.2 together with its binary packages, let's hope that I get enough feedback on these packages so that I can do that.

Enjoy!

First IETF syslog-protocol related question

2009-03-11T22:55:00.003+01:00

I'm happy as I've received the first question about the new IETF specified syslog-protocol support. There's a need for that after all :)

Next event on the horizon

2009-03-11T08:31:00.005+01:00

I didn't realize it is already that time of the year, but I was reminded that I'm going to give a talk on syslog-ng 3.0 on Open Source Data Center conference in Nürnberg, Germany at the end of April. I'm going to talk about the nifty new features of syslog-ng 3.0.

It would be very nice to meet syslog-ng users there. :)

An introduction to db-parser()

2009-03-03T10:58:00.005+01:00

As promised on the mailing list here comes a short description of the new db-parser functionality of syslog-ng. For an introduction to parsers in general see my previous blog post here.

The aim for db-parser is two-fold:

extract interesting information from a log message
attach tags to a log message for later classification.

For instance here's a log sample (lines broken for readability):


Feb 24 11:55:22 bzorp sshd[4376]: Accepted password for bazsi \
      from 10.50.0.247 port 42156 ssh2

This message states that a user named "bazsi" has logged into the host named "bzorp" using SSH2 from the quoted IP and port. When you read this message as a human, the event that happened is perfectly clear. However if it is not a human, but a piece of software that has to make out the meaning of the message, you need to identify the event (e.g. that a user login has happened) and the additional information associated with the event (e.g. that he used 10.50.0.247 as the client).

If I wanted to express this as name-value pairs, it would be something like this:


event="user login", protocol="ssh2", \
      client="10.50.0.247:42156", method="password"

Surely this latter form is easier to analyze than the first. So the first step of all kinds of log analysis is to extract information from messages. At a first glance, the easiest way to extract this information is the use of
regular expressions. For example:


^\w{3} [ :0-9]{11} [._[:alnum:]-]+ sshd\[[0-9]+\]: Accepted \
  (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) \
  for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?

Once you match with the regular expression above (courtesy of the logcheck project), the parentheseses mark the variable part of the information that you can reference as $1, $2 and so on.

The problem with regular expressions are several fold:

they are difficult to write (just look at the example above)
they are even more difficult to understand, once written (again, please look at the example)
they are slow and they scale poorly with the number of regexps that we need to match against the incoming message stream.

Projects like logcheck use regular expressions, but with the number of patterns increasing, the time needed to analyze logs skyrockets, which makes the whole thing unfeasible. Also, logcheck does not aim at extracting information from messages, it merely classifies them.

Clearly a different approach is needed. And that's what db-parser in syslog-ng is.

The db-parser() functionality of syslog-ng has the following objectives:

use a database to match various messages (and not filters embedded in the configuration file)
classify events into logcheck-like classes (cracking, violation, ignore, unknown)
extract variable information from messages, and place those into name-value pairs
be fast, scale to a high number of events/sec and high number of patterns
integrate well to the rest of syslog-ng

db-parser() is a generic parser, fits nicely to the parser framework inside syslog-ng. You can use it just like csv-parser():


...
parser p_db { db-parser(); };
...
log { source(src); parser(p_db); destination(d_parsed); };
...

The database used by db-parser is an XML file that is read during syslog-ng startup. Here is an example entry from the db-parser() database:


<patterndb>
<ruleset name='sshd'>
 <pattern>sshd</pattern>
 <rules>
   <rule provider='balabit' id='1' class='system'>
     <patterns>
       <pattern>Accepted rsa for@QSTRING:username: @from\
@QSTRING:client_addr: @port @NUMBER:port:@ ssh2</pattern>
     </patterns>
   </rule>
   ...
 </rules>
</ruleset>
</patterndb>

As you can see the database is structured, and the first selection criteria to apply is the name of the application (e.g. the value for $PROGRAM). Then each rule matches against the message payload (e.g. the value for $MESSAGE) with the syslog header stripped off. The rule specifies the classification (e.g. 'system' in the example above) and lists one or more patterns. If any of the patterns match, the rule is considered a match.

The variable part of the pattern is specified using special sequences, starting and ending with a '@' character. Within the enclosing '@' characters a colon separated list of parameters are listed:

the parser to apply (QSTRING and NUMBER in the example above)
the name of the value to be extracted from this position
additional arguments to be passed to the parser

The available parsers are currently not really documented, but here is a
list of them (you can find these in the radix.c source file):

IPv4: to parse an IPv4 address
NUMBER: to parse a number
STRING: to parse a word
ESTRING: to parse a sequence of characters ending with a specific character
QSTRING: to parse a string enclosed within quotes

Of course further parsers can be added to the code easily. You don't have to specify monsterous regexps to match an IPv4 address anymore. Not to mention IPv6 :)

If a message matches a rule, the db-parser() will make the following list of values defined for the given message:

.classifier.class: logcheck-like classification
.classifier.rule_id: the ID of the database entry that matched
pattern specific values: variable part that get extracted from the message by patterns

Each of the values defined previously can be referenced inside syslog-ng using a macro, e.g. you can do things like:


# You can use them in a filter:
filter f_class {
match("system" value(".classifier.class"));
};

# but you can also use them in the names of files:
destination d_parsed {
file("/var/log/messages/${.classifier.class}.log");
};

That's a rough skeleton of what db-parser() is. If you are interested, you can find the db-parser() implementation in syslog-ng OSE 3.0:

http://www.balabit.com/network-security/syslog-ng/opensource-logging-system/

You can also find some example pattern databases here:

http://www.balabit.com/downloads/files/patterndb/

We are also thinking about further ideas to enhance db-parser() and make it the foundation of an Open Source log analysis framework. Stay tuned!

GStaticMutex and AIX

2009-01-18T12:02:00.003+01:00

If you use GLib on non-Linux platforms such as AIX and think that G_STATIC_MUTEX_INIT does nothing but zero-initialize the mutex, think twice. Although quite clearly stated in the documentation, I thought I was smarter and used a GStaticMutex embedded in a structure that was zero initialized.

If you look at the definition of G_STATIC_MUTEX_INIT on most platforms (Linux, Solaris, BSDs), it contains nothing but zeroes. This lead me to the impression that zero filling a GStaticMutex instance is enough to initialize it.

In reality it isn't. On AIX this renders the mutex to be entirely useless without warnings or aborts. The results are of course bugs that are difficult to track down and fix.

This took me an entire day to figure out, as the SQL driver in syslog-ng had this problem. This was fixed since, but if you are running syslog-ng on AIX with the SQL driver, be sure to have this patch applied.

syslog-ng OSE 3.0 finally released

2009-01-15T13:20:00.003+01:00

Finally I could take the time to actually announce the freshly released syslog-ng OSE 3.0 branch. It was uploaded to our website during the winter holidays, but I had to integrate syslog-ng OSE to our new release infrastructure, which among others has a much nicer web interface.

Here is a summary on what is new in syslog-ng 3.0:

http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch01s04.html

Enjoy!

include file support implemented

2008-12-11T23:41:00.004+01:00

I've implemented basic include file/directory functionality in syslog-ng, using the format numbered second in my previous post.

I've now pushed an expermental implementation of include files in the syslog-ng OSE 3.0 repository, in a separate branch called 'include'.

E.g. in order to test the code, please clone the syslog-ng 3.0 repository:

$ git clone git://git.balabit.hu/bazsi/syslog-ng-3.0.git

Then check out the 'include' branch:

$ git checkout --track -b include origin/include

Then compile as usual. I didn't want to integrate it right into syslog-ng OSE 3.0 tree as I'd like to release that first as 3.0.1.

include syntax

2008-12-10T16:16:00.003+01:00

I'm about to implement configuration file includes, and although the implementation is quite straightforward, the syntax to be used is something to give a thought or two.

Currently the syslog-ng configuration file consists of statements, each with the following basic format:

stmt [] { ... };

The "id" gives a unique identifier of the statement, and the braces enclose the contents. Currently only the ID part is optional, the braces are always there.

To make the include statement consistent with that, it'd have to look something like:

include { "filename" };

Obviously I don't like this too much, as it is way different from all other applications permitting the use of include statements. What about this:

include "filename";

E.g. use the ID part the name of the file to be included. I like this better. A third option might be the use of 'pragma' directives, currently only used to specify the file format compatibility in the case of syslog-ng 3.0:

@version: 3.0

This'd mean that include statements would look like this:

@include: filename

The problem with this last option is that pragmas are currently only processed at the beginning of the configuration file. So that code should also be generalized.

I think I'd go with the second option, that's not completely inconsistent, but still the most intuitive to use.

What do you think?

syslog-ng 3.0 and SNMP traps

2008-11-23T13:55:00.007+01:00

Last time I've written about how syslog-ng is able to change message contents. I thought it'd be useful to give you a more practical example, instead of a generic description.

It is quite common to convert SNMP traps to syslog messages. The easiest implementation is to run snmptrapd and have it create a log message based on the trap. There's a small issue though: snmptrapd uses the UNIX syslog() API, and as such it is not able to propagate the originating host of the SNMP trap to the hostname portion of the syslog message. This means that all traps are logged as messages coming from the host running snmptrapd, and the hostname information is part of the message payload.

Of course it'd be much easier to process syslog messages, if this were not the case.

A solution would be to patch snmptrapd to send complete syslog frames, but that would require changing snmptrapd source. The alternative is to use the new parse and rewrite features of syslog-ng 3.0.

First, you need to filter snmptrapd messages:

filter f_snmptrapd { program("snmptrapd"); };

Then we'd need to grab the first field of the message payload, where snmptrapd is configured to put it:

rewrite r_snmptrapd {
subst("^([^ ]+) (.*)$ ", "${2}");
set("${1}" value("HOST"));
};

Both rewrite expression kinds are demonstrated here:

subst() has two arguments: the first is a regexp to search for, the second is a template to be substituted if there's a match
set() has a single argument: a template to be used as the new value

Rewrite rules operate by the contents of the $MESSAGE value by default, which holds the message payload. Of course this can be changed by specifying the value() option. The notion 'value' in syslog-ng 3.0 refers to a name-value pair, in syslog-ng 3.0 every message is composed of a set of name-value pairs. The names of standard values match the name of the corresponding macro, but without the '$' sign.

Please NOTE that the new value is a template which makes it possible to use macros such as $HOST or $MESSAGE defined by syslog-ng.

Now let's wire the complete configuration together:

filter f_snmptrapd { program("snmptrapd"); };

rewrite r_snmptrapd {
subst("^([^ ]+) (.*)$ ", "${2}");
set("${1}" value("HOST"));
};

log {
source(s_all);
filter(f_snmptrapd);
rewrite(r_snmptrapd);
destination(d_all);
flags(final);
};

log {
source(s_all);
destination(d_all);
flags(final);
};

Of course this is only an example of the power of what syslog-ng is now capable of doing. Please let me know if you can think of other uses.

The current 3.0 branch of syslog-ng has not been released yet, it is available in the git repository at git.balabit.hu, and also as nightly snapshots.

I'd be grateful for any kind of feedback you might have, please post it either as comments on this blog, or to the mailing list.

syslog-ng statistics

2008-11-08T09:05:00.003+01:00

For a long time I meant to give the "log statistics" feature of syslog-ng an overhaul, and finally with the advent of syslog-ng 3.0, this was done.

I'm not sure all of you know, but even earlier syslog-ng versions (2.1 and 2.0) did collect some per-source and per-destination statistics. These were reported periodically in the system log. The problem with this approach that it didn't really scale: with a large configuration the statistics message could become kilobytes long, and parsing this information from a file possibly several gigabytes in size is daunting.

syslog-ng 3.0 has two important changes in this area: it adds several new kinds of counters (like per-host counters), and a UNIX domain socket where you can query the current status of these counters.

As counters certainly have an overhead, you can now control how much statistics you want to gather. The new stats_level() option has three levels for now:

stats_level(0) is basically the same as earlier syslog-ng versions, per-source and per-destination statistics are kept here. This is the default.
stats_level(1) adds new counters without a big overhead, that is it adds counters for TCP connections, but does not keep per-host counters
stats_level(2) adds counters that can have a measurable performance impact, it adds for example per-host (as in $HOST) counters and also keeps track of the time the last message was received from a given host. These counters usually require an hash table lookup in the fastpath.

Once you have the counters, you can still use the venerable "log statistics" message, by setting the stats_freq() option which defaults to 10 minutes, just like in earlier versions.

However if you don't want to dig the logs produced by syslog-ng, you can also use the new UNIX domain socket at /var/run/syslog-ng/syslog-ng.ctl (the path might depend on the compilation options).

If you connect to this socket using netcat (some netcat versions do support UNIX domain sockets), and you send a "STATS" command to it, you get the list of counters.

There are no proper, command line clients for the UNIX domain channel yet, but if you have some scripting ability, you can start gather statistics easily, without the hassles of parsing log files, right after installing a syslog-ng 3.0 snapshot. :)

syslog-ng message parsing

2008-10-30T11:45:00.006+01:00

Earlier this month, I announced the new syslog-ng 3.0 git tree, adding a lot of new features to syslog-ng Open Source Edition. I thought it'd be useful to describe the new features with some more details, so this time I'd write about message parsing.

First of all, the message structure was a bit generalized in syslog-ng. Earlier it was encapsulating a syslog message and had little space to anything beyond that. That is, every log message that syslog-ng handled had date, host, program and message fields, but syslog-ng didn't care about message contents.

This has changed, a LogMessage became a set of name-value pairs, with some "built-in" pairs that correspond to the parts of a syslog message.

The aim with this change is: new name-value pairs can be associated with messages through the use of a parsing. It is now possible to parse non-syslog logs and use the columns the same way you could do it with syslog fields. Use them in the name of files, SQL tables or columns in an SQL table.

Here is an example:


parser p_parse_apache_logs { ... };

destination d_peruser { 
  file("/var/log/apache/${APACHE.USER_NAME}.log"); 
};

log { 
  source(s_local); 
  parser(p_parse_apache_logs); 
  destination(d_peruser_files); 
};

This means that you can "extract" information from the payload and use this information for naming destination files or SQL tables, basically anywhere where you can use a template.

There are currently two parsers implemented in syslog-ng:

a generic CSV (comma separated-values) parser, which can be parameterized to basically accept any kind of formally formatted input (so tab/space separated is also ok)
a database based parser, which uses a log pattern database to recognize messages belonging to specific applications and extract information on that.

Since the database based parser is quite complex so it deserves its own post, I'd skip that for now. The CSV parser has the following options:

template: defines the input to be used for parsing, can use macros
columns: list of strings, the names to be associated with the columns parsed
delimiters: the set of characters that delimit columns
quotes or quote_pairs: the quote characters to support, quote_pairs makes it possible to use different start and end quote (like enclosing fields in braces)
null: the null value which if found should substituted with an empty string
flags: see the documentation

The csv parser is capable of parsing real CSV data, e.g. it knows about quoting rules. So if you have an application that logs into files using space or comma separated data, you can almost be sure that you can process it with CSV parser.

Here is an example that parses Apache logs, so that each field in the message becomes a name-value pair:


parser p_apache {
csv-parser(columns("APACHE.CLIENT_IP",
                "APACHE.IDENT_NAME",
                "APACHE.USER_NAME",
                "APACHE.TIMESTAMP",
                "APACHE.REQUEST_URL",
                "APACHE.REQUEST_STATUS",
                "APACHE.CONTENT_LENGTH",
                "APACHE.REFERER",
                "APACHE.USER_AGENT",
                "APACHE.PROCESS_TIME",
                "APACHE.SERVER_NAME")
               # flags:
               #   escape-none,escape-backslash,escape-double-char,
               #   strip-whitespace
               flags(escape-double-char,strip-whitespace)
               delimiters(" ")
               quote-pairs('""[]')
    );
};

parser p_apache_timestamp {
    csv-parser(columns("APACHE.TIMESTAMP.DAY",
                       "APACHE.TIMESTAMP.MONTH",
                       "APACHE.TIMESTAMP.YEAR",
                       "APACHE.TIMESTAMP.HOUR",
                       "APACHE.TIMESTAMP.MIN",
                       "APACHE.TIMESTAMP.MIN",
                       "APACHE.TIMESTAMP.ZONE")
               delimiters("/: ")
               flags(escape-none)
               template("${APACHE.TIMESTAMP}"));
};

The first parser splits the major fields, and the second splits the timestamp to manageable pieces. You can then bind this parser to a log path of your choosing:


log {
  source(s_apache);
  parser(p_apache); parser(p_apache_timestamp);
  destination(d_apache);
};

As you can see the second parser uses a value created by the previous parser, using its template() option. Once this parsing is done, you can use any of the values created this way
in your d_apache destination, be it the name of the file, or a column in an SQL table.

6th Netfilter workshop

2008-10-08T21:36:00.005+02:00

I've spent my last week in Paris, where this year's Netfilter Workshop was held. I wanted to take this opportunity to thank Eric of INL for the organization. It was a wonderful and useful event, and I enjoyed it a lot. It is always nice to meet these wonderful guys.

Here are some blog posts about the same event:

INL: http://nfws.inl.fr/en/
DaveM: http://vger.kernel.org/~davem/cgi-bin/blog.cgi/2008/10/05#nfws2008
Patrick McHardy: http://people.netfilter.org/kaber/weblog/

Finally we could get Transparent Proxying merged, now queued for 2.6.28.

syslog-ng OSE 3.0 git tree published

2008-10-01T16:03:00.002+02:00

I could finally get my syslog-ng 3.0 OSE tree published at git.balabit.hu. No nightly snapshots yet and I still have to prepare a formal announcement to post on the mailing list, but for those I teased with functions from the 3.0 branch, here it comes.

From the top of my head, OSE 3.0 supports:

TLS encrypted channels,
syslog message rewrite,
parse parts of the syslog message and use the parsed parts in macros
PCRE and glob filters (in addition to POSIX regexps),
support for the new IETF syslog protocols,
program sources,
new statistics framework that can be queried using UNIX domain sockets
etc.

I just wanted to get the word out. Success/failure reports would be appreciated.

Migrate over to PCRE?

2008-06-30T16:30:00.003+02:00

As of now the development of the generic rewrite feature has been completed in one of my private git repositories. The new code uses PCRE and I'm somewhat undecided how to move forward with PCRE.

For those who might not know PCRE is an implementation of regular expressions and is an acronym for "Perl Compatible Regular Expressions". PCRE adds a lot more features and seems to perform better than its POSIX equivalent.

So the situation is as follows:

various filters use POSIX regexps
rewrite uses PCRE

This is not a very consistent combination, thus I'm planning to add PCRE support for filters too. The only question is whether it is needed to have two independent regexp styles in syslog-ng in the long run.

If I decide that one of them is enough, then I'd deprecate POSIX style regexps in filters and wouldn't implement POSIX in rewrite rules. This combination would yield a syslog-ng that would give warnings when POSIX-style regular expressions are in use and in a forthcoming release I'd change the default regexp style to PCRE, and yet another syslog-ng release later, I'd phase out POSIX completely.

If the decision is to keep them both in the long run, it would mean that I'd need to implement POSIX style regexps for rewrite rules as well. This would probably the least intrusive for users, but also a lot more work. Also, this would allow adding other filtering options like globbing or prefix search.

What do you think? Is the addition of modular search algorithms worth it?

Please send your opinions to the mailing list: syslog-ng@lists.balabit.hu

Nordic Nagios Meet '08

2008-06-08T14:14:00.005+02:00

I've spent the good part of last week in Stockholm at a gathering of Nagios users and developers. I was invited to give a talk on syslog-ng and security issues about collecting syslog data centrally as syslog-ng is often used hand-in-hand with Nagios.

This was my first time in Sweden and I can say that Swedish people are the most hospitable nation I met so far, and Stockholm is a very nice city. Also op5, the company organizing the event did their best to make us - the speakers - very welcome.

Thanks op5.

If you have a chance visit Stockholm I'd recommend to do so.

First incarnation of LogStore

2008-04-07T20:37:00.001+02:00

I've disappeared from this blog in the recent month but I've not been idle: I've implemented initial support for LogStore in the Premium Edition of syslog-ng.

LogStore is a binary log file format, in semantics very similar to a plain log file. But the format allows much more:

on-line compression via gzip,
encryption via AES and X.509 certificates,
integrity protection via hmac-sha1.

And furthermore: it is indexed based on time, and it is quite efficient to look for a specific time range in GBs of log data. I'm quite satisfied, although there are some more work left to be done, for instance the query interface for the time based indexing is not completed.

In use it is quite simple: replace the "file" destination with "logstore" and you are done. More or less the same amount of options are supported: macro based file names, template based formatting, etc.

I'm still pondering with the idea of storing the complete internal representation of the logrecord in serialized form, so it'd be possible to perform template() based formatting in off-line mode.

This code will be released as an experimental part of syslog-ng PE 2.1 and will be finalized in syslog-ng PE 2.2.

libdbi patches online

2008-02-28T16:39:00.000+01:00

I've published our set of dbi and dbi-drivers patches in a git repository to push changes upstream. The patches were updated against the latest libdbi versions.

You can find these repositories at the BalaBit's git server, more precisely:

git://git.balabit.hu/bazsi/libdbi.git
git://git.balabit.hu/bazsi/libdbi-drivers.git

The "master" branch contains the direct import of the libdbi CVS tree, our fixes are in the 'upstream-fixes' branch. This setup will make it easier for me to publish patches and regularly rebase the not-yet-merged set against the latest upstream.

Among other small things, you can find a quite important patch against the Oracle driver. Without this patch Oracle 10.2 (the server!) segfaults and dumps core. So beware.

SFTP proxy

2008-02-18T22:33:00.001+01:00

I installed Google analytics on this blog, and as it seems a number of people come here looking for "SFTP proxy", because of an old article I posted last July. Those interested primarily in my syslog-ng related articles may skip this post as this contains completely unrelated information, others please read on. :)

For those who don't know: SFTP is a file-system sharing protocol running on top of SSH. It is not yet an IETF standard, however more and more enterprises replaces the aging FTP protocol for SFTP. The reasons are numerous:

FTP uses plain text passwords,
FTP uses multiple TCP connections for file transfer,
FTP has inherent problems like bounce attacks,
FTP does not encrypt traffic,
FTP only supports filesystem metadata (last modification time, etc.) via extensions
and others.

All-in-all SFTP is newer, shinier and designed better. There's one problem though: SFTP uses SSH and SSH is encrypted. But wait, I said this is a drawback for FTP. Right, using encryption is good and bad at the same time. Good, because it prevents eavesdropping, bad because it cannot be controlled by security devices at the network perimeter.

Sometimes is it quite useful to see what's going on in a traffic crossing the network borders: you can restrict the usage of SFTP to a set of trustworthy clients, not for everyone. And even them can be controlled by enabling a full transaction log.

If your enterprise allows FTP traffic, there are tools to log FTP transfers, and in extreme cases to log actual data. For SFTP this is not so simple, once you permit outgoing port 22 (used for SSH), complete file system sharing can cross your firewall without you noticing. Scary, eh?

There are currently two solutions for this problem:

Disable SSH and use FTP instead. This has the drawback that passwords travel in unencrypted form, and the traffic itself is easily sniffable.
Use something like our Shell Control Box product, it is based on Zorp, with a complete SSH man-in-the-middle implementation, controls various SSH channels, limits what can get through, can log transaction data, and furthermore: at the end of the day the transmitted data is still encrypted on untrusted networks.

SCB is not using any of the OpenSSH code, it is a complete reimplementation of the SSH protocol stack, and because of Zorp all of it can run transparently (even in bridge mode) working in concert with your other firewalls/security devices.

So if you need to install proper SFTP controls, be sure to check it out.

syslog-ng feature sheet

2008-02-13T20:55:00.000+01:00

We were asked to publish some more detailed "syslog-ng feature sheet". Albeit it might go into syslog-ng specific details we tried to be as generic as possible. And certainly everyone doing such feature sheets is biased, just as we were :)

It is available at http://www.balabit.com/network-security/syslog-ng/features/detailed/.

Redesigning syslog-ng internals

2008-02-08T09:17:00.000+01:00

As promised earlier on the mailing list, I am designing the new message rewrite capabilities in syslog-ng.

As you probably know, syslog-ng currently supports message templates for each destination, and this template can be used to rewrite the message payload. Each template may contain literal text and macro references. Macros can either expand to parts of the original message or parts that were matched using a regexp.

Here's an example:

destination d_file { file("/var/log/messages" template("<$PRI> $HOST $MSG -- literal text $1\n")); };

The example above uses the format string specified at template to define the log file structure. The words starting with '$' are macros and expand to well defined parts of the original message. Numbered macros like $1 above are substituted to the last regular expression matches, all other characters are put into the result intact.

While this functionality is indeed useful, it is somewhat limited: you cannot use sed-like search-and-replace functions that some of the users requested.

My problem with rewriting message contents was somewhat fundamental: my original intention was to keep message pipelines independent from each other. If the message would be changed while traversing one pipe, this change would be propagated to the pipelines processed later.

This behaviour is sometimes desirable, sometimes directly unwanted. In the case of anonymization the changes would have to be global, e.g. all log paths would receive the anonimized messages, but if you want to store an unanomized version of the logs for troubleshooting, you want the original message, not a stripped version.

The solution I came up with is to generalize the log pipeline concept. Currently a pipe connects one or more sources with one or more destinations with some filtering added. In the new concept everything becomes a pipe element:

a filter is a pipe that either drops or forwards messages
a destination is a pipe that sends the message to a specific destination and the forwards the message to the next node

The current log statement becomes a pipeline:

source -> filter1 -> filter2 -> ... -> filterN -> destination1 -> destination2 -> ... -> destinationN

Each pipeline may fork into several pipes, e.g. it is possible to do the following:


                                                 destination1 -> destination2 -> ... -> destinationN
                                                /
source -> filter1 -> filter2 -> ... -> filterN -
                                                \
                                                 destination1' -> destination2' -> ... -> destinationN'

This is still nothing new, but consider this:


                                              destination1 -> destination2 -> ... -> destinationN
                                              /
 source -> filter1 -> ... -> ... -> rewrite -
                                             \
                                               destination1' -> destination2' -> ... -> destinationN'

This means that rewrite happens before forking to the two set of destinations, they both receive the rewritten message. However if the user had another global pipeline in her configuration, it would start with the original, unchanged message.

In syslog-ng configuration file speak, this would be something like this:


log { source(s_all); rewrite(r_anonimize);
          log { filter(f_anonimized_files); destination(d_files); flags(final); };
          log { filter(f_anonimized_rest); destination(d_rest_log); };
};

log { source(s_all); destination(d_troubleshoot_logs); };

E.g. you can have log statements embedded into another log statement, log statements at the same level receive the same log message, and have retain the power of filters and log pipe construction at each level.

Not to mention that message pipelines are a natural place for paralellization, e.g. each log statement could be processed by a separate thread, which becomes necessary if the message transformations become CPU intensive.

Whew, this was a long post, expect another post about the message parsing capability I basically finished already.

syslog-ng OSE 2.1 released

2008-01-28T21:27:00.000+01:00

I have just uploaded the first release in the syslog-ng Open Source Edition 2.1 branch to our website. It is currently only available in source format at this location:

http://www.balabit.com/downloads/files/syslog-ng/sources/2.1/src

This release synchronizes the core of syslog-ng to the latest PE version and adds the SQL destination driver.

This is an alpha release and thus might be rough around the edges, but it basically only contains code already tested in the context of the Premium Edition. The SQL functionality requires a patched libdbi package, which is available at the same link. We're going to work on integrating all our libdbi related patches to the upstream package.

If you want to know how the SQL logging works, please see the Administrator's Guide or our latest white paper Collecting syslog messages into an SQL database with syslog-ng. The latter describes the Premium Edition, but it applies to the Open Source one equally well.

syslog-ng roadmap 2.1 & 2.2

2008-01-11T21:53:00.000+01:00

We had a meeting on the syslog-ng roadmap today where we decided some important things, and I thought I'd use this channel to tell you about it.

The Open Source Edition will see a 2.1 release incorporating all core changes currently in the Premium Edition and additionally the SQL destination driver. We are going to start development on the 2.2 PE features, but some of those will also be incorporated in the open source version:

support for the latest work of IETF syslog protocols
unique sequence numbering for messages
support for parsing message contents

Previously syslog-ng followed the odd/even version numbering to denote development/stable releases. I'm going to abandon this numbering now: the next syslog-ng OSE release is going to have a 2.1 version number and will basically come out with tested code changes only.

The current feature set in PE were developed in a closed manner and I don't want to repeat this mistake. The features that were decided to be part of the Open Source version will be developed as openly as possible: the features listed above are going to be developed and published in the open source branch with version number 2.2.x. The "alpha" and "beta" releases are going to be numbered 2.2alpha1, 2.2beta1 etc; the final stable is going to be called 2.2.1.

The aim is to avoid the mess of PE and OSE having a different version number.

syslog-ng fun with performance

2007-12-28T15:31:00.000+01:00

I like christmas for a number of reasons: in addition to the traditional "meet and have fun with your family", eat lots of delicious food and so on, I like it because this is the season of the year when I have some time to do whatever I feel like.

This year I felt like doing some syslog-ng performance analysis. After reading Ulrich Deppert's series about stuff "What every programmer should know about memory" on LWN, I thought I'm more than prepared to improve syslog-ng performance. Before going any further, I'd recommend this reading to any programmer, it's a bit long but every second reading it is worth it.

As you need to measure performance in order to improve it, I wrote a tool called "loggen". This program generates messages messages at a user-specifyable rate. Apart from the git repository you can get this tool from the latest syslog-ng snapshots.

Loggen supports TCP, UDP and UNIX domain sockets, so really almost everything can be measured.

Then I've put together a test environment, which consisted of a 4-way Opteron box as a server (two dual-core CPUs at 2.6GHz and 1MB of cache), where the syslog-ng center ran in 64 bit mode, and a venerable P4 Xeon 2.4GHz as client. I verified that the client was more than capable of saturating a 100MBit link that was used to connect the two boxes. Then I've installed syslog-ng on the server, using the simplest configuration possible: fetching messages from the TCP/UDP socket and writing everything to disk into a plain, text file without macros in the filename.

Syslog-ng 2.o OSE performed somewhat better than I had anticipated. When using TCP it could successfully process messages at about 44000 messages/sec without losing a single message. Each message was 150 byte long (I've started with 200, but the 100MBit link proved to be the bottleneck) Some funny findings:

Enabling flow-control did not really make the results worse.
Increasing log-fetch-limit to a large number (10000) made the results worse.
Using the Glib GSlice allocator instead of malloc/free didn't improve the numbers.

Of course once you have a baseline, there are a lot of possibilities to try how that given change affects performance, but I only had a day :)

I've found some things that improved performance even further, the most important bottleneck was the time related functions in libc (localtime, mktime, strftime, etc.) For some reason they reread /etc/localtime upon every invocation. I'm going to file a ticket in their bugzilla as it's completely unnecessary to do that, especially if the value of the TZ environment variable does not change.

At the end of the day I finished with syslog-ng chewing messages at around 68500 messages/sec which is a 55.9% improvement. I can see some further possibilities, but I doubt I could increase performance over 75000 msg/sec. This means that syslog-ng can process messages at about wirespeed of a 100MBit/sec ethernet link. (68500*150 = 10275000 bytes/sec)

I was very satisfied at this point, even explained my findings to my wife and my elder brother :)

This of course does not apply to legacy UDP based syslog traffic directly, unless a really large socket buffer is set for syslog-ng. I'd say that you need 3-5 seconds worth of receive buffer in order to avoid losing messages, which with the above rate would be about 30MB-50MB of non-swappable kernel memory.

These changes were committed to the Premium Edition of syslog-ng, although the loggen program is GPLed, so anyone can do performance testing their own setup/configuration.

syslog-ng stuff and christmas

2007-12-22T00:50:00.000+01:00

We've been busy recently on the syslog-ng front. A new release came out from both the Open Source and Premium Editions, covering various bugfixes, the most important being a fix for an easy denial of service. Please upgrade to at least OSE 2.0.6 and PE 2.1.8, my bugtraq posting has more details.

A new release from the syslog-ng documentation was also published, it contains a new chapter on losing messages and another one which explains the "Log statistics" message generated by syslog-ng.

And since, I'm writing this entry I wanted to say "Merry Christmas" to all the readers of this blog:
Marry Christmas and a Happy New Year.

I'm spending the winter holiday with my wife and I'm going to travel a lot around the country to visit relatives. So I'm not sure I can be very productive in the coming days.

syslog-ng disk based buffering

2007-12-07T11:50:00.000+01:00

I've just seen a post on the loganalysis mailing list how easy it is to implement disk-based buffering with perl and a few hours time. The implementation would be as simple as sending the messages to a file and using a script like "tail -f" to follow the file and send messages to the desired log collectors.

Although the scheme that was described would work, I see three important problems:

latency: because the solution would work by polling the log file, the latency is severely increased, when you have thousands of log entries per second, a second is a long time. And you don't want to poll more often than every second.
disk usage: relaying the data would store everything on the local disk, no upper bound on disk usage, if the disk is full, data is lost
load: using an interpreted language and the requirement to store all data on disk puts an enormous load on the system that might be spent better elsewhere.

Syslog-ng on the other hand uses a spool file, but this file is only written when the memory based buffer becomes full. For the generic case all messages come and go, without touching the disk at all or having to poll the disk for changes. The disk space requirements are bounded. Disk buffering with the combination of flow-control and disk buffers you can avoid message loss.

And what's more, the disk spooling in syslog-ng is an independent feature for all flow-controllable destinations: tcp(), unix-stream(), pipe(), program() and also sql().

Adding a disk buffer to a destination is as simple as specifying the buffer size in bytes:

destination d_tcp { tcp("logserver" log_disk_fifo_size(100000000)); };

This means that a 100MB of space is allocated for disk-based spooling to store messages whenever "logserver" is not fast enough or is unavailable.

For more information read the syslog-ng documentation about the way this feature works.

IBM System i

2007-12-01T08:45:00.000+01:00

I was learning computer programming while at secondary school, at as a school assignment we had to create an accounting system on a venerable IBM 360. We had 386dx computers back then with 2MB RAM, the first Linux encounter at this time required me and my friend to put all of our RAM modules to a single computer to get 4 megs and make it suitable for Linux.

This 360 machine had two CPUs, each the size of a large office desk, speed comparable to an 386, had 8MB of RAM. After getting used to the speed it provided (had 8 terminals on a 2400 baud modem) it was fun, completely different to the PCs we've had, but still, interesting.

Some of this memories were brought back by our partnership with Patownsend & Associates, a US based company to deliver System i (formerly AS400) and System z (the mainframe) software. Our partnership focuses on the development of a Syslog agent for System i, this way we could extend our syslog-ng Premium Edition offering with support for IBM midrange servers.

AS/400 (or System i), is a very interesting platform, somewhat reminding me to the IBM mainframe at secondary school. Text display on tn3270 terminals, all commands always starting with the letter 'Q', filenames condensed to a point where it is difficult to understand their meaning (QAUDJRN anyone? :) ), but at the same time a very consistent and very use-case oriented interface. The system is very complete, while a UNIX contains the stream-of-bytes file concept, OS/400 supports many different file types, for instance there's a file type of SQL tables, a complete DB2 is integrated into the OS itself.

It was a great experience to work with the Patownsend guys, knowledgeable people, who know what they are doing. I'm looking forward to our cooperation.

public bugzilla

2007-11-07T21:16:00.001+01:00

I've started creating a public bugzilla installation for syslog-ng in the last couple of days, however I was interrupted before being to finish it all.

I want to move all opened internal tickets to the public installation, but I only managed to review half of them. Not that it is a large task, I just got distracted all the time by my fellow collegues and customers. But that's the life of a software developer, right? :)

You can find this at http://bugzilla.balabit.com/ , although it is not yet officially announced. Once I'm done with reviewing/translating tickets, I'll send an official announcement to the mailing list.

syslog-ng git repository moved

2007-10-10T21:28:00.000+02:00

As I announced on the syslog-ng mailing list, the official syslog-ng git repositry has moved and has become accessible using the more effective "git" protocol, instead of plain HTTP.

The change has been described on the syslog-ng webpage, gitweb is also available.

syslog-ng HP-UX updates

2007-09-22T16:57:00.000+02:00

I've just pushed out an update to syslog-ng, which contains various HP-UX fixes backported from the Premium Edition of syslog-ng.

The GPL version of syslog-ng should now work flawlessly on HP-UX. There's one caveat though: there's a buggy system header in HP-UX and gcc 4.x fails to compile it. I copied this header to the gcc private include directory (/usr/local/lib/gcc/hppa2.0w-hp-hpux11.11/4.1.0/include on my system) and applied the following patch:

438a439
> #ifndef _APP32_64BIT_OFF_T
442a444
> #endif

(HP-UX diff does not know how to produce unified diffs), the point is that there are two conflicting declarations of a function and the preprocessor conditional above fixes that. Once this patch is in place, and you have the proper build dependencies syslog-ng works fine on HP-UX.

As I've just pushed these changes to my git repository, you'll need to wait for another day to get a daily snapshot. But hey "git" is what the pros use :)

Reasons of my silence

2007-09-15T19:02:00.000+02:00

Apart from the previous entry I was not posting to this blog for two months. The reason was that I was heavily involved in the development of BalaBit Audit Player, a graphical application to replay RDP/SSH sessions, recorded by our SCB product.

BAP became much larger than I originally expected, it's about 20k lines of code, and the end of the development was done in a rush to meet our deadline of Sep 1, 2007. We've slipped a couple of days, but we've released BAP 2.0.0 on 7th September. Then I spent a week in Karlsruhe on the 5th Netfilter Developer's Workshop.

I returned to Hungary on Friday, I'm spending the weekend with my parents, and hopefully I can be more active on other things, like the syslog-ng mailing list, or this blog. :)

A release of syslog-ng GPL is long due, hopefully I can prepare it next week. I'll also need to schedule some syslog-ng development time as there are some open feature requests by customers.

Netfilter workshop

2007-09-15T18:48:00.001+02:00

I've just returned from this year's Netfilter Developer's Workshop, this time held in Karlsruhe, Germany. This year's workshop was the fifth such event, and this time even David S. Miller was there.

All in all the organization was wonderful, kudos to the Astaro guys. We've had about 30 attendants, the largest workshop ever. You can read more about the workshop at http://nfws.inl.fr/en/

Krisztian Kovacs and me were trying to push our TProxy4 patches for merging, the future for tproxy seems bright, as everyone was positive.

We also have some other, minor patches in the queue, I'm working on finalizing them and submit them for inclusion. These are:

SO_MARK socket option to be able to specify the mark field of outgoing packets, generated sent from a given socket
an extension to the "addrtype" match to limit the match to the incoming interface
the notion of interface groups, that make it possible to match a group of similar interfaces

As always, it was very nice to meet Netfilter people, let's hope we meet in Paris next time. :)

SFTP proxy

2007-07-28T23:44:00.000+02:00

I was spending the last couple of hours to implement a simple SFTP proxy, that is capable of logging file transfers, into our Shell Control Box product line. The core idea behind SCB is to perform RDP/SSH screening independently from the end-systems. This SFTP functionality will be a small bonus: in addition to dumping the SSH traffic to an audit trail, we are going to be able to write log transactions to syslog, which is way easier to analyse, if all you want to know is the list of files accessed via SFTP.

I originally thought that SFTP was as simple as FTP, with a transaction being a complete file transfer.

On the contrary, SFTP is much closer to NFS (and other network file system protocols) in spirit: in FTP you have a "RETR" command that fetches a complete file, in SFTP you need to open the file and read it separately using a series of "READ" commands.

Now I understand how sshfs is possible. I thought I'd let you know :)

By the way, syslog-ng 2.0.5 has been released recently. Hopefully this will decrease the stream of "Syslog-ng does not compile, please help" complaints, which was caused by my lazyness to enable spoof-source support unconditionally by default, without writing a proper configure.in test whether libnet was present on the system.

Syslog-ng status

2007-07-22T10:41:00.000+02:00

Things were progressing steadily on the syslog-ng front. About 4 weeks ago I released syslog-ng Premium Edition 2.1.5 which was the first version with integrated SQL support. Since then the first production deployments of PE have been done, shaking out some newly introduced bugs in the process (thus the releases 2.1.5a - 2.1.5d).

Disk buffering works nicely, especially when combined with the new rate-limit option (throttle), if your backend systems are calibrated to a given rate of incoming messages, syslog-ng can ensure that the limit is never exceeded. Whenever bursts end, syslog-ng feeds the messages towards the back-end systems in their idle time. This introduced some latency though.

I've started working on support for the new syslog-protocol work in IETF, but nothing is ready yet. After this is finished, I'm planning to release syslog-ng 2.1 under the GPL license, with some of the new features added.

The GPLd branch also saw a couple of fixes, no release though. If you want the latest set of fixes, please use a daily snapshot.

Latest happenings

2007-05-26T19:14:00.000+02:00

I forgot to mention here that syslog-ng 2.0.4 was released about 10 days ago. Before making an upgrade, be sure to revalidate your max-connections() setting. As previous versions in the 2.0.x tree has failed to enforce this limit.

Apart from the usual bunch of bug fixes, this release features case-insensitive regexps and the ability to track logrotated source files. Now you can actually track any log file, even if it is rotated automatically.

We have also worked hard on the new syslog-ng webpage. Hopefully no information was lost during the transition. If you miss anything that was present on the old one and not available on the new, please let me know.

I'm planning to set up a public bugzilla and a wiki for syslog-ng, so that it will have the standard infrastructure needed for a modern open source projects. Version control, bug tracking and Wiki.

On the Premium Edition front, both the syslog-ng Agent program for Windows and syslog-ng has seen a public release. Check out the syslog-ng Premium Edition pages for more information.

syslog-ng database support and other fixes

2007-04-30T01:14:00.000+02:00

As the readers of this blog might know I've been working on persistent disk-buffering and SQL support recently. The configuration interface of the SQL destination became quite close to the description I gave in my last post.

I think from the user side it is pretty neat, no need to rely on the mysql client program or to create "buffer files" that are later fed to the database server. You simply define an SQL destination and tables are created automagically with proper (possibly even disk-based) queueing, flow control and error handling. And by using libdbi we immediately have support for 4 or 5 different database servers.

The implementation side was somewhat more intrusive however, the client libraries for various database servers use a blocking I/O model and syslog-ng was completely non-blocking until now. I had to create a separate thread for inserting records to tables, and there came some required changes to syslog-ng to support that: various reference counters and the internal acknowledge mechanism had to be made thread-aware.

This was not without benefit though, a change of this scale required a thorough review of the code involved and I have found and fixed several bugs that also affected the 2.0.x tree. I have committed some of these to my public git tree already, so it should be synchronized to our public webserver real soon now.

My next thing to do is to prepare 2.0.4 and probably a first public release of the 2.1.x open source tree. Stay tuned. :)

Persistent disk-buffering in syslog-ng

2007-04-24T08:27:00.000+02:00

Two blog spots in a row on two consecutive days, wow :) Things are happening fast these times. Because of my company's efforts to create a commercial fork of syslog-ng, I have somewhat more time to do syslog-ng hacking. This time I've finished persistent disk-buffering, an often requested feature to be released in the commercial version.

It means that if a target server is down, then in addition to the in-memory buffers, syslog-ng is able to store messages in a disk-based queue until the connection is restored. What's more, this queue is persistent and syslog-ng keeps its contents accross restarts.

One less reason to keep logs locally. :)

Next on my list is native SQL support, combined with this disk-buffer feature to cover times when the database is too slow processing INSERTs. I'm thinking along the lines of:

destination d_sql {
sql(type(pgsql) host("loghost") user("syslog-ng") password("secret-password")
database("logs")
table("messages_${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")
columns("date", "host", "program", "pid", "message")
values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY"));

};

Tables would be created/altered automatically on-demand, just like destination files. Values might refer to builtin macros, or matches within the message (like in sed, e.g. $1 refers to the first match).

The actual implementation might be somewhat different though.

Switching version control systems

2007-04-22T16:58:00.000+02:00

We have been using GNU arch the last couple of years as a version control system, however Tom Lords' implementation does not scale well, and some of our software packages have 10 thousands of commits. This means that a single commit operation may take _minutes_. It is awful to wait so much time for a single commit, and it really degrades productivity.

I was considering Mercurial, Bazaar-NG and git, however this was not an easy decision, as the "modern" version control systems promote the use of branches over anything else, and our current version control model relied on cherry-picking heavily:

developer commits the solution for each bug separately to his/her branch
QA people pick patches from developer branches and integrate them to a 'test' branch, once the test was successful,
release manager picks patches from the 'test' branch and integrates to mainline, if he doesn't find anything odd during review

This worked wonderfully in GNU arch, but new VC systems lack in this area. Bazaar has no cherry picking support at all, Mercurial has some incomplete support with a plugin named transplant, git has cherry picking, but that relies on heuristics (it guesses whether a patch was integrated by using a checksum of the patch).

I was considering to change the process I outlined above, but I'm not sure how that would work out. We sometimes need to work with people not really experienced with VC systems at all, asking them to manage their own branches for each bugfix/problem group seems to raise the bar a bit too high.

Nevertheless git seemed to have solutions for both worlds (e.g. picking patches AND merging branches), so I choose git over the other two, and now I converted some of the syslog-ng history to git in order to gather some real-life experience.

I like what I see so far, git 1.5.x is really way better than older versions on the usability and documentation front. I now feel comfortable enough with git as I could finally understand the working model and the structure of the git history.And git is fast like lightning :)

syslog-ng 2.1 is branched

2007-03-26T19:09:00.000+02:00

I've just finished preparations for the release of syslog-ng 2.0.3 and now I consider the 2.0.x branch feature complete. As far as I know there's no feature that was present in 1.6.x and missing from the 2.0.x rewrite. The last missing bits were spoof-source and TCP wrapper support, but as those are present in 2.0.3 I consider syslog-ng 2.0 feature complete. I don't plan to add further code that would cause destabilization.

Now some plans about the newly opened 2.1 branch: we (as BalaBit) played with an idea of creating a commercial fork of syslog-ng which could increase the amount of resources that I can allocate to developing syslog-ng code. From 2.1 on, two parallel editions of syslog-ng will become available:

syslog-ng 2.1 Open Source Edition: the same as syslog-ng 2.0, available under the GPL
syslog-ng 2.1 Premium Edition: a commercial fork of syslog-ng,

The two releases will be produced from the same source tree with some of the commercial functionality stripped during the release process. This ensures that bugfixes/changes that are developed in the branch will automatically be released with the open source edition as well.

In the first release, the difference between the two editions are as follows:

TLS support,
Support for a Windows agent, capable of sending log messages in syslog-ng's extended log format and TLS encryption.
New, updated manual (also covering the Open Source Edition).

spoof-source added to 2.0.x

2007-02-03T18:27:00.000+01:00

As a commenter missed spoof-source support from 2.0, I got my act together for an afternoon hacking session and implemented it in the 2.0.x tree.

The packet generation for IPv4 packets was straightforward, I could simply "forward-port" it from 1.6.x, after creating the proper place for it in the 2.0.x tree. IPv6 support was a bit more difficult as libnet has a nasty bug with IPv6 and UDP. I was looking at my code for hours, when I tried Google codesearch to check whether I was doing something wrong. Then I've found this. Note the comment above the libnet_build_ipv6() call. I also disabled UDP checksumming and now it works like a charm. The last libnet release was almost 3 years ago, it is not reassuring that its webpage is also down. Too bad, libnet is a fine piece of software, but now as its website is down where can I point syslog-ng users to download libnet from?

By the way, I did not announce syslog-ng 2.0.2 here, it was released about a week ago. Nothing fancy, primarily portability fixes and a more important fix in the usertty() destination. Grab it from the usual place.

Long time no post

2007-01-30T07:17:00.000+01:00

It's been a while since I last posted here. The reason is simple: work and the lack of time. OK, I know, people have time for what they want, but:

* We are launching a nifty new product based on Zorp, but with a much tighter focus. A device capable of recording and replaying SSH sessions for audit purposes.
* We have decided to migrate from C to C++ for our management interface, as it seems Gtk+ style OO programming in C really requires a skill, and a lot of developers don't have that. C++ holds their hands somewhat. Thus, I've been refreshing my C++ skills and learning gtkmm while writing a prototype.
* and of course the "usual" amount of work.

I've tried to provide timely response on the syslog-ng mailing list. I'm sorry if you didn't receive an answer/patch from me in time.

By the way I released syslog-ng 2.0.2 yesterday. Nothing really important, some portability fixes, some usertty fixes and an option missed in the 1.6.x -> 2.0.x change.

Another good news is that Fedora seems to be migrating from syslogd to syslog-ng in Fedora 7. That's simply great :) Guys, if you need help, just drop me a line.

syslog-ng 2.0.1 released

2006-12-22T20:09:00.000+01:00

I have released syslog-ng 2.0.1, available at the usual places. I have added various missing bits that fell out in the 1.6.x -> 2.0.x change. These include:

DNS cache support,
overwrite_if_older (used to be called remove_if_older),
various fixes,

All-in-all 2.0.0 was a successful release, hopefully this one will not be much worse. See the NEWS entry of 2.0.1 for more information.

Interview with me on linux.com

2006-12-14T09:31:00.000+01:00

Just a short note, an interview with me was published on linux.com, more specifically here.

2.0.0 experiences

2006-11-08T21:21:00.000+01:00

As it turned out the 2.0.0 release was not so bad after all. At least I have not received show-stopper bugreports, which either means that noone is using it, or everything is fine and dandy :). Hopefully it is the latter, rc releases were tested by a few people.

In the meanwhile I started adding a few missing bits that were still present in 1.6.x but I never got around to implementing in the 2.0.0 tree. Among them I readded the remove_if_older() option. By the way, I don't really like the name of this option, does anyone have a better idea? If you do, please put it in a comment here or send me an email. (I was thinking about retention_time() but I'm afraid it is more difficult to understand what it would do)

The other bit is the new/shiny DNS cache, which also supports persistent entries. This means that syslog-ng can read your /etc/hosts file, resolve IPs that are present there, and use IP addresses for anything else. This removes the dependency on DNS, and should also improve overall performance.

So all in all, syslog-ng 2.0 is in a good shape, give it a try. Testing the latest snapshots, especially the new DNS cache parts, would be appreciated.

Syslog-ng 2.0.0 released

2006-11-04T22:27:00.000+01:00

You might have already noticed, but I thought I'd write an entry on this blog on this topic: the wait is over I have released syslog-ng 2.0.0. It took a bit longer than I have anticipated, I needed to prepare 4 rc releases, as each had some bug here or there and I really wanted to release a stable 2.0.0

I hopefully succeeded, no breakage reported so far. 2.0.0 was released last saturday, but the announcement went out only on Friday.

Some rarely used functionality is still missing though (a prominent example is spoof_source), and I already committed the remove_if_older() option after the release of 2.0.0. I'm going to concentrate on filling the missing bits.

I'll try to avoid committing to 1.6.x, migrating to 2.0.x is strongly recommended.

Catching up with things

2006-10-04T22:36:00.000+02:00

It's been a long while since my last post here, but I was really busy in the past months and after I left for a two weeks vacation to Corsica. I returned on Sunday, started to catch up with work and such but I still was not able to read my syslog-ng mailing list folder, containing almost 100 unread messages. Please be patient, if you have a question open and forgiving in the unfortunate case I'd forget to reply.

On the other hand Corsica is a beatiful island, be sure to visit it if you can. Nature is almost untouched at a couple of places, everything is green and a lot of mountains. A pair of hiking shoes is a useful item if you are in Corsica. :)

So the island is beatiful we had some minor nuances with waiters, as neither me or my wife speaks French, and this seems to be a sin in the eyes of Corsican waiters. So at the end we came up with cooking for ourselves, lucky us our apartment was nicely equipped with cooking gear.

Back to syslog-ng, I'd really like to release 2.0.0 now. I released 2.0rc3 right before I left, if it had no problems in the last two weeks, it should be a reasonable 2.0.0 release. But I first need to read the 100 unread
mails on the topic :).

Some kernel hacking

2006-08-01T19:45:00.000+02:00

After some time I needed to do some kernel coding again. To seamlessly support dynamically created interfaces in Zorp, we need something I called "interface groups".

Each interface might belong to a single group that basically describes how the interface was created. For instance there's an interface group for each PPP profile, but an interface group can encapsulate interfaces created by PPTPD.

It is quite difficult to match dynamic interfaces by their nature: iptables sports wildcard interface name matching with the '+' character but it only works if interface names have some kind of prefix _AND_ if you don't want to differentiate between two groups.

If you have two sets of PPP devices (like in the example I described above), then you have no way to create a separate ruleset, unless you reload iptables everytime a new interface is added to the system.

Adding to the burden, in Zorp we want to be able to bind a service to these dynamically created interfaces, of course without listing the actual IP address in the configuration file.

The idea is simple, I added an "interface group ID" to the net_device struct, and an option to the "ip link" command to set/query this ID. Once an interface is created by some kind of program (for instance pppd), a script is executed in its /etc/ppp/ip-up.d directory and userspace can assign a group ID based on the PPP profile name. Then Zorp gets notified about the change through NETLINK and can react by binding to the IP address of the new interface. The configuration remains static, no reloading needs to be done when such a change happens, and you can create firewall policies for something like: please allow this set of services for everyone using this PPP profile, without entering one specific IP address to the configuration. Neat, eh?

I posted my work on netdev and netfilter-devel, I'm curious what the kernel maintainers will think about it.

syslog-ng 2.0rc1 released

2006-07-09T11:35:00.000+02:00

After my last requests for testing of the latest 1.9.x code base, I have received a couple of bug reports, which were fixed in recent weeks. Since I have received no reports the past two weeks I decided to name the new release as "2.0rc1" to raise awareness of the new codebase.

I'm planning to create the new branch for 2.1.x, I have some exciting features in my mind, which I did not want to start before the release of 2.0.0. The old stable series 1.6.x is still supported, but expect less development time to be dedicated in maintaining that release.

Build queues for various architectures are not yet up, so only a Debian sarge binary is available for those with binary maintenance contracts.

Thoughts on the patent system

2006-07-07T20:45:00.000+02:00

You might know that there is a standardization effort on the syslog protocol in the IETF. The work has started several years ago and the efforts produced RFC3164, the first documentation of the BSD syslog protocol after being in use for over two decades.

This group also produced RFC3195 in 2001, a reliable syslog protocol using the BEEP framework which did not really take of. I personally did not implement this in syslog-ng due to its highly verbose nature and the complexity which BEEP brings in.

Couple of months ago an effort started to create a simpler, but still reliable syslog protocol somewhat similar to what syslog-ng has been using for a couple of years now. First some layering was decided, e.g. to define the syslog protocol in a transport independent manner and then define various transports, like legacy UDP and TLS encrypted TCP.

After syslog-transport-udp was written by Rainer Gerhards, work has started on the TLS encrypted transport and someone from Huawei (you know the Chinese Cisco clone) volunteered to write the draft, which he did with the help of other group members. Basically the contents of the ID represented consensus (a rare event in the syslog group) and was heavily based on the previous years' work.

The ID was published and we were finally approaching a standardized syslog over TCP protocol, everything was nice and dandy.

Except that a few weeks later Huawei published a patent claim on the contents of the published ID, they basically said that they have a not-yet-published patent pending which covers at least parts of the ID. It is yet to be determined which sections of the internet draft is affected, but as far as I know it takes several months till this information is going to be available.

So what now? Basically I don't know, some prior art is certainly available, I personally found articles describing the combination of syslog-ng TCP transport and stunnel. Even if the patent will not be granted, the work of the working group is endangered by patent threat.

Did I mention already that I don't like US style patents? I'm happy to live in Europe, we are still not affected,
assuming that syslog-ng is developed and used within Europe. In the US, even end-users can be threatened if they use a product that uses protected technology and which does not license the patent.

I need to make a difficult decision:

avoid using the recent work of the working group and fall back to using an updated version of RFC3195, OR
don't care about the IPR claim Huawei has published in the hope that the patent will not be granted.

How would you decide?

Getting married

2006-06-20T22:30:00.000+02:00

This is just a quick note that I'm not yet lost, I'm on holidays for two weeks, as I'm getting married this weekend, or to be more precise 24th June, 2006.

I'm returning to work on 3rd July, 2006. As my internet connection is not perfect, I might not be able to respond to e-mail messages timely.

Syslog-ng 2.0.0 release date

2006-05-24T19:40:00.000+02:00

It was just a week or two ago when someone asked me about the planned release date of syslog-ng 2.0.0, the first stable release of the third incarnation of syslog-ng. Probably I did not even respond to the email as I did not know the answer. "When it's ready" is an answer users do not usually perceive very well.

It is very difficult to judge when a rewrite of such a critical software package is stable enough for production use: I wrote both functional and unit tests, used syslog-ng on my laptop for over a year now, but as I currently lack a system were non-production code can be uploaded, syslog-ng was drifting slowly in the stabilization process: whenever someone reported a bug, I fixed it.

So the release date in the current state is determined by the syslog-ng user community and not me. IF there's certain confidence that a pile of code runs fine, it can be tagged stable and everyone can be happy. If there is no feedback, an optimist might think that everything is going fine, the pessimist would say that nobody is using the product.

My point is that positive feedback is _VERY_ important, it is an indication that people are using the code, but have no problems.

syslog-ng 1.9.x is currently in feature freeze, I don't plan to do anything that threatens stability, but this also means that people waiting for things like message rewrite capabilities need to wait until syslog-ng 2.0.0 is out of the door. And the key to that is YOUR participation: download the latest release, try it and report back. Even, if it works. Especially if you are not running Debian, which I happen to run on my notebook.

Thinking about rewrite rules

2006-05-19T23:59:00.000+02:00

Again the question on Solaris message IDs was raised in an email sent to me in private. For those who don't know how a Solaris msgid looks like, look at this example:

May 14 18:51:57 inbound2 su: [ID 366847 auth.notice] 'su root' succeeded

I was asked to include an MSGNOID macro which excludes this msgid in the final destination. The problem I have with this approach is that it simply does not scale: there are simply too many combinations to cover with various macros, an example using the msgid case:

a macro that includes neither the name of the program, nor the msgid
a macro that includes program name only
a macro that includes msgid but not the program name
a macro that includes both the program name and the msgid

As you can imagine this quickly becomes a maintenance nightmare even if one finds out a proper name for all of these combinations, especially if you add that other devices have their own extensions to syslog.

What I am pondering is to renew my old ideas about adding sed-like rewrite rules to syslog-ng, something along the lines of:

rewrite r_msgid { msg("s/\[ID [0-9]+ [a-z]+\.[a-z]+\]//"); log { source(s_local); filter(f_noid); rewrite(r_msgid); destination(d_messages); };

Of course similar functionality would be added to manipulate all syslog message parts, like hostname. The results would become part of the message itself, thus macros would use the rewritten message. And by the way backreferences could be used to refer various parts of the message, matched by regexps.

What do you think?

syslog-ng 1.6.11 released

2006-05-06T09:52:00.000+02:00

I have released syslog-ng 1.6.11 which fixes the problems outlined in the previous post. You can find it at the BalaBit website.

syslog-ng 1.6.10 broken

2006-05-02T21:34:00.000+02:00

Just a quick one, it turned out that syslog-ng 1.6.10 is broken in several ways, first reading messages from /proc/kmsg is broken, and second the time_sleep() feature that was added in 1.6.10 has missed an important chunk from the parser code which made time_sleep() unconfigurable.

So a feature that cannot be used and an important problem. :(

I'm going to release syslog-ng 1.6.11 soon.

Infosec in London

2006-05-02T21:16:00.000+02:00

I spent the last week in London, visiting InfoSec Europe. It was a great fun, I liked the exhibition as well as the city itself.

I have not been to London before (except for a single-day business trip two years ago, but that does not count), and I liked the city very much. I walked about 40-50km on these three days, I had my legs completely worn out. British people are quite strange I would say. Everything is completely in the reverse: the cars, the direction the trains arrive from, the way the taps need to be opened, I think even the screws must be unmounted in the reverse direction. I hated these non-mixing taps, one tap for cold another for hot water, no way to mix something tepid. Beside this strangeness I liked the atmosphere of the city, I visited all the important places, I even spent two hours in the British Museum, but it was nothing but a scratch on the surface.

The exhibition was also interesting, met a couple of interesting persons, like the Watchfire guys who invented HTTP request smuggling and some real computer forensics guys. We were talking about the problems with encryption vs. forensics and what possible solutions there are to this problem.

All in all it was an exhausting week.

Committed IPv6 support for syslog-ng

2006-04-23T12:49:00.000+02:00

I have finished IPv6 support for syslog-ng, I'm wondering how this will improve the number of people actually using the new syslog-ng 1.9.x tree.

In the implementation I've created separate udp6() and tcp6() source and destination drivers, because this was somewhat easier to implement. I'm expecting some portability trouble, but otherwise the implementation is nice and simple.

Some smaller fixes went in recently as well, like:

avoid chown/chmod files that do not exist as it clobbered error reporting,
added close-on-exec flag to file descriptors to avoid child processes to inherit tcp/udp connection fds,
fixed an off-by-one in flush_lines calculation,
a possible memory leak, and
a fix for non-existing filter references in the internal() message path

Apart from IPv6 support these are mainly bugfixes and I'm confident we can have a 2.0.0 real soon now.

syslog-ng and IPv6

2006-04-17T13:24:00.000+02:00

I received an email last week asking about IPv6 support in syslog-ng. The question also referred to an IPv6 application page where most of the system logging applications were listed red showing that they lack IPv6 support.

I started hacking on it but I thought I would ask you how important you think adding IPv6 support to syslog-ng is?

My idea was to add tcp6() and udp6() source and destination drivers, however a lot of applications seem to do IPv6 support with a single listener, e.g. open an AF_INET6 socket and assume that the system shares IPv4 and IPv6 sockets. Which of the two approaches is preferable? The first gives more control the latter seems to be a bit easier to use, it works out of the box.

Released syslog-ng 1.9.10 for real this time

2006-04-15T08:25:00.000+02:00

Just a quick one this time, I have released syslog-ng 1.9.10 available at the usual places. The release contains mainly bugfixes and an implementation of the previously missing netmask() filter and support for bad_hostname() and check_hostname() options.

Timezone woes

2006-04-09T22:09:00.000+02:00

As I have written in my previous post there was a timezone related problem triggered by one of the unit test programs of syslog-ng. Apart from a minor issue in the testprogram itself, it turned out that there's a timezone conversion problem at the reception of messages. Syslog-ng 1.9.x has support for messages that use the ISO 6501 timestamp. As an example the current local time here right now in ISO 6501 is: 2006-04-09T22:43:24+02:00. The important part is that it includes an explicit timezone offset. This offset is processed by syslog-ng and it can convert timezones when necessary.

I spent about half a day to fix timezone conversion, I even used a pen and a sheet of paper to do some calculations. All I can say that there's an important building block missing from the POSIX time handling functions, which would have made my job as an application developer way easier: one that converts a broken down time representation to a UNIX time_t value, where the time to be converted is NOT in the local timezone, but in GMT. The other side of this conversion exists: localtime() converts a UNIX timestamp to the local timezone, and gmtime() does the same but instead of using the local timezone and daylight saving settings, it uses GMT as timezone.

The only portable function to convert a human readable timestamp to UNIX timestamp is mktime(3), which assumes that the converted timestamp is in local time. At first blick this can be easily used in place of our imaginery mktimegm() function: mktime() returns a value offseted by the local timezone, but we also know the local timezone offset, so we substract this from the return value of mktime() and we have a stamp in GMT, right? No, not right.

There are cases when mktime() changes its incoming broken down time representation when Daylight Saving kicks in: the value of "2006-03-26 02:00:00 CET" does not exist, it is equal to "2006-03-26 03:00:00 CEST" (CET is +01:00, CEST the daylight saving time is +02:00), and this happens to every value in this time interval, e.g. 2:33 CET becomes 3:33 CEST.

Remember, I have a timestamp with an explicitly specified timezone offset where the daylight saving settings of the syslog-ng process should not count, e.g. the sender sends something like 2006-03-26 02:00:00 +02:00, which is converted to 2006-03-26 03:00:00 +02:00 by the mktime() function, e.g. it is off one hour. And all this happens only in the transition hour. Good, heh?

The solution was to check this change in the time by mktime() and adjust the returned value, this seems to work reasonably well for the transition hour.

While writing this post I have found that there is a GNU extension defined, a function named timegm(3), which seems to do exactly what I have wanted. The problem that this function does not seem to be too portable. The notes in the manpage say that for achieve timegm() functionality, the application should change its own environment, set the TZ environment variable, call mktime(), and reset the environment variable. This does not look too clean I would even call that ugly. IIRC setenv() allocates memory, I would need to call this kludge for each and every incoming message.

I think this important hole in the API should be plugged, there are a lot of applications that need to work with various timezones and I have a bet that a lot of those work incorrectly in daylight saving transition hours.

I already have one example: GNU date program also allows specifying an explicit timezone offset:

bazsi@bzorp:~$ date -d "2006-03-26 01:59:59 +0100"
Sun Mar 26 01:59:59 CET 2006
bazsi@bzorp:~$ date -d "2006-03-26 02:00:00 +0100"
Sun Mar 26 04:00:00 CEST 2006

The second one should only be one second later than the first, e.g. it should be 03:00:00 CEST, and not 04:00:00 CEST. Try it with your favourite application :)

Almost released syslog-ng 1.9.10

2006-04-06T22:55:00.000+02:00

... but at the end I didn't. I prepared the NEWS file, changed version number etc, but the end it turned out that one of my unit test program which tests macro expansions failed.

I still have not looked into the issue, hopefully it is only the test program, time related macros seem to use a bad timezone offset. Again I seem to have made a timezone related bug :(

Although timezones and time related functions seem to be simple at first, it proved to be a problematic area, it already had a lot of bugs and again here is this one. Not to mention the problem that different platforms have different set of variables/functions to cover the issue. For instance "timezone" is a global variable on Linux and a function on BSD. Linux has a "tm_gmtoff" member in "struct tm", BSD doesn't.

OK, I quit whining now :) Hopefully I'm going to have some free time to look into this bug in the nearfuture.

I also have two other issues on my radar for syslog-ng 1.9.10, first I've received some reports about missing configuration keywords (namely bad_hostnames and check_hostnames), and second I want to change some currently reserved words to identifiers, so that "kernel" can be used as the name of sources again. And oh yes, I have also received a report on an abort(), although I don't have enough info on this one yet.

One thing is certain: the 1.9.10 release of syslog-ng is coming.

SSH publickey authentication implemented

2006-04-01T22:37:00.000+02:00

I have hacked on our SSH gateway today to add publickey authentication support. By the way I may not have explained this before, so a short introduction is due: Zorp is an application layer gateway with support for 21 protocols, among them an SSH gateway capable of looking into the encrypted SSH stream and restricting the protocol to a subset that you really want to allow to your users. (e.g. you can forbid TCP port forwarding while still allowing terminal access).

The problem with publickey authentication is that the signature covers the so called SSH session_id which is a unique value derived during key exchange. My proxy implements a man-in-the-middle, so the client<->proxy and proxy<->server connections have a different session id, thus simply replaying the authentication packets of the client will not work since the SSH session ids do not match.

The solution is that we are going to replace user keys transparently when crossing the firewall, which means that private keys need to be stored there. This is both a feature and a drawback: a feature since you can control which keys you are allowing to leave your perimeter and a drawback as this requires additional management tasks. It would have been so much nicer if we could do this transparently, but I am afraid this is not possible unless we modify all clients out there or alternatively we manage to find a way to crack the Diffie-Hellmann key exchange algorithm.

On the syslog-ng side I have committed a fix to make files over 2GB work again. It should be available in the next snapshot shortly. I'm also thinking about preparing 1.9.10 with the fixes accumulated so far.

OpenSSL problem found

2006-03-31T22:41:00.000+02:00

I have finally tracked down the issue I was writing about in my previous blog post. It turned out to be a problem with OpenSSL on Linux 2.6 and NPTL. The default implementation of CRYPTO_thread_id() assumes that getpid() returns a unique value for each thread, however with NPTL each thread has the same pid and only their tid (thread id) value differ.

This made OpenSSL to hash its thread-specific error state to the same memory area, thus possibly overwriting memory concurrently freed in another thread. This caused a heap corruption which in turn caused crashes every now and then, of course showing a backtrace completely unrelated to the original problem.

The funny part is that I had a suspicion (see the yesterday's post) about this error state allocation I just have not seen the obvious reason: I had the impression that getpid() returns a different value for threads just like it did with LinuxThreads. Knowing the exact reason makes the whole issue trivial :).

I have posted this as an email on openssl-dev, I'm wondering what the reactions are going to be.
Take care.

Spending time in gdb...

2006-03-30T22:59:00.000+02:00

I have spent the last three days debugging an ugly crash in the upcoming Zorp 3.1. First I had some problems with the core files produced with Linux 2.6.12, as the register values proved to be invalid, thus the backtrace was even more unusable than it is usual with heap corruptions.

I could get access to the original register values as Zorp dumps part of its stack when a fatal signal is encountered. Using that information I could locate the stack frame of the signal handler and luckily Linux passes a "struct sigcontext" to each signal handler as parameter which contains register information. But nevertheless it made analyzing the core files difficult.

After a post to the gdb mailing list it turned out to be a kernel problem rather than a gdb problem and with the help of my collegue Krisztián Kovács (of Netfilter ct_sync fame) we could solve the problem by backporting a fix from 2.6.15, so core files are now ok.

The problem however seems to be difficult, I have already studied the libc malloc implementation, disassembled and annotated the _int_malloc and _int_free functions, I'm now able to read hexdumps of heap areas fluently but I still don't have a fix for the problem. Lucky us Zorp restarts itself in this situation and the scenario where this problem occurs is not frequently used.

My suspicion is that the SSL error state for threads are the cause of the problem as I have evidence that the freed heap block is overwritten by ERR_clear_state(), which destroys the next and prev pointers in the freed memory block, thus resulting in the crash. The error states are supposedly thread-specific variables, but the way the allocation is done is suspicious.

I hope I can finally find this problem tomorrow.

Preparing syslog-ng release

2006-03-28T21:53:00.000+02:00

I have started to prepare syslog-ng 1.6.10 for release, the tarball has already been uploaded to the website, but I still have not sent an announcement to the mailing lists. So if you read this here, you might download a still unannounced version :)

Nothing really important in the release, a cleanup in the documentation with several fixes and a migration to DocBook/XML from the SGML favour and a new tunable called time_sleep().

The latter was worked out together with John Morrissey who did some profiling and found that on hosts with a lot of syslog connections syslog-ng might become a bottleneck. The option does nothing but sleep() a defined amount of time which makes syslog-ng to process incoming messages in batches, this way decreasing the number of poll() loop iterations which was listed high (about 67%) in the profiles generated by John.

Setting time_sleep() to about 50ms decreased the CPU load by 80% which is quite significant I'd say.

As Rusty Russell would say I have just received a SIGWIFE, so going to bed now :)

Starting my blog

2006-03-28T19:33:00.000+02:00

I have considered starting my own blog for some time now and have finally started doing something for it. I first tried to set up blosxom but as I did not want to spend too much time customizing it I finally gave up and tried to find a nice blogger website which does everything for me. This is blogger.com, I like what I see so far.

Ops, I should have started by introducing myself: my name is Balázs Scheidler, I live in Budapest, Hungary and I started this blog because I would have some things to publish about some free software projects I am involved in and it is trendy to have a blog anyway :).

Back to my projects, I am the author of syslog-ng that you might know as an alternative system logging package for UNIX based systems. And also Zorp, an application layer gateway. You can find out more about these at http://www.balabit.com/. I also contribute patches to a couple of others (whenever I encounter something I don't like or which simply bugs me) and I sometimes poke into kernel development as well (generally netfilter related development like transparent proxying support for Linux).

So far, so good. Hopefully I won't give up too soon.

Bazsi