Bazsi's Home Automation blog

I've implemented basic include file/directory functionality in syslog-ng, using the format numbered second in my previous post . I've now pushed an expermental implementation of include files in the syslog-ng OSE 3.0 repository, in a separate branch called 'include'. E.g. in order to test the code, please clone the syslog-ng 3.0 repository: $ git clone git://git.balabit.hu/bazsi/syslog-ng-3.0.git Then check out the 'include' branch: $ git checkout --track -b include origin/include Then compile as usual. I didn't want to integrate it right into syslog-ng OSE 3.0 tree as I'd like to release that first as 3.0.1.

I'm about to implement configuration file includes, and although the implementation is quite straightforward, the syntax to be used is something to give a thought or two. Currently the syslog-ng configuration file consists of statements, each with the following basic format: stmt [ ] { ... }; The "id" gives a unique identifier of the statement, and the braces enclose the contents. Currently only the ID part is optional, the braces are always there. To make the include statement consistent with that, it'd have to look something like: include { "filename" }; Obviously I don't like this too much, as it is way different from all other applications permitting the use of include statements. What about this: include "filename"; E.g. use the ID part the name of the file to be included. I like this better. A third option might be the use of 'pragma' directives, currently only used to specify the file format compatibility in the case of syslog-ng...

Last time I've written about how syslog-ng is able to change message contents. I thought it'd be useful to give you a more practical example, instead of a generic description. It is quite common to convert SNMP traps to syslog messages. The easiest implementation is to run snmptrapd and have it create a log message based on the trap. There's a small issue though: snmptrapd uses the UNIX syslog() API, and as such it is not able to propagate the originating host of the SNMP trap to the hostname portion of the syslog message. This means that all traps are logged as messages coming from the host running snmptrapd, and the hostname information is part of the message payload. Of course it'd be much easier to process syslog messages, if this were not the case. A solution would be to patch snmptrapd to send complete syslog frames, but that would require changing snmptrapd source. The alternative is to use the new parse and rewrite features of syslog-ng 3.0. First, you need to f...

For a long time I meant to give the "log statistics" feature of syslog-ng an overhaul, and finally with the advent of syslog-ng 3.0, this was done. I'm not sure all of you know, but even earlier syslog-ng versions (2.1 and 2.0) did collect some per-source and per-destination statistics. These were reported periodically in the system log. The problem with this approach that it didn't really scale: with a large configuration the statistics message could become kilobytes long, and parsing this information from a file possibly several gigabytes in size is daunting. syslog-ng 3.0 has two important changes in this area: it adds several new kinds of counters (like per-host counters), and a UNIX domain socket where you can query the current status of these counters. As counters certainly have an overhead, you can now control how much statistics you want to gather. The new stats_level() option has three levels for now: stats_level(0) is basically the same as earlier syslog-ng ...

Earlier this month, I announced the new syslog-ng 3.0 git tree, adding a lot of new features to syslog-ng Open Source Edition. I thought it'd be useful to describe the new features with some more details, so this time I'd write about message parsing. First of all, the message structure was a bit generalized in syslog-ng. Earlier it was encapsulating a syslog message and had little space to anything beyond that. That is, every log message that syslog-ng handled had date, host , program and message fields, but syslog-ng didn't care about message contents. This has changed, a LogMessage became a set of name-value pairs , with some "built-in" pairs that correspond to the parts of a syslog message. The aim with this change is: new name-value pairs can be associated with messages through the use of a parsing. It is now possible to parse non-syslog logs and use the columns the same way you could do it with syslog fields. Use them in the name of files, SQL tables or c...

I've spent my last week in Paris, where this year's Netfilter Workshop was held. I wanted to take this opportunity to thank Eric of INL for the organization. It was a wonderful and useful event, and I enjoyed it a lot. It is always nice to meet these wonderful guys. Here are some blog posts about the same event: INL: http://nfws.inl.fr/en/ DaveM: http://vger.kernel.org/~davem/cgi-bin/blog.cgi/2008/10/05#nfws2008 Patrick McHardy: http://people.netfilter.org/kaber/weblog/ Finally we could get Transparent Proxying merged, now queued for 2.6.28.

I could finally get my syslog-ng 3.0 OSE tree published at git.balabit.hu. No nightly snapshots yet and I still have to prepare a formal announcement to post on the mailing list, but for those I teased with functions from the 3.0 branch, here it comes. From the top of my head, OSE 3.0 supports: TLS encrypted channels, syslog message rewrite, parse parts of the syslog message and use the parsed parts in macros PCRE and glob filters (in addition to POSIX regexps), support for the new IETF syslog protocols, program sources, new statistics framework that can be queried using UNIX domain sockets etc. I just wanted to get the word out. Success/failure reports would be appreciated.

As of now the development of the generic rewrite feature has been completed in one of my private git repositories. The new code uses PCRE and I'm somewhat undecided how to move forward with PCRE. For those who might not know PCRE is an implementation of regular expressions and is an acronym for "Perl Compatible Regular Expressions". PCRE adds a lot more features and seems to perform better than its POSIX equivalent. So the situation is as follows: various filters use POSIX regexps rewrite uses PCRE This is not a very consistent combination, thus I'm planning to add PCRE support for filters too. The only question is whether it is needed to have two independent regexp styles in syslog-ng in the long run. If I decide that one of them is enough, then I'd deprecate POSIX style regexps in filters and wouldn't implement POSIX in rewrite rules. This combination would yield a syslog-ng that would give warnings when POSIX-style regular expressions are in use and in a fo...

I've spent the good part of last week in Stockholm at a gathering of Nagios users and developers. I was invited to give a talk on syslog-ng and security issues about collecting syslog data centrally as syslog-ng is often used hand-in-hand with Nagios. This was my first time in Sweden and I can say that Swedish people are the most hospitable nation I met so far, and Stockholm is a very nice city. Also op5 , the company organizing the event did their best to make us - the speakers - very welcome. Thanks op5. If you have a chance visit Stockholm I'd recommend to do so.

I've disappeared from this blog in the recent month but I've not been idle: I've implemented initial support for LogStore in the Premium Edition of syslog-ng. LogStore is a binary log file format, in semantics very similar to a plain log file. But the format allows much more: on-line compression via gzip, encryption via AES and X.509 certificates, integrity protection via hmac-sha1. And furthermore: it is indexed based on time, and it is quite efficient to look for a specific time range in GBs of log data. I'm quite satisfied, although there are some more work left to be done, for instance the query interface for the time based indexing is not completed. In use it is quite simple: replace the "file" destination with "logstore" and you are done. More or less the same amount of options are supported: macro based file names, template based formatting, etc. I'm still pondering with the idea of storing the complete internal representation of the logre...

I've published our set of dbi and dbi-drivers patches in a git repository to push changes upstream. The patches were updated against the latest libdbi versions. You can find these repositories at the BalaBit's git server, more precisely: git://git.balabit.hu/bazsi/libdbi.git git://git.balabit.hu/bazsi/libdbi-drivers.git The "master" branch contains the direct import of the libdbi CVS tree, our fixes are in the 'upstream-fixes' branch. This setup will make it easier for me to publish patches and regularly rebase the not-yet-merged set against the latest upstream. Among other small things, you can find a quite important patch against the Oracle driver. Without this patch Oracle 10.2 (the server!) segfaults and dumps core. So beware.

I installed Google analytics on this blog, and as it seems a number of people come here looking for "SFTP proxy", because of an old article I posted last July. Those interested primarily in my syslog-ng related articles may skip this post as this contains completely unrelated information, others please read on. :) For those who don't know: SFTP is a file-system sharing protocol running on top of SSH. It is not yet an IETF standard, however more and more enterprises replaces the aging FTP protocol for SFTP. The reasons are numerous: FTP uses plain text passwords, FTP uses multiple TCP connections for file transfer, FTP has inherent problems like bounce attacks, FTP does not encrypt traffic, FTP only supports filesystem metadata (last modification time, etc.) via extensions and others. All-in-all SFTP is newer, shinier and designed better. There's one problem though: SFTP uses SSH and SSH is encrypted. But wait, I said this is a drawback for FTP. Right, using encryptio...

We were asked to publish some more detailed "syslog-ng feature sheet". Albeit it might go into syslog-ng specific details we tried to be as generic as possible. And certainly everyone doing such feature sheets is biased, just as we were :) It is available at http://www.balabit.com/network-security/syslog-ng/features/detailed/ .

As promised earlier on the mailing list, I am designing the new message rewrite capabilities in syslog-ng. As you probably know, syslog-ng currently supports message templates for each destination, and this template can be used to rewrite the message payload. Each template may contain literal text and macro references. Macros can either expand to parts of the original message or parts that were matched using a regexp. Here's an example: destination d_file { file("/var/log/messages" template(" $HOST $MSG -- literal text $1\n")); }; The example above uses the format string specified at template to define the log file structure. The words starting with '$' are macros and expand to well defined parts of the original message. Numbered macros like $1 above are substituted to the last regular expression matches, all other characters are put into the result intact. While this functionality is indeed useful, it is somewhat limited: you cannot use sed-like search...

I have just uploaded the first release in the syslog-ng Open Source Edition 2.1 branch to our website. It is currently only available in source format at this location: http://www.balabit.com/downloads/files/syslog-ng/sources/2.1/src This release synchronizes the core of syslog-ng to the latest PE version and adds the SQL destination driver. This is an alpha release and thus might be rough around the edges, but it basically only contains code already tested in the context of the Premium Edition. The SQL functionality requires a patched libdbi package, which is available at the same link. We're going to work on integrating all our libdbi related patches to the upstream package. If you want to know how the SQL logging works, please see the Administrator's Guide or our latest white paper Collecting syslog messages into an SQL database with syslog-ng. The latter describes the Premium Edition, but it applies to the Open Source one equally well.

We had a meeting on the syslog-ng roadmap today where we decided some important things, and I thought I'd use this channel to tell you about it. The Open Source Edition will see a 2.1 release incorporating all core changes currently in the Premium Edition and additionally the SQL destination driver. We are going to start development on the 2.2 PE features, but some of those will also be incorporated in the open source version: support for the latest work of IETF syslog protocols unique sequence numbering for messages support for parsing message contents Previously syslog-ng followed the odd/even version numbering to denote development/stable releases. I'm going to abandon this numbering now: the next syslog-ng OSE release is going to have a 2.1 version number and will basically come out with tested code changes only. The current feature set in PE were developed in a closed manner and I don't want to repeat this mistake. The features that were decided to be part of the Open ...