Bazsi's Home Automation blog

Last time I've written about how syslog-ng is able to change message contents. I thought it'd be useful to give you a more practical example, instead of a generic description. It is quite common to convert SNMP traps to syslog messages. The easiest implementation is to run snmptrapd and have it create a log message based on the trap. There's a small issue though: snmptrapd uses the UNIX syslog() API, and as such it is not able to propagate the originating host of the SNMP trap to the hostname portion of the syslog message. This means that all traps are logged as messages coming from the host running snmptrapd, and the hostname information is part of the message payload. Of course it'd be much easier to process syslog messages, if this were not the case. A solution would be to patch snmptrapd to send complete syslog frames, but that would require changing snmptrapd source. The alternative is to use the new parse and rewrite features of syslog-ng 3.0. First, you need to f...

For a long time I meant to give the "log statistics" feature of syslog-ng an overhaul, and finally with the advent of syslog-ng 3.0, this was done. I'm not sure all of you know, but even earlier syslog-ng versions (2.1 and 2.0) did collect some per-source and per-destination statistics. These were reported periodically in the system log. The problem with this approach that it didn't really scale: with a large configuration the statistics message could become kilobytes long, and parsing this information from a file possibly several gigabytes in size is daunting. syslog-ng 3.0 has two important changes in this area: it adds several new kinds of counters (like per-host counters), and a UNIX domain socket where you can query the current status of these counters. As counters certainly have an overhead, you can now control how much statistics you want to gather. The new stats_level() option has three levels for now: stats_level(0) is basically the same as earlier syslog-ng ...

Earlier this month, I announced the new syslog-ng 3.0 git tree, adding a lot of new features to syslog-ng Open Source Edition. I thought it'd be useful to describe the new features with some more details, so this time I'd write about message parsing. First of all, the message structure was a bit generalized in syslog-ng. Earlier it was encapsulating a syslog message and had little space to anything beyond that. That is, every log message that syslog-ng handled had date, host , program and message fields, but syslog-ng didn't care about message contents. This has changed, a LogMessage became a set of name-value pairs , with some "built-in" pairs that correspond to the parts of a syslog message. The aim with this change is: new name-value pairs can be associated with messages through the use of a parsing. It is now possible to parse non-syslog logs and use the columns the same way you could do it with syslog fields. Use them in the name of files, SQL tables or c...

I've spent my last week in Paris, where this year's Netfilter Workshop was held. I wanted to take this opportunity to thank Eric of INL for the organization. It was a wonderful and useful event, and I enjoyed it a lot. It is always nice to meet these wonderful guys. Here are some blog posts about the same event: INL: http://nfws.inl.fr/en/ DaveM: http://vger.kernel.org/~davem/cgi-bin/blog.cgi/2008/10/05#nfws2008 Patrick McHardy: http://people.netfilter.org/kaber/weblog/ Finally we could get Transparent Proxying merged, now queued for 2.6.28.

I could finally get my syslog-ng 3.0 OSE tree published at git.balabit.hu. No nightly snapshots yet and I still have to prepare a formal announcement to post on the mailing list, but for those I teased with functions from the 3.0 branch, here it comes. From the top of my head, OSE 3.0 supports: TLS encrypted channels, syslog message rewrite, parse parts of the syslog message and use the parsed parts in macros PCRE and glob filters (in addition to POSIX regexps), support for the new IETF syslog protocols, program sources, new statistics framework that can be queried using UNIX domain sockets etc. I just wanted to get the word out. Success/failure reports would be appreciated.

As of now the development of the generic rewrite feature has been completed in one of my private git repositories. The new code uses PCRE and I'm somewhat undecided how to move forward with PCRE. For those who might not know PCRE is an implementation of regular expressions and is an acronym for "Perl Compatible Regular Expressions". PCRE adds a lot more features and seems to perform better than its POSIX equivalent. So the situation is as follows: various filters use POSIX regexps rewrite uses PCRE This is not a very consistent combination, thus I'm planning to add PCRE support for filters too. The only question is whether it is needed to have two independent regexp styles in syslog-ng in the long run. If I decide that one of them is enough, then I'd deprecate POSIX style regexps in filters and wouldn't implement POSIX in rewrite rules. This combination would yield a syslog-ng that would give warnings when POSIX-style regular expressions are in use and in a fo...

I've spent the good part of last week in Stockholm at a gathering of Nagios users and developers. I was invited to give a talk on syslog-ng and security issues about collecting syslog data centrally as syslog-ng is often used hand-in-hand with Nagios. This was my first time in Sweden and I can say that Swedish people are the most hospitable nation I met so far, and Stockholm is a very nice city. Also op5 , the company organizing the event did their best to make us - the speakers - very welcome. Thanks op5. If you have a chance visit Stockholm I'd recommend to do so.

I've disappeared from this blog in the recent month but I've not been idle: I've implemented initial support for LogStore in the Premium Edition of syslog-ng. LogStore is a binary log file format, in semantics very similar to a plain log file. But the format allows much more: on-line compression via gzip, encryption via AES and X.509 certificates, integrity protection via hmac-sha1. And furthermore: it is indexed based on time, and it is quite efficient to look for a specific time range in GBs of log data. I'm quite satisfied, although there are some more work left to be done, for instance the query interface for the time based indexing is not completed. In use it is quite simple: replace the "file" destination with "logstore" and you are done. More or less the same amount of options are supported: macro based file names, template based formatting, etc. I'm still pondering with the idea of storing the complete internal representation of the logre...

I've published our set of dbi and dbi-drivers patches in a git repository to push changes upstream. The patches were updated against the latest libdbi versions. You can find these repositories at the BalaBit's git server, more precisely: git://git.balabit.hu/bazsi/libdbi.git git://git.balabit.hu/bazsi/libdbi-drivers.git The "master" branch contains the direct import of the libdbi CVS tree, our fixes are in the 'upstream-fixes' branch. This setup will make it easier for me to publish patches and regularly rebase the not-yet-merged set against the latest upstream. Among other small things, you can find a quite important patch against the Oracle driver. Without this patch Oracle 10.2 (the server!) segfaults and dumps core. So beware.

I installed Google analytics on this blog, and as it seems a number of people come here looking for "SFTP proxy", because of an old article I posted last July. Those interested primarily in my syslog-ng related articles may skip this post as this contains completely unrelated information, others please read on. :) For those who don't know: SFTP is a file-system sharing protocol running on top of SSH. It is not yet an IETF standard, however more and more enterprises replaces the aging FTP protocol for SFTP. The reasons are numerous: FTP uses plain text passwords, FTP uses multiple TCP connections for file transfer, FTP has inherent problems like bounce attacks, FTP does not encrypt traffic, FTP only supports filesystem metadata (last modification time, etc.) via extensions and others. All-in-all SFTP is newer, shinier and designed better. There's one problem though: SFTP uses SSH and SSH is encrypted. But wait, I said this is a drawback for FTP. Right, using encryptio...

We were asked to publish some more detailed "syslog-ng feature sheet". Albeit it might go into syslog-ng specific details we tried to be as generic as possible. And certainly everyone doing such feature sheets is biased, just as we were :) It is available at http://www.balabit.com/network-security/syslog-ng/features/detailed/ .

As promised earlier on the mailing list, I am designing the new message rewrite capabilities in syslog-ng. As you probably know, syslog-ng currently supports message templates for each destination, and this template can be used to rewrite the message payload. Each template may contain literal text and macro references. Macros can either expand to parts of the original message or parts that were matched using a regexp. Here's an example: destination d_file { file("/var/log/messages" template(" $HOST $MSG -- literal text $1\n")); }; The example above uses the format string specified at template to define the log file structure. The words starting with '$' are macros and expand to well defined parts of the original message. Numbered macros like $1 above are substituted to the last regular expression matches, all other characters are put into the result intact. While this functionality is indeed useful, it is somewhat limited: you cannot use sed-like search...

I have just uploaded the first release in the syslog-ng Open Source Edition 2.1 branch to our website. It is currently only available in source format at this location: http://www.balabit.com/downloads/files/syslog-ng/sources/2.1/src This release synchronizes the core of syslog-ng to the latest PE version and adds the SQL destination driver. This is an alpha release and thus might be rough around the edges, but it basically only contains code already tested in the context of the Premium Edition. The SQL functionality requires a patched libdbi package, which is available at the same link. We're going to work on integrating all our libdbi related patches to the upstream package. If you want to know how the SQL logging works, please see the Administrator's Guide or our latest white paper Collecting syslog messages into an SQL database with syslog-ng. The latter describes the Premium Edition, but it applies to the Open Source one equally well.

We had a meeting on the syslog-ng roadmap today where we decided some important things, and I thought I'd use this channel to tell you about it. The Open Source Edition will see a 2.1 release incorporating all core changes currently in the Premium Edition and additionally the SQL destination driver. We are going to start development on the 2.2 PE features, but some of those will also be incorporated in the open source version: support for the latest work of IETF syslog protocols unique sequence numbering for messages support for parsing message contents Previously syslog-ng followed the odd/even version numbering to denote development/stable releases. I'm going to abandon this numbering now: the next syslog-ng OSE release is going to have a 2.1 version number and will basically come out with tested code changes only. The current feature set in PE were developed in a closed manner and I don't want to repeat this mistake. The features that were decided to be part of the Open ...

I like christmas for a number of reasons: in addition to the traditional "meet and have fun with your family", eat lots of delicious food and so on, I like it because this is the season of the year when I have some time to do whatever I feel like. This year I felt like doing some syslog-ng performance analysis. After reading Ulrich Deppert's series about stuff "What every programmer should know about memory" on LWN, I thought I'm more than prepared to improve syslog-ng performance. Before going any further, I'd recommend this reading to any programmer, it's a bit long but every second reading it is worth it. As you need to measure performance in order to improve it, I wrote a tool called "loggen". This program generates messages messages at a user-specifyable rate. Apart from the git repository you can get this tool from the latest syslog-ng snapshots. Loggen supports TCP, UDP and UNIX domain sockets, so really almost everything can be me...

We've been busy recently on the syslog-ng front. A new release came out from both the Open Source and Premium Editions, covering various bugfixes, the most important being a fix for an easy denial of service. Please upgrade to at least OSE 2.0.6 and PE 2.1.8, my bugtraq posting has more details. A new release from the syslog-ng documentation was also published, it contains a new chapter on losing messages and another one which explains the "Log statistics" message generated by syslog-ng. And since, I'm writing this entry I wanted to say "Merry Christmas" to all the readers of this blog: Marry Christmas and a Happy New Year. I'm spending the winter holiday with my wife and I'm going to travel a lot around the country to visit relatives. So I'm not sure I can be very productive in the coming days.

I've just seen a post on the loganalysis mailing list how easy it is to implement disk-based buffering with perl and a few hours time. The implementation would be as simple as sending the messages to a file and using a script like "tail -f" to follow the file and send messages to the desired log collectors. Although the scheme that was described would work, I see three important problems: latency: because the solution would work by polling the log file, the latency is severely increased, when you have thousands of log entries per second, a second is a long time. And you don't want to poll more often than every second. disk usage: relaying the data would store everything on the local disk, no upper bound on disk usage, if the disk is full, data is lost load: using an interpreted language and the requirement to store all data on disk puts an enormous load on the system that might be spent better elsewhere. Syslog-ng on the other hand uses a spool file, but this file ...

I was learning computer programming while at secondary school, at as a school assignment we had to create an accounting system on a venerable IBM 360. We had 386dx computers back then with 2MB RAM, the first Linux encounter at this time required me and my friend to put all of our RAM modules to a single computer to get 4 megs and make it suitable for Linux. This 360 machine had two CPUs, each the size of a large office desk, speed comparable to an 386, had 8MB of RAM. After getting used to the speed it provided (had 8 terminals on a 2400 baud modem) it was fun, completely different to the PCs we've had, but still, interesting. Some of this memories were brought back by our partnership with Patownsend & Associates , a US based company to deliver System i (formerly AS400) and System z (the mainframe) software. Our partnership focuses on the development of a Syslog agent for System i, this way we could extend our syslog-ng Premium Edition offering with support for IBM midrange s...

I've started creating a public bugzilla installation for syslog-ng in the last couple of days, however I was interrupted before being to finish it all. I want to move all opened internal tickets to the public installation, but I only managed to review half of them. Not that it is a large task, I just got distracted all the time by my fellow collegues and customers. But that's the life of a software developer, right? :) You can find this at http://bugzilla.balabit.com/ , although it is not yet officially announced. Once I'm done with reviewing/translating tickets, I'll send an official announcement to the mailing list.

As I announced on the syslog-ng mailing list, the official syslog-ng git repositry has moved and has become accessible using the more effective "git" protocol, instead of plain HTTP. The change has been described on the syslog-ng webpage , gitweb is also available.