Skip to main content

syslog-ng contributions redefined

syslog-ng has been around for about 12 years now, but I think the biggest change in the project's life is imminent: with the upcoming release of syslog-ng OSE 3.2, syslog-ng will become an independent entity.

Until now, syslog-ng was primarily maintained & developed by BalaBit, copyrights needed to be reassigned in order to grant BalaBit special privileges. BalaBit used her privileges to create a dual-licensed fork of syslog-ng, named "syslog-ng Premium Edition". The value we offer over the Open Source Edition of syslog-ng are things that larger enterprises require:
  • support on a large number of UNIX platforms (27 as of 3.1),
  • smaller and larger feature differences (like the encrypted/digitally signed logfile feature)
  • better test coverage and release management
  • longer term support
Although perfectly legal, this business model was not welcome in various Free Software communities, and has caused friction and harm, because BalaBit has enjoyed a privilege that no others could get. We plan to fix this situation and we've worked hard in the past months to make this possible.

We're letting syslog-ng go: no signed Contributory License Agreements will be required in order to contribute to syslog-ng in the future.

This is not just a matter of policy: while BalaBit wants to be a true citizen of the Free Software world, we also need to ensure the continued revenue stream that the Premium Edition provides. The adjusted business model allows us to deliver what our customers need, and at the same time make it possible for anyone else in the community to have the same privileges as BalaBit has.

The syslog-ng changes in 3.2 that I feel are most important are detailed below.

Plugins

syslog-ng was transformed from being a monolithic executable, to a core and a set of plugins.

Plugins range from source and destination drivers, filters, parsers, rewrite objects, anything can be extended with the use of a relatively simple plugin.

And this change didn't transform the configuration file format, it remained the same as before: readable and flexible. If you don't want to look under the hood, no functionality has changed, except plugins are loaded at runtime, and as a side effect, the syntax error reports are way better.

The core is licensed under the LGPL and plugins are licensed under the GPL. This legal framework allows BalaBit to deliver value to its customers in non-free plugins and syslog-ng related services.

syslog-ng Configuration Library

syslog-ng is an infrastructure element: very flexible, but sometimes intimidating. For example, having to specify the pad_size() option for an HP-UX /dev/log device explicitly makes syslog-ng flexible, but not very user-friendly.

It is common to share working configuration snippets on the mailing list or elsewhere in the syslog-ng community. We try to concentrate this effort with the creation of the syslog-ng Configuration Library (aka SCL). This library is a set of config snippets that can be included into a syslog-ng configuration file, using proper defaults but still allowing customization.

For example: we've created a source driver named "system()" which automatically expands to the local log devices as needed by the current Operating System syslog-ng runs on, but it is also possible to create application specific configuration snippets (for example apache source?) and ship it as a ready to use config block. See this post for more information.

This makes it possible to create a configuration file that will run on each of your platforms. In fact, we did that too: syslog-ng now comes with a default configuration file.

Support for non-syslog messages

Since syslog-ng is now plugin based and even the "syslog" message format as such is a plugin, it is now quite easy to add support for non-syslog message sources. As an example, we've added a plugin to parse Linux process accounting records, which makes it trivial to collect this data as well and possibly use it as a source of information when correllating data.

Future plugins like creating a MIB aware SNMP listener, or possibly processing netflow data, but I'd like to create a generic SQL source as well.

Support for these formats doesn't mean that syslog-ng would transform them into textual messages: proper name-value structure is kept and it is possible to put them into nicely structured SQL tables. Of course if all you want to transform them to text for readability, that's possible too with the use of a template().

patterndb project

Like I announced before, we're starting a parallel project to create a set of message patterns directly usable with syslog-ng's db-parser().

Current state

The changes are happening in the syslog-ng 3.2 repository at:

http://git.balabit.hu/

We have created a source-only 3.2alpha1 release of the current state. It runs our automated test harness on Linux, but of course it is not yet recommended in production environments. The release can be downloaded from www.balabit.com.

There were significant changes in how syslog-ng is compiled, thus it is expected that we have build issues on non-Linux systems. We expect to address these issues as they are found and create a 3.2beta1 release, for a longer beta testing.

Future

Parallel to fixing up the remaining issues on syslog-ng 3.2, we're going to open new ways to improve syslog-ng in the 3.3 branch. The most important item on our product backlog is to address the scalability problems and improve performance of syslog-ng.

Summary

With the upcoming changes in syslog-ng OSE 3.2 possible contributions will be greatly expanded:
  • write a pattern for your favourite application and process log data faster + easier
  • write an SCL configuration snippet, make your application easier to integrate with syslog-ng
  • write a plugin for your favourite NoSQL database,
  • write a plugin for a transformation that syslog-ng is not capable of doing right now (what about facility / severity rewrite rules? easy as a piece of cake)
  • write a plugin for things that you do with an external script
  • or contribute to the core of syslog-ng.
And all this without having to sign paperwork.

Comments

Popular posts from this blog

syslog-ng fun with performance

I like christmas for a number of reasons: in addition to the traditional "meet and have fun with your family", eat lots of delicious food and so on, I like it because this is the season of the year when I have some time to do whatever I feel like. This year I felt like doing some syslog-ng performance analysis. After reading Ulrich Deppert's series about stuff "What every programmer should know about memory" on LWN, I thought I'm more than prepared to improve syslog-ng performance. Before going any further, I'd recommend this reading to any programmer, it's a bit long but every second reading it is worth it. As you need to measure performance in order to improve it, I wrote a tool called "loggen". This program generates messages messages at a user-specifyable rate. Apart from the git repository you can get this tool from the latest syslog-ng snapshots. Loggen supports TCP, UDP and UNIX domain sockets, so really almost everything can be me

syslog-ng roadmap 2.1 & 2.2

We had a meeting on the syslog-ng roadmap today where we decided some important things, and I thought I'd use this channel to tell you about it. The Open Source Edition will see a 2.1 release incorporating all core changes currently in the Premium Edition and additionally the SQL destination driver. We are going to start development on the 2.2 PE features, but some of those will also be incorporated in the open source version: support for the latest work of IETF syslog protocols unique sequence numbering for messages support for parsing message contents Previously syslog-ng followed the odd/even version numbering to denote development/stable releases. I'm going to abandon this numbering now: the next syslog-ng OSE release is going to have a 2.1 version number and will basically come out with tested code changes only. The current feature set in PE were developed in a closed manner and I don't want to repeat this mistake. The features that were decided to be part of the Open

syslog-ng 3.0 and SNMP traps

Last time I've written about how syslog-ng is able to change message contents. I thought it'd be useful to give you a more practical example, instead of a generic description. It is quite common to convert SNMP traps to syslog messages. The easiest implementation is to run snmptrapd and have it create a log message based on the trap. There's a small issue though: snmptrapd uses the UNIX syslog() API, and as such it is not able to propagate the originating host of the SNMP trap to the hostname portion of the syslog message. This means that all traps are logged as messages coming from the host running snmptrapd, and the hostname information is part of the message payload. Of course it'd be much easier to process syslog messages, if this were not the case. A solution would be to patch snmptrapd to send complete syslog frames, but that would require changing snmptrapd source. The alternative is to use the new parse and rewrite features of syslog-ng 3.0. First, you need to f