Skip to main content

syslog-ng 3.2alpha2 released

I've just uploaded syslog-ng 3.2alpha2 to the release directory. The last alpha release didn't compile on all supported platforms and the automatic test-suite was disabled, because it only worked if syslog-ng got installed first.

These obstacles have been overcome and together with some fixes and a couple of new features, 3.2alpha2 is now available. I've also forward ported all bugfixes from syslog-ng 3.1.2.

For those who are starting to experiment with the 3.2 branch, here's the list of new features compared to 3.1. Those who tried 3.2alpha1, the list of changes compared to 3.2alpha1 is at the end of this post.

Since the documentation of syslog-ng is not yet up-to-date with the new features introduced, I've tried to also include URLs for the best known descriptions. The references may not be 100% accurate, but should give anyone interested an idea how to start experimenting.

Also, please note that although this is an alpha release, the bulk of the changes are in the configuration parser, so once your configuration was parsed properly and syslog-ng starts up, an almost unchanged code is processing it. This means that this release should be good enough to start playing with. And feedback about what kind of syslog-ng.conf parsing errors you encounter on real-life configuration files is more than welcome.

Code quality & functionality wise, this could be a beta release, I only expect "procedural" changes, like cleaning up the plugin names, which wouldn't be nice to do in a beta release (though not unheard of :)

New features in 3.2:
  • Plugins: the new architecture replaces the old monolithic one, all syslog-ng functionality is loaded from external plugins when needed. It is possible to write plugins to extend syslog-ng functionality in the following areas: sources, destinations, filter expression, parsers, rewrite ops, message format.
  • The framework for a "syslog-ng configuration library" (aka SCL) a collection of configuration snippets installed along syslog-ng, simplifying the authoring of syslog-ng configuration files.
  • pdbtool match is now able to read a file containing syslog messages and apply patterndb and a filter expression on the contents.
  • pdbtool test is now able to perform pattern testing automatically based on the supplied example log message.
  • Persistent state containing the current file position for file sources is now continously updated during runtime, instead of updating it only at exit, which makes it much more reliable in case syslog-ng doesn't terminate normally.
  • Better syntax error reporting in the configuration file.
  • Support for reusable configuration snippets, similar to macros with parameters, named "blocks".
  • Added a confgen plugin that includes the output of a program into the configuration file, making it possible to generate configuration file snippets dynamically.
  • Support for BSD-style process accounting logs via the pacct() source driver defined in by SCL and the underlying pacctformat plugin.
  • Support for explicit COMMITs in the SQL driver, this speeds up SQL INSERT rate significantly if flush_lines() is non-zero.
  • It is now possible to supply a filter to rewrite expressions and only apply the rewrite rule in case the filter matches.
  • It is now possible to use multiple parser expressions in a single parser object, similar to rewrite rules.
  • Added support for using the include statement from anywhere in the configuration file, instead of only at top-level. Also introduced syslog-ng "global values" that can be defined and the substituted anywhere in the configuration file.

  • Default configuration file supplied as part of SCL.

Incompatible changes:
  • syslog-ng traditionally expected an optional hostname field even when a syslog message is received on a local transport (e.g. /dev/log). However no UNIX version is known to include this field. This caused problems when the application creating the log message has a space in its program name field. This behaviour has been changed for the unix-stream/unix-dgram/pipe drivers if the config version is 3.2 and can be restored by using an explicit 'expect-hostname' flag for the specific source.

Changes since 3.2alpha1:
  • Now compiles on all platforms and the unit/functional tests also run. (tested: AIX, HP-UX, Solaris, FreeBSD, Linux, Tru64)
  • Fixed pdbtool match --debug-pattern output for ESTRING parsers.
  • Fixed a possible memory leak in the lexer, which would accumulate in case SIGHUPs.
  • Fixed Solaris STREAMS device support.
  • Forward ported all bugfixes from syslog-ng OSE 3.0 & 3.1
  • Disable process accounting module by default as it doesn't compile on non-Linux platforms.
  • Added "pdbtool match --file" option to read and parse an existing logfile.
  • Added "pdbtool test" to check the log samples in the patterndb file.
  • Added "dont-create-tables" flag for the SQL destination to inhibit automatic table creation.
  • Added "condition()" support for rewrite expressions, which makes it possible to skip rewrite rules that do not match a filter expression.
  • Added "--module-path" command line option to control where modules are loaded from from the command line.

Happy logging!

Labels

Labels:

Comments

Post a Comment

Popular posts from this blog

syslog-ng fun with performance

I like christmas for a number of reasons: in addition to the traditional "meet and have fun with your family", eat lots of delicious food and so on, I like it because this is the season of the year when I have some time to do whatever I feel like. This year I felt like doing some syslog-ng performance analysis. After reading Ulrich Deppert's series about stuff "What every programmer should know about memory" on LWN, I thought I'm more than prepared to improve syslog-ng performance. Before going any further, I'd recommend this reading to any programmer, it's a bit long but every second reading it is worth it. As you need to measure performance in order to improve it, I wrote a tool called "loggen". This program generates messages messages at a user-specifyable rate. Apart from the git repository you can get this tool from the latest syslog-ng snapshots. Loggen supports TCP, UDP and UNIX domain sockets, so really almost everything can be me...
17 comments
Read more

syslog-ng contributions redefined

syslog-ng has been around for about 12 years now, but I think the biggest change in the project's life is imminent: with the upcoming release of syslog-ng OSE 3.2, syslog-ng will become an independent entity. Until now, syslog-ng was primarily maintained & developed by BalaBit, copyrights needed to be reassigned in order to grant BalaBit special privileges. BalaBit used her privileges to create a dual-licensed fork of syslog-ng, named "syslog-ng Premium Edition". The value we offer over the Open Source Edition of syslog-ng are things that larger enterprises require: support on a large number of UNIX platforms (27 as of 3.1), smaller and larger feature differences (like the encrypted/digitally signed logfile feature) better test coverage and release management longer term support Although perfectly legal, this business model was not welcome in various Free Software communities, and has caused friction and harm, because BalaBit has enjoyed a privilege that no others cou...
Post a Comment
Read more

syslog-ng message parsing

Earlier this month, I announced the new syslog-ng 3.0 git tree, adding a lot of new features to syslog-ng Open Source Edition. I thought it'd be useful to describe the new features with some more details, so this time I'd write about message parsing. First of all, the message structure was a bit generalized in syslog-ng. Earlier it was encapsulating a syslog message and had little space to anything beyond that. That is, every log message that syslog-ng handled had date, host , program and message fields, but syslog-ng didn't care about message contents. This has changed, a LogMessage became a set of name-value pairs , with some "built-in" pairs that correspond to the parts of a syslog message. The aim with this change is: new name-value pairs can be associated with messages through the use of a parsing. It is now possible to parse non-syslog logs and use the columns the same way you could do it with syslog fields. Use them in the name of files, SQL tables or c...
1 comment
Read more