Thursday, April 15, 2010

syslog-ng 3.2 changes

I've just pushed a round of updates to the syslog-ng 3.2 repository, featuring some interesting stuff, such as:
  • SQL reorganization: Patrick Hemmer sent in a patch to implement explicit transaction support instead of the previous auto-commit mode used by syslog-ng. I threw in some fixes and refactored the code somewhat.
  • Configuration parser changes: the syntax errors produced by syslog-ng became much more user-friendly: not only the column is displayed, but also the erroneous line is printed and the error location is also highlighted.
  • Additional plugin modules were created: afsql for the SQL destination, and afstreams for Solaris STREAMS devices. Creating a new plugin from core code takes about 15 minutes. I'm quite satisfied. With the addition of these two modules, it is now possible to use syslog-ng without any kind of runtime dependency except libc.
  • The already existing afsocket module (providing tcp/udp sources & destinations) is compiled twice: once with and once without SSL support, so it is now possible to choose which one to use at runtime.
And since a blog post is not complete without a "screenshot", here's how the new error message looks like:

Error parsing plugin unix-stream, syntax error in etc/syslog-ng-null.conf at line 3, column 34:

source s_log { unix-stream("log" default-facilitz(syslog)); };

Neat, eh?

One more thing I need to think about is how to configure module loading. I guess it'd be less than user friendly if I'd rely on the user to load the modules that so far were the core functionality of syslog-ng.

E.g. right now if you want udp() support, you'd need something like this in the syslog-ng.conf header:

@module: afsocket

Having to remember all the module names is cumbersome and irritating. On the other hand I'd like to make it possible to run a bare-bones syslog-ng without socket support, so autoloading modules without any configurability is also out of the question.

In the current implementation syslog-ng automatically loads core modules, if the configuration version is below 3.2, and does nothing if it is 3.2 (e.g. you need an explicit @module directive for the current configuration format).

If you have an idea about how you think configuring modules should look like, just drop me an email/comment.

Monday, April 12, 2010

Explicit transaction support in SQL

The SQL destination in syslog-ng so far assumed that databases automatically start a new transaction for each INSERT statement that syslog-ng issues. This works fine, however there is a significant overhead of starting new transactions, with sqlite I've measured about 20 times performance increase on my development notebook and my debug build.

With explicit-commits:
bazsi@bzorp:~/.zwa/install/syslog-ng-ose-3.2$ loggen -x -r 1000000 -I 10 -S log
average rate = 9377.28 msg/sec, count=93776, time=10.003, msg size=256, bandwidth=2344.32 kB/sec

With per-statement (automatic) commits:
bazsi@bzorp:~/.zwa/install/syslog-ng-ose-3.2$ loggen -x -r 1000000 -I 10 -S log
average rate = 529.46 msg/sec, count=5299, time=10.083, msg size=256, bandwidth=132.36 kB/sec

So this really seem to matter.

In order to configure it you can use the following options in an SQL destination:
  • flush_lines/flush_timeout controls how much messages get into the same transaction, similar to what these parameters mean for standard log files
  • flags("explicit-commits") enables explicit transaction handling
Also an option named "session_statements" was added where you can list initial SQL commands, issued right after the connection is established.

This work has been started by Patrick Hemmer (thanks again Patrick). I had to do some work on it though, since in order to avoid races the timer code of the main loop couldn't be used.

You can find all this in the syslog-ng OSE 3.2 branch. I'd love to hear success/failure stories and performance numbers you can measure with your favourite database.

Monday, April 05, 2010

syslog-ng 3.2 opened, experimental "blocks" branch opened

After last the stable syslog-ng 3.1.0 release last week, I've opened the 3.2 branch to receive the new stuff. The first bits are already in the repository: the basic plugin framework and the conversion of the socket related stuff (tcp, udp, unix-dgram, unix-stream, syslog drivers) into a separate plugin.

The reason of the afsocket plugin conversion is to help moving the OpenSSL dependency to a separate package in distributions where this dependency cannot be associated with core packages like syslog-ng.

But I see a lot of potential behind the plugin framework, and I still have a lot of ideas, just check my last blog post in the topic.

Also, there's an even more experimental feature in the "blocks" branch right now, it is still incomplete, the naming and the syntax is still vague. The aim is there to provide syslog-ng.conf C++ template-like functionality in order to make the configuration easier by using pre-configured config snippets.

For example:

block source tomcat<root_dir="/opt/tomcat"> {
   file("<<root_dir>>/var/tomcat.log" follow_freq(1));

source s_local {

The idea is that if "tomcat" is referenced in a source statement, it'll be expanded to the content defined before that, all with proper argument passing and default values. This example actually works fine with the "blocks" branch right now.

Combining this feature with include files I foresee something like a "syslog-ng configuration library", which would contain pre-configured syslog-ng sources containing settings needed in order for a specific application to log properly, this way avoiding the hassle of integrating each and every application to syslog separately.

Another (not-yet-working) example would be to get rid off platform specific /dev/log declarations:

block source system<> {
@if (OS == "solaris")
@  if (OS_VERSION >= "10")
    sun-streams("/dev/log" door("/var/run/syslog_door"));
@  else
    sun-streams("/dev/log" door("/etc/.syslog_door"));
@  endif
@elif (OS == "linux")
    file("/proc/kmsg" program_override("kernel"));
@elif (OS == ...)

source s_local { system(); };

This would make the initial configuration of syslog-ng much easier, and would allow you to use the same configuration file everywhere.

As I said this is still very experimental, I'm not yet sure about:
  • the name (I would have called these templates, but that is already taken in syslog-ng, just like "macro"). so if you have an idea of a better name I'd love hear about it
  • argument passing, whether to allow positional arguments
  • argument expansion syntax, "<<arg>>", and "$$arg" both occurred to me, the latter seems to be used more universally. I'd want to include both in strings (like in the example above) and in non-string arguments, e.g. follow-freq($$arg)
If you have comments, ideas for names, please post them as comments, or send an email to the syslog-ng mailing list.

Also, while I was there changing the parser/lexer framework for pluginization, I added more details to the error message to make it easier to locate the syntax error in the configuration file.

As for the next things, I'm going to start integrating the external contributions I've received so far, so stay tuned.