Skip to main content

syslog-ng 3.2 changes

I've just pushed a round of updates to the syslog-ng 3.2 repository, featuring some interesting stuff, such as:
  • SQL reorganization: Patrick Hemmer sent in a patch to implement explicit transaction support instead of the previous auto-commit mode used by syslog-ng. I threw in some fixes and refactored the code somewhat.
  • Configuration parser changes: the syntax errors produced by syslog-ng became much more user-friendly: not only the column is displayed, but also the erroneous line is printed and the error location is also highlighted.
  • Additional plugin modules were created: afsql for the SQL destination, and afstreams for Solaris STREAMS devices. Creating a new plugin from core code takes about 15 minutes. I'm quite satisfied. With the addition of these two modules, it is now possible to use syslog-ng without any kind of runtime dependency except libc.
  • The already existing afsocket module (providing tcp/udp sources & destinations) is compiled twice: once with and once without SSL support, so it is now possible to choose which one to use at runtime.
And since a blog post is not complete without a "screenshot", here's how the new error message looks like:

Error parsing plugin unix-stream, syntax error in etc/syslog-ng-null.conf at line 3, column 34:

source s_log { unix-stream("log" default-facilitz(syslog)); };
                                 ^^^^^^^^^^^^^^^^

Neat, eh?

One more thing I need to think about is how to configure module loading. I guess it'd be less than user friendly if I'd rely on the user to load the modules that so far were the core functionality of syslog-ng.

E.g. right now if you want udp() support, you'd need something like this in the syslog-ng.conf header:

@module: afsocket

Having to remember all the module names is cumbersome and irritating. On the other hand I'd like to make it possible to run a bare-bones syslog-ng without socket support, so autoloading modules without any configurability is also out of the question.

In the current implementation syslog-ng automatically loads core modules, if the configuration version is below 3.2, and does nothing if it is 3.2 (e.g. you need an explicit @module directive for the current configuration format).

If you have an idea about how you think configuring modules should look like, just drop me an email/comment.

Comments

Anonymous said…
It IMO makes sense use the explicit
@module: afsocket
way.

One possible way is to introduce versions for the config settings
allowing some tricks with default values.
Another possibility would be to make it possible that e.g. udp()
triggers module loading, but this makes the parser step complex
I think.

There are global options hat have to be at the begin of the config
anyway. It would be also possible to add an "autoload on/off" flag
causing to load all available modules or not (and a "compatibility"
flag specifying the version to be compatible with).

syslog-ng-3.x could change to use syslog-ng.conf with a default of
autoload=off in version 3.x and when the (compatibility version) is
not set, it assumes to run in compatibility mode to the oldest
supported version where autoload=on is used by default.

Les complex, would be to just assume:
autoload not specified => autoload=on + warning [compat mode]

Default "dist" config makes use of autoload=off.


More important here is IMO, that there are (optionally) two paths,
where the modules can be installed:

/lib[64]/syslog-ng and /usr/lib[64]/syslog-ng

The second is required to make it possible to run syslog-ng on
systems with iscsi/nfs-root. It is not possible to install any
binary or library in /lib when it has link dependencies to libs
in /usr (e.g. database drivers).

At early boot time, syslo-ng is started with syslog-ng-early.conf;
When /usr is available, syslog-ng can be reloaded or even restarted
(using a different config path) to use the full blown config.

The early version could log into some persistent store, where the
full blown one pickups the logs later at start time.
Bazsi said…
Thanks, the idea about the multiple plugin directories certainly makes sense.

I'm still making a couple of changes to the configuration parser to make it easier to use and after that I'm going to work on the plugin loading functionality again.
I dunno if this is still an open issue or not, regarding the module loading, but perhaps a way to define and/or load module groups? That way people don't have to explicitly list out every module they want.

So, for example, you might have something like:

@module: CORE

That would load a set of core modules matching what most people would expect (probably based on feature availability from previous releases of Syslog-NG).
Bazsi said…
In the meanwhile the implementation has progressed, and I've came up with two possible ways to use syslog-ng core modules easily.

1) if the autoload-compiled-modules global variable is "1" (the default), then all compile-time-enabled modules will be loaded automatically.

2) I've added default configuration files, which ships a "modules.conf" file included into the main config, which explicitly loads modules. This config file can then be customized.

I have a lot of changes in the git tree, but didn't get to blog about it yet. However there are really important changes there.

Popular posts from this blog

syslog-ng fun with performance

I like christmas for a number of reasons: in addition to the traditional "meet and have fun with your family", eat lots of delicious food and so on, I like it because this is the season of the year when I have some time to do whatever I feel like. This year I felt like doing some syslog-ng performance analysis. After reading Ulrich Deppert's series about stuff "What every programmer should know about memory" on LWN, I thought I'm more than prepared to improve syslog-ng performance. Before going any further, I'd recommend this reading to any programmer, it's a bit long but every second reading it is worth it. As you need to measure performance in order to improve it, I wrote a tool called "loggen". This program generates messages messages at a user-specifyable rate. Apart from the git repository you can get this tool from the latest syslog-ng snapshots. Loggen supports TCP, UDP and UNIX domain sockets, so really almost everything can be me

syslog-ng roadmap 2.1 & 2.2

We had a meeting on the syslog-ng roadmap today where we decided some important things, and I thought I'd use this channel to tell you about it. The Open Source Edition will see a 2.1 release incorporating all core changes currently in the Premium Edition and additionally the SQL destination driver. We are going to start development on the 2.2 PE features, but some of those will also be incorporated in the open source version: support for the latest work of IETF syslog protocols unique sequence numbering for messages support for parsing message contents Previously syslog-ng followed the odd/even version numbering to denote development/stable releases. I'm going to abandon this numbering now: the next syslog-ng OSE release is going to have a 2.1 version number and will basically come out with tested code changes only. The current feature set in PE were developed in a closed manner and I don't want to repeat this mistake. The features that were decided to be part of the Open

syslog-ng 3.0 and SNMP traps

Last time I've written about how syslog-ng is able to change message contents. I thought it'd be useful to give you a more practical example, instead of a generic description. It is quite common to convert SNMP traps to syslog messages. The easiest implementation is to run snmptrapd and have it create a log message based on the trap. There's a small issue though: snmptrapd uses the UNIX syslog() API, and as such it is not able to propagate the originating host of the SNMP trap to the hostname portion of the syslog message. This means that all traps are logged as messages coming from the host running snmptrapd, and the hostname information is part of the message payload. Of course it'd be much easier to process syslog messages, if this were not the case. A solution would be to patch snmptrapd to send complete syslog frames, but that would require changing snmptrapd source. The alternative is to use the new parse and rewrite features of syslog-ng 3.0. First, you need to f