Skip to main content

syslog-ng roadmap 2.1 & 2.2

We had a meeting on the syslog-ng roadmap today where we decided some important things, and I thought I'd use this channel to tell you about it.

The Open Source Edition will see a 2.1 release incorporating all core changes currently in the Premium Edition and additionally the SQL destination driver. We are going to start development on the 2.2 PE features, but some of those will also be incorporated in the open source version:
  • support for the latest work of IETF syslog protocols
  • unique sequence numbering for messages
  • support for parsing message contents
Previously syslog-ng followed the odd/even version numbering to denote development/stable releases. I'm going to abandon this numbering now: the next syslog-ng OSE release is going to have a 2.1 version number and will basically come out with tested code changes only.

The current feature set in PE were developed in a closed manner and I don't want to repeat this mistake. The features that were decided to be part of the Open Source version will be developed as openly as possible: the features listed above are going to be developed and published in the open source branch with version number 2.2.x. The "alpha" and "beta" releases are going to be numbered 2.2alpha1, 2.2beta1 etc; the final stable is going to be called 2.2.1.

The aim is to avoid the mess of PE and OSE having a different version number.

Comments

Anonymous said…
I see from the datasheet that syslog-ng PE have syslog architecture management. do you have any feature list? the product was release? and if not do you have timeline for it?
3:23 PM
Bazsi said…
I'm not sure what you are referring to exactly, there's a product in the works that is based on syslog-ng PE, but I'm not sure it's what you mean.
9:46 AM
Anonymous said…
Hello Bazsi! Thank you for good software (syslog-ng). You wrote on your blog that "The Open Source Edition will see a 2.1 release incorporating all core changes currently in the Premium Edition".

Does this mean that the syslog-ng OSE 2.1 now includes TLS support for protecting the logs when travelling from the log source to the central log server?

Sincerely,

/Fredrik
8:38 AM
Bazsi said…
No, this does not include TLS. Under "core" changes I meant all changes to the syslog-ng core functionality, e.g. message processing, some of the performance improvment patches, etc.

Feature-wise, syslog-ng OSE 2.1 includes only the "SQL" destination as an addition.

Sorry for the confusing language in my original post.
3:36 PM
Matt P. said…
Hello: I am using syslog-ng version 1.6.12. I created a conf file containing 1250 filter statements where each filter contained two "match" keywords for two unique strings in each message (note: I did not use any regular expressions). Next, I created a single log statement using just one of the filters.

Using linux 'time' utility, I ran a shell script to log messages using linux "logger" shell cmd. What I found was that real time to have the messages processed by syslog-ng took much longer when I had a conf file with 1250 filters (but only using one) versus a conf file with a single filter statement. The difference in real time was in the order of 3 to 4 times longer in the case with 1250 filters versus a single. Why is there such a big difference in these two cases? My single log statement is the same syntax, specifying a single filter. Have any performance improvements been made to syslog-ng to address this testcase? I tried simplifying the filter name and still no luck in improving performance. Any suggestions? The reason I am creating 1250 filters is because in our installation, we have more than 127 facilities (LOG_FACMASK is too small), so I am thinking I need to parse the msg portion of my messages to achieve the behavior I need.

Thanks.
11:12 PM
Bazsi said…
Did you measure the time needed for "logger" or "syslog-ng"?

Another thing that affect your scenario is that syslog-ng resolves filter references once. This resolution takes longer if you have 1250 filters than if you have only 1.

Are you sure you sent enough messages to overcome the slower startup time?
11:20 AM
Matt P. said…
Hello: Thanks for responding to my question. I was measuring the time for my shell script to complete. So time reflects how long logger commands were taking as well. I am unclear what you mean by resolve filter references once. Here is a snapshot of the times I am seeing. Where the only differece between the two runs is the number of filters defined in the conf

# Define 1250 filters but only use one in a log statement
[root@slimjim logging]# vim /etc/syslog-ng/syslog-ng.conf
[root@slimjim logging]# killall -HUP syslog-ng

# kasi.sh script contains 4000 logger shell cmds issuing various different message strings.
[root@slimjim logging]# vim kasi.sh
[root@slimjim logging]# time ./kasi.sh

real 0m7.632s
user 0m1.079s
sys 0m2.787s
[root@slimjim logging]# time ./kasi.sh

real 0m7.473s
user 0m1.032s
sys 0m2.914s
[root@slimjim logging]# time ./kasi.sh

real 0m7.439s
user 0m1.025s
sys 0m2.969s
[root@slimjim logging]# time ./kasi.sh

real 0m7.391s
user 0m1.053s
sys 0m2.943s
[root@slimjim logging]# time ./kasi.sh

real 0m7.425s
user 0m1.033s
sys 0m2.938s
[root@slimjim logging]# time ./kasi.sh

real 0m7.389s
user 0m1.038s
sys 0m2.969s

# changing syslog-ng to have one filter statement defined with the same single log statement as above

[root@slimjim logging]# vim /etc/syslog-ng/syslog-ng.conf

[root@slimjim logging]# killall -HUP syslog-ng
[root@slimjim logging]# time ./kasi.sh

real 0m4.311s
user 0m1.105s
sys 0m3.290s
[root@slimjim logging]# time ./kasi.sh

real 0m4.342s
user 0m1.080s
sys 0m3.312s
[root@slimjim logging]# time ./kasi.sh

real 0m4.343s
user 0m1.042s
sys 0m3.382s
[root@slimjim logging]# time ./kasi.sh

real 0m4.316s
user 0m1.055s
sys 0m3.365s
[root@slimjim logging]# time ./kasi.sh

real 0m4.310s
user 0m1.133s
sys 0m3.291s
[root@slimjim logging]# time ./kasi.sh

real 0m4.318s
user 0m1.070s
sys 0m3.319s
[root@slimjim logging]#

I would imagine my shell script time is fixed in this equation.

Thanks again for taking the time to help me understand the difference in time.
4:26 PM
Bazsi said…
Hm... interesting. The time you measured was not directly related to the CPU time used by syslog-ng, but it indicates the latency that happens when syslog-ng accepts a UNIX domain connection from the logger command.

I don't see how those could relate, however syslog-ng 1.6.x is very old code, I might not remember everything.

In syslog-ng 2.0.x the number of filter statements should only affect startup time.
4:47 PM
Kasi Viswanath said…
Hi Bazsi, Thanks for syslog-ng.

Can we increase the number of facilities that can be supported by syslog-ng. Currently, it's 128. Basically, my requirement would be to increase those 7 bits to atleast 10 bits. How complicated would you think, the code changes be for this? Also, are there any side-effects in doing so?

Thanks in advance,
1:24 AM
Bazsi said…
syslog-ng already does support facilities beyond the 7 bits that the syslog RFC specifies. You only have to specify the facility by number instead of by name.

So use something like:

filter f_custom_fac { facility(50); };

Facility ranges are not supported in this case.

The only question is how you convince the application to use your custom facility.
10:04 AM
Anonymous said…
Hi,

Does syslog-ng OSE 3.0.x has support for TLS..?
11:51 AM
Bazsi said…
yes.
4:23 PM
Post a Comment

Popular posts from this blog

syslog-ng fun with performance

I like christmas for a number of reasons: in addition to the traditional "meet and have fun with your family", eat lots of delicious food and so on, I like it because this is the season of the year when I have some time to do whatever I feel like. This year I felt like doing some syslog-ng performance analysis. After reading Ulrich Deppert's series about stuff "What every programmer should know about memory" on LWN, I thought I'm more than prepared to improve syslog-ng performance. Before going any further, I'd recommend this reading to any programmer, it's a bit long but every second reading it is worth it. As you need to measure performance in order to improve it, I wrote a tool called "loggen". This program generates messages messages at a user-specifyable rate. Apart from the git repository you can get this tool from the latest syslog-ng snapshots. Loggen supports TCP, UDP and UNIX domain sockets, so really almost everything can be me...
17 comments
Read more

syslog-ng OSE 2.1 released

I have just uploaded the first release in the syslog-ng Open Source Edition 2.1 branch to our website. It is currently only available in source format at this location: http://www.balabit.com/downloads/files/syslog-ng/sources/2.1/src This release synchronizes the core of syslog-ng to the latest PE version and adds the SQL destination driver. This is an alpha release and thus might be rough around the edges, but it basically only contains code already tested in the context of the Premium Edition. The SQL functionality requires a patched libdbi package, which is available at the same link. We're going to work on integrating all our libdbi related patches to the upstream package. If you want to know how the SQL logging works, please see the Administrator's Guide or our latest white paper Collecting syslog messages into an SQL database with syslog-ng. The latter describes the Premium Edition, but it applies to the Open Source one equally well.
6 comments
Read more

syslog-ng 3.2 changes

I've just pushed a round of updates to the syslog-ng 3.2 repository, featuring some interesting stuff, such as: SQL reorganization: Patrick Hemmer sent in a patch to implement explicit transaction support instead of the previous auto-commit mode used by syslog-ng. I threw in some fixes and refactored the code somewhat. Configuration parser changes: the syntax errors produced by syslog-ng became much more user-friendly: not only the column is displayed, but also the erroneous line is printed and the error location is also highlighted. Additional plugin modules were created: afsql for the SQL destination, and afstreams for Solaris STREAMS devices. Creating a new plugin from core code takes about 15 minutes. I'm quite satisfied. With the addition of these two modules, it is now possible to use syslog-ng without any kind of runtime dependency except libc. The already existing afsocket module (providing tcp/udp sources & destinations) is compiled twice: once with and once withou...
4 comments
Read more