I'm proud to announce that syslog-ng OSE 3.1 has been released and uploaded to our webserver. This version is new in two ways:
1) of course it has new features, see below for the most interesting bits
2) it is a "feature release", which means that once syslog-ng 3.2 or syslog-ng 4.0 is released, the support for this release will be ceased. See our new version policy at this link:
https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/roadmap.bbx
Since the documentation is not yet up to date with this beta release, I'll try to include the most crucial information about the new features right here in this announcement.
For those who hurry, here's a link for the source code:
https://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.1beta1/source/syslog-ng_3.1beta1.tar.gz
And here are the binaries for Linux/FreeBSD systems:
https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/
Select the Downloads tab, and in the Version selector select 3.1beta1.
Please try this beta version. Any feedback, positive or negative is appreciated, if you have comments, please post them on the mailing list at: syslog-ng@lists.balabit.hu
And now the new features in this release:
Support for patterndb v3
syslog-ng 3.1 now supports the patterndb v3 format, along with a bunch of new parsers: ANYSTRING, IPv6, IPvANY and FLOAT.Patterndb (more exactly the db-parser()) is a high performance message classifier and information extraction tool, that makes it easy to get away from the unstructured nature of syslog.
Patterndb has evolved since it was first introduced in syslog-ng 3.0. It is at the 3rd iteration, hopefully slowly reaching its final form. syslog-ng OSE 3.0 supported v1, our SSB product supports v2 and now syslog-ng OSE is the first version supporting v3.
Patterndb in general and the v1 format database is described in the syslog-ng manual at:
http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch02s12.html
The XML schemas that describe the different patterndb versions are available in the syslog-ng source tree:
http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=tree;f=doc/xsd;hb=HEAD
The changes in the patterndb format as they evolved were described in Marton Illes's blog at
http://marci.blogs.balabit.com/2009/06/new-db-parser-format-and-other.html
But see the other related posts as well.
Old patterndb databases can be converted to the new format by putting them in the /opt/syslog-ng/etc/patterns.d directory and using the pdbtool utility using the command:
$ pdbtool merge -p /opt/syslog-ng/var/patterndb.xml \
-D /opt/syslog-ng/etc/patterns.d
Assuming the installation prefix of syslog-ng is /opt/syslog-ng. The above filenames are also the recommended and default names for patterndb related files.
Some v2 format patterns are distributed by BalaBit itself for its SSB product, download location:
https://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/
You can convert these db files using pdbtool as described above.
Work is ongoing to publish a more comprehensive patterndb, but more on that in a separate post.
pdbtool
Added a new "pdbtool" utility to manage patterndb files: convert them from v1 or v2 format, merge mulitple patterndb files into one and look up matching patterns given a specific message.
See the manpage (by adding /opt/syslog-ng/share/man in the MANPATH) and Marci's post:
http://marci.blogs.balabit.com/2009/08/db-parser-new-utility-pdbtool.html
Message tags
Support for message tags was added: tags can be assigned to log messages as they enter syslog-ng: either by the source driver or via patterndb. Later it these tags can be used for efficient filtering.
http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html
Rewrite structured data
Earlier structured data fields in the new RFC5424 style syslog protocol were only read-only values that could be referenced in a template, but they couldn't be changed, and neither was it possible to add new fields in an already existing syslog message.
Now all these became possible by using the same syntax that didn't work earlier, e.g.
rewrite r_sd { set("55555" value(".SDATA.meta.sequenceId")); };
Macro and name-value integration
Macros and name-value pairs got a little tighter integration. syslog-ng 3.0 was limited in the use of macros in the value() option of the match() filter: it could only use name-value pairs, although intiutively it should have supported macros as well. This was changed, starting with 3.1 it is now possible to use macros as well.
The following now works:
match("regexp" value("R_DATE"));
syslog-ng is now warning you in case you are using '
Name-value pair performance improvements
With the advent of patterndb and the spreading use of name-value pairs in syslog-ng, a strong limitation was the performance penalty of using dynamically created name-value pairs. This was now solved, 3.1 features a new data structure to store message payload and name-value pairs in, which results in a 3 times better performance when looking up a name-value pair.
Patterndb parser enhancements
Some parsers got additional features: NUMBER is now able to parse hexadecimal numbers, ESTRING is now able to search for a sequence of characters as the end of the string. These changes make it easier to describe log messages in patterns.
Information about non-portable facilities
Added non-standard and non-portable facility codes (range 10-15), and decoupled syslog-ng facility name database from the system used to compile syslog-ng on.
Until this time the facility codes as understood by syslog-ng were dependant on the platform syslog-ng was compiled on. This is not true anymore, syslog-ng comes with its own "facility" code assignments, based on the RFC, and adding some non-standard values found on various UNIX systems. prefix in the value syntax, because you can't use the full template syntax when you specify a value to match against.
Name-value pair performance improvements
With the advent of patterndb and the spreading use of name-value pairs in syslog-ng, a strong limitation was the performance penalty of using dynamically created name-value pairs. This was now solved, 3.1 features a new data structure to store message payload and name-value pairs in, which results in a 3 times better performance when looking up a name-value pair.
Patterndb parser enhancements
Some parsers got additional features: NUMBER is now able to parse hexadecimal numbers, ESTRING is now able to search for a sequence of characters as the end of the string. These changes make it easier to describe log messages in patterns.
Information about non-portable facilities
Added non-standard and non-portable facility codes (range 10-15), and decoupled syslog-ng facility name database from the system used to compile syslog-ng on.
Until this time the facility codes as understood by syslog-ng were dependant on the platform syslog-ng was compiled on. This is not true anymore, syslog-ng comes with its own "facility" code assignments, based on the RFC, and adding some non-standard values found on various UNIX systems.
1) of course it has new features, see below for the most interesting bits
2) it is a "feature release", which means that once syslog-ng 3.2 or syslog-ng 4.0 is released, the support for this release will be ceased. See our new version policy at this link:
https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/roadmap.bbx
Since the documentation is not yet up to date with this beta release, I'll try to include the most crucial information about the new features right here in this announcement.
For those who hurry, here's a link for the source code:
https://www.balabit.com/downloads/files/syslog-ng/open-source-edition/3.1beta1/source/syslog-ng_3.1beta1.tar.gz
And here are the binaries for Linux/FreeBSD systems:
https://www.balabit.com/network-security/syslog-ng/opensource-logging-system/
Select the Downloads tab, and in the Version selector select 3.1beta1.
Please try this beta version. Any feedback, positive or negative is appreciated, if you have comments, please post them on the mailing list at: syslog-ng@lists.balabit.hu
And now the new features in this release:
Support for patterndb v3
syslog-ng 3.1 now supports the patterndb v3 format, along with a bunch of new parsers: ANYSTRING, IPv6, IPvANY and FLOAT.Patterndb (more exactly the db-parser()) is a high performance message classifier and information extraction tool, that makes it easy to get away from the unstructured nature of syslog.
Patterndb has evolved since it was first introduced in syslog-ng 3.0. It is at the 3rd iteration, hopefully slowly reaching its final form. syslog-ng OSE 3.0 supported v1, our SSB product supports v2 and now syslog-ng OSE is the first version supporting v3.
Patterndb in general and the v1 format database is described in the syslog-ng manual at:
http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch02s12.html
The XML schemas that describe the different patterndb versions are available in the syslog-ng source tree:
http://git.balabit.hu/?p=bazsi/syslog-ng-3.1.git;a=tree;f=doc/xsd;hb=HEAD
The changes in the patterndb format as they evolved were described in Marton Illes's blog at
http://marci.blogs.balabit.com/2009/06/new-db-parser-format-and-other.html
But see the other related posts as well.
Old patterndb databases can be converted to the new format by putting them in the /opt/syslog-ng/etc/patterns.d directory and using the pdbtool utility using the command:
$ pdbtool merge -p /opt/syslog-ng/var/patterndb.xml \
-D /opt/syslog-ng/etc/patterns.d
Assuming the installation prefix of syslog-ng is /opt/syslog-ng. The above filenames are also the recommended and default names for patterndb related files.
Some v2 format patterns are distributed by BalaBit itself for its SSB product, download location:
https://www.balabit.com/downloads/files/patterndb/1.0-20081117/patterndb/
You can convert these db files using pdbtool as described above.
Work is ongoing to publish a more comprehensive patterndb, but more on that in a separate post.
pdbtool
Added a new "pdbtool" utility to manage patterndb files: convert them from v1 or v2 format, merge mulitple patterndb files into one and look up matching patterns given a specific message.
See the manpage (by adding /opt/syslog-ng/share/man in the MANPATH) and Marci's post:
http://marci.blogs.balabit.com/2009/08/db-parser-new-utility-pdbtool.html
Message tags
Support for message tags was added: tags can be assigned to log messages as they enter syslog-ng: either by the source driver or via patterndb. Later it these tags can be used for efficient filtering.
http://marci.blogs.balabit.com/2009/05/tag-support-in-syslog-ng.html
Rewrite structured data
Earlier structured data fields in the new RFC5424 style syslog protocol were only read-only values that could be referenced in a template, but they couldn't be changed, and neither was it possible to add new fields in an already existing syslog message.
Now all these became possible by using the same syntax that didn't work earlier, e.g.
rewrite r_sd { set("55555" value(".SDATA.meta.sequenceId")); };
Macro and name-value integration
Macros and name-value pairs got a little tighter integration. syslog-ng 3.0 was limited in the use of macros in the value() option of the match() filter: it could only use name-value pairs, although intiutively it should have supported macros as well. This was changed, starting with 3.1 it is now possible to use macros as well.
The following now works:
match("regexp" value("R_DATE"));
syslog-ng is now warning you in case you are using '
Name-value pair performance improvements
With the advent of patterndb and the spreading use of name-value pairs in syslog-ng, a strong limitation was the performance penalty of using dynamically created name-value pairs. This was now solved, 3.1 features a new data structure to store message payload and name-value pairs in, which results in a 3 times better performance when looking up a name-value pair.
Patterndb parser enhancements
Some parsers got additional features: NUMBER is now able to parse hexadecimal numbers, ESTRING is now able to search for a sequence of characters as the end of the string. These changes make it easier to describe log messages in patterns.
Information about non-portable facilities
Added non-standard and non-portable facility codes (range 10-15), and decoupled syslog-ng facility name database from the system used to compile syslog-ng on.
Until this time the facility codes as understood by syslog-ng were dependant on the platform syslog-ng was compiled on. This is not true anymore, syslog-ng comes with its own "facility" code assignments, based on the RFC, and adding some non-standard values found on various UNIX systems. prefix in the value syntax, because you can't use the full template syntax when you specify a value to match against.
Name-value pair performance improvements
With the advent of patterndb and the spreading use of name-value pairs in syslog-ng, a strong limitation was the performance penalty of using dynamically created name-value pairs. This was now solved, 3.1 features a new data structure to store message payload and name-value pairs in, which results in a 3 times better performance when looking up a name-value pair.
Patterndb parser enhancements
Some parsers got additional features: NUMBER is now able to parse hexadecimal numbers, ESTRING is now able to search for a sequence of characters as the end of the string. These changes make it easier to describe log messages in patterns.
Information about non-portable facilities
Added non-standard and non-portable facility codes (range 10-15), and decoupled syslog-ng facility name database from the system used to compile syslog-ng on.
Until this time the facility codes as understood by syslog-ng were dependant on the platform syslog-ng was compiled on. This is not true anymore, syslog-ng comes with its own "facility" code assignments, based on the RFC, and adding some non-standard values found on various UNIX systems.
Comments