Bazsi's Home Automation blog

I've just pushed out an update to syslog-ng, which contains various HP-UX fixes backported from the Premium Edition of syslog-ng. The GPL version of syslog-ng should now work flawlessly on HP-UX. There's one caveat though: there's a buggy system header in HP-UX and gcc 4.x fails to compile it. I copied this header to the gcc private include directory (/usr/local/lib/gcc/hppa2.0w-hp-hpux11.11/4.1.0/include on my system) and applied the following patch: 438a439 > #ifndef _APP32_64BIT_OFF_T 442a444 > #endif (HP-UX diff does not know how to produce unified diffs), the point is that there are two conflicting declarations of a function and the preprocessor conditional above fixes that. Once this patch is in place, and you have the proper build dependencies syslog-ng works fine on HP-UX. As I've just pushed these changes to my git repository, you'll need to wait for another day to get a daily snapshot. But hey "git" is what the pros use :)

Apart from the previous entry I was not posting to this blog for two months. The reason was that I was heavily involved in the development of BalaBit Audit Player, a graphical application to replay RDP/SSH sessions, recorded by our SCB product . BAP became much larger than I originally expected, it's about 20k lines of code, and the end of the development was done in a rush to meet our deadline of Sep 1, 2007. We've slipped a couple of days, but we've released BAP 2.0.0 on 7th September. Then I spent a week in Karlsruhe on the 5th Netfilter Developer's Workshop. I returned to Hungary on Friday, I'm spending the weekend with my parents, and hopefully I can be more active on other things, like the syslog-ng mailing list, or this blog. :) A release of syslog-ng GPL is long due, hopefully I can prepare it next week. I'll also need to schedule some syslog-ng development time as there are some open feature requests by customers.

I've just returned from this year's Netfilter Developer's Workshop , this time held in Karlsruhe, Germany. This year's workshop was the fifth such event, and this time even David S. Miller was there. All in all the organization was wonderful, kudos to the Astaro guys. We've had about 30 attendants, the largest workshop ever. You can read more about the workshop at http://nfws.inl.fr/en/ Krisztian Kovacs and me were trying to push our TProxy4 patches for merging, the future for tproxy seems bright, as everyone was positive. We also have some other, minor patches in the queue, I'm working on finalizing them and submit them for inclusion. These are: SO_MARK socket option to be able to specify the mark field of outgoing packets, generated sent from a given socket an extension to the "addrtype" match to limit the match to the incoming interface the notion of interface groups , that make it possible to match a group of similar interfaces As always, it was ...

I was spending the last couple of hours to implement a simple SFTP proxy, that is capable of logging file transfers, into our Shell Control Box product line. The core idea behind SCB is to perform RDP/SSH screening independently from the end-systems. This SFTP functionality will be a small bonus: in addition to dumping the SSH traffic to an audit trail, we are going to be able to write log transactions to syslog, which is way easier to analyse, if all you want to know is the list of files accessed via SFTP. I originally thought that SFTP was as simple as FTP, with a transaction being a complete file transfer. On the contrary, SFTP is much closer to NFS (and other network file system protocols) in spirit: in FTP you have a "RETR" command that fetches a complete file, in SFTP you need to open the file and read it separately using a series of "READ" commands. Now I understand how sshfs is possible. I thought I'd let you know :) By the way, syslog-ng 2.0.5 has be...

Things were progressing steadily on the syslog-ng front. About 4 weeks ago I released syslog-ng Premium Edition 2.1.5 which was the first version with integrated SQL support. Since then the first production deployments of PE have been done, shaking out some newly introduced bugs in the process (thus the releases 2.1.5a - 2.1.5d). Disk buffering works nicely, especially when combined with the new rate-limit option (throttle), if your backend systems are calibrated to a given rate of incoming messages, syslog-ng can ensure that the limit is never exceeded. Whenever bursts end, syslog-ng feeds the messages towards the back-end systems in their idle time. This introduced some latency though. I've started working on support for the new syslog-protocol work in IETF, but nothing is ready yet. After this is finished, I'm planning to release syslog-ng 2.1 under the GPL license, with some of the new features added. The GPLd branch also saw a couple of fixes, no release though. If you wan...

I forgot to mention here that syslog-ng 2.0.4 was released about 10 days ago. Before making an upgrade, be sure to revalidate your max-connections() setting. As previous versions in the 2.0.x tree has failed to enforce this limit. Apart from the usual bunch of bug fixes, this release features case-insensitive regexps and the ability to track logrotated source files. Now you can actually track any log file, even if it is rotated automatically. We have also worked hard on the new syslog-ng webpage. Hopefully no information was lost during the transition. If you miss anything that was present on the old one and not available on the new, please let me know . I'm planning to set up a public bugzilla and a wiki for syslog-ng, so that it will have the standard infrastructure needed for a modern open source projects. Version control, bug tracking and Wiki. On the Premium Edition front, both the syslog-ng Agent program for Windows and syslog-ng has seen a public release. Check out the s...

As the readers of this blog might know I've been working on persistent disk-buffering and SQL support recently. The configuration interface of the SQL destination became quite close to the description I gave in my last post . I think from the user side it is pretty neat, no need to rely on the mysql client program or to create "buffer files" that are later fed to the database server. You simply define an SQL destination and tables are created automagically with proper (possibly even disk-based) queueing, flow control and error handling. And by using libdbi we immediately have support for 4 or 5 different database servers. The implementation side was somewhat more intrusive however, the client libraries for various database servers use a blocking I/O model and syslog-ng was completely non-blocking until now. I had to create a separate thread for inserting records to tables, and there came some required changes to syslog-ng to support that: various reference counters and ...

Two blog spots in a row on two consecutive days, wow :) Things are happening fast these times. Because of my company's efforts to create a commercial fork of syslog-ng, I have somewhat more time to do syslog-ng hacking. This time I've finished persistent disk-buffering, an often requested feature to be released in the commercial version. It means that if a target server is down, then in addition to the in-memory buffers, syslog-ng is able to store messages in a disk-based queue until the connection is restored. What's more, this queue is persistent and syslog-ng keeps its contents accross restarts. One less reason to keep logs locally. :) Next on my list is native SQL support, combined with this disk-buffer feature to cover times when the database is too slow processing INSERTs. I'm thinking along the lines of: destination d_sql { sql(type(pgsql) host("loghost") user("syslog-ng") password("secret-password") database("logs"...

We have been using GNU arch the last couple of years as a version control system, however Tom Lords' implementation does not scale well, and some of our software packages have 10 thousands of commits. This means that a single commit operation may take _minutes_. It is awful to wait so much time for a single commit, and it really degrades productivity. I was considering Mercurial, Bazaar-NG and git, however this was not an easy decision, as the "modern" version control systems promote the use of branches over anything else, and our current version control model relied on cherry-picking heavily: developer commits the solution for each bug separately to his/her branch QA people pick patches from developer branches and integrate them to a 'test' branch, once the test was successful, release manager picks patches from the 'test' branch and integrates to mainline, if he doesn't find anything odd during review This worked wonderfully in GNU arch, but new VC s...

I've just finished preparations for the release of syslog-ng 2.0.3 and now I consider the 2.0.x branch feature complete. As far as I know there's no feature that was present in 1.6.x and missing from the 2.0.x rewrite. The last missing bits were spoof-source and TCP wrapper support, but as those are present in 2.0.3 I consider syslog-ng 2.0 feature complete. I don't plan to add further code that would cause destabilization. Now some plans about the newly opened 2.1 branch: we (as BalaBit) played with an idea of creating a commercial fork of syslog-ng which could increase the amount of resources that I can allocate to developing syslog-ng code. From 2.1 on, two parallel editions of syslog-ng will become available: syslog-ng 2.1 Open Source Edition: the same as syslog-ng 2.0, available under the GPL syslog-ng 2.1 Premium Edition: a commercial fork of syslog-ng, The two releases will be produced from the same source tree with some of the commercial functionality stripped duri...

As a commenter missed spoof-source support from 2.0, I got my act together for an afternoon hacking session and implemented it in the 2.0.x tree. The packet generation for IPv4 packets was straightforward, I could simply "forward-port" it from 1.6.x, after creating the proper place for it in the 2.0.x tree. IPv6 support was a bit more difficult as libnet has a nasty bug with IPv6 and UDP. I was looking at my code for hours, when I tried Google codesearch to check whether I was doing something wrong. Then I've found this . Note the comment above the libnet_build_ipv6() call. I also disabled UDP checksumming and now it works like a charm. The last libnet release was almost 3 years ago, it is not reassuring that its webpage is also down. Too bad, libnet is a fine piece of software, but now as its website is down where can I point syslog-ng users to download libnet from? By the way, I did not announce syslog-ng 2.0.2 here, it was released about a week ago. Nothing fancy, pri...

It's been a while since I last posted here. The reason is simple: work and the lack of time. OK, I know, people have time for what they want, but: * We are launching a nifty new product based on Zorp, but with a much tighter focus. A device capable of recording and replaying SSH sessions for audit purposes. * We have decided to migrate from C to C++ for our management interface, as it seems Gtk+ style OO programming in C really requires a skill, and a lot of developers don't have that. C++ holds their hands somewhat. Thus, I've been refreshing my C++ skills and learning gtkmm while writing a prototype. * and of course the "usual" amount of work. I've tried to provide timely response on the syslog-ng mailing list. I'm sorry if you didn't receive an answer/patch from me in time. By the way I released syslog-ng 2.0.2 yesterday. Nothing really important, some portability fixes, some usertty fixes and an option missed in the 1.6.x -> 2.0.x change. Anot...

I have released syslog-ng 2.0.1, available at the usual places . I have added various missing bits that fell out in the 1.6.x -> 2.0.x change. These include: DNS cache support, overwrite_if_older (used to be called remove_if_older), various fixes, All-in-all 2.0.0 was a successful release, hopefully this one will not be much worse. See the NEWS entry of 2.0.1 for more information.

Just a short note, an interview with me was published on linux.com, more specifically here.

As it turned out the 2.0.0 release was not so bad after all. At least I have not received show-stopper bugreports, which either means that noone is using it, or everything is fine and dandy :). Hopefully it is the latter, rc releases were tested by a few people. In the meanwhile I started adding a few missing bits that were still present in 1.6.x but I never got around to implementing in the 2.0.0 tree. Among them I readded the remove_if_older() option. By the way, I don't really like the name of this option, does anyone have a better idea? If you do, please put it in a comment here or send me an email. (I was thinking about retention_time() but I'm afraid it is more difficult to understand what it would do) The other bit is the new/shiny DNS cache, which also supports persistent entries. This means that syslog-ng can read your /etc/hosts file, resolve IPs that are present there, and use IP addresses for anything else. This removes the dependency on DNS, and should also improve...

You might have already noticed, but I thought I'd write an entry on this blog on this topic: the wait is over I have released syslog-ng 2.0.0. It took a bit longer than I have anticipated, I needed to prepare 4 rc releases, as each had some bug here or there and I really wanted to release a stable 2.0.0 I hopefully succeeded, no breakage reported so far. 2.0.0 was released last saturday, but the announcement went out only on Friday. Some rarely used functionality is still missing though (a prominent example is spoof_source), and I already committed the remove_if_older() option after the release of 2.0.0. I'm going to concentrate on filling the missing bits. I'll try to avoid committing to 1.6.x, migrating to 2.0.x is strongly recommended.

It's been a long while since my last post here, but I was really busy in the past months and after I left for a two weeks vacation to Corsica. I returned on Sunday, started to catch up with work and such but I still was not able to read my syslog-ng mailing list folder, containing almost 100 unread messages. Please be patient, if you have a question open and forgiving in the unfortunate case I'd forget to reply. On the other hand Corsica is a beatiful island, be sure to visit it if you can. Nature is almost untouched at a couple of places, everything is green and a lot of mountains. A pair of hiking shoes is a useful item if you are in Corsica. :) So the island is beatiful we had some minor nuances with waiters, as neither me or my wife speaks French, and this seems to be a sin in the eyes of Corsican waiters. So at the end we came up with cooking for ourselves, lucky us our apartment was nicely equipped with cooking gear. Back to syslog-ng, I'd really like to release 2.0.0...

After some time I needed to do some kernel coding again. To seamlessly support dynamically created interfaces in Zorp, we need something I called "interface groups". Each interface might belong to a single group that basically describes how the interface was created. For instance there's an interface group for each PPP profile, but an interface group can encapsulate interfaces created by PPTPD. It is quite difficult to match dynamic interfaces by their nature: iptables sports wildcard interface name matching with the '+' character but it only works if interface names have some kind of prefix _AND_ if you don't want to differentiate between two groups. If you have two sets of PPP devices (like in the example I described above), then you have no way to create a separate ruleset, unless you reload iptables everytime a new interface is added to the system. Adding to the burden, in Zorp we want to be able to bind a service to these dynamically created interfaces, o...

After my last requests for testing of the latest 1.9.x code base, I have received a couple of bug reports, which were fixed in recent weeks. Since I have received no reports the past two weeks I decided to name the new release as "2.0rc1" to raise awareness of the new codebase. I'm planning to create the new branch for 2.1.x, I have some exciting features in my mind, which I did not want to start before the release of 2.0.0. The old stable series 1.6.x is still supported, but expect less development time to be dedicated in maintaining that release. Build queues for various architectures are not yet up, so only a Debian sarge binary is available for those with binary maintenance contracts.

You might know that there is a standardization effort on the syslog protocol in the IETF. The work has started several years ago and the efforts produced RFC3164, the first documentation of the BSD syslog protocol after being in use for over two decades. This group also produced RFC3195 in 2001, a reliable syslog protocol using the BEEP framework which did not really take of. I personally did not implement this in syslog-ng due to its highly verbose nature and the complexity which BEEP brings in. Couple of months ago an effort started to create a simpler, but still reliable syslog protocol somewhat similar to what syslog-ng has been using for a couple of years now. First some layering was decided, e.g. to define the syslog protocol in a transport independent manner and then define various transports, like legacy UDP and TLS encrypted TCP. After syslog-transport-udp was written by Rainer Gerhards, work has started on the TLS encrypted transport and someone from Huawei (you know the Chin...