As the readers of this blog might know I've been working on persistent disk-buffering and SQL support recently. The configuration interface of the SQL destination became quite close to the description I gave in my last post.
I think from the user side it is pretty neat, no need to rely on the mysql client program or to create "buffer files" that are later fed to the database server. You simply define an SQL destination and tables are created automagically with proper (possibly even disk-based) queueing, flow control and error handling. And by using libdbi we immediately have support for 4 or 5 different database servers.
The implementation side was somewhat more intrusive however, the client libraries for various database servers use a blocking I/O model and syslog-ng was completely non-blocking until now. I had to create a separate thread for inserting records to tables, and there came some required changes to syslog-ng to support that: various reference counters and the internal acknowledge mechanism had to be made thread-aware.
This was not without benefit though, a change of this scale required a thorough review of the code involved and I have found and fixed several bugs that also affected the 2.0.x tree. I have committed some of these to my public git tree already, so it should be synchronized to our public webserver real soon now.
My next thing to do is to prepare 2.0.4 and probably a first public release of the 2.1.x open source tree. Stay tuned. :)
I think from the user side it is pretty neat, no need to rely on the mysql client program or to create "buffer files" that are later fed to the database server. You simply define an SQL destination and tables are created automagically with proper (possibly even disk-based) queueing, flow control and error handling. And by using libdbi we immediately have support for 4 or 5 different database servers.
The implementation side was somewhat more intrusive however, the client libraries for various database servers use a blocking I/O model and syslog-ng was completely non-blocking until now. I had to create a separate thread for inserting records to tables, and there came some required changes to syslog-ng to support that: various reference counters and the internal acknowledge mechanism had to be made thread-aware.
This was not without benefit though, a change of this scale required a thorough review of the code involved and I have found and fixed several bugs that also affected the 2.0.x tree. I have committed some of these to my public git tree already, so it should be synchronized to our public webserver real soon now.
My next thing to do is to prepare 2.0.4 and probably a first public release of the 2.1.x open source tree. Stay tuned. :)
Comments