syslog-ng disk based buffering

I've just seen a post on the loganalysis mailing list how easy it is to implement disk-based buffering with perl and a few hours time. The implementation would be as simple as sending the messages to a file and using a script like "tail -f" to follow the file and send messages to the desired log collectors.

Although the scheme that was described would work, I see three important problems:

• latency: because the solution would work by polling the log file, the latency is severely increased, when you have thousands of log entries per second, a second is a long time. And you don't want to poll more often than every second.
• disk usage: relaying the data would store everything on the local disk, no upper bound on disk usage, if the disk is full, data is lost
• load: using an interpreted language and the requirement to store all data on disk puts an enormous load on the system that might be spent better elsewhere.
  

Syslog-ng on the other hand uses a spool file, but this file is only written when the memory based buffer becomes full. For the generic case all messages come and go, without touching the disk at all or having to poll the disk for changes. The disk space requirements are bounded. Disk buffering with the combination of flow-control and disk buffers you can avoid message loss.

And what's more, the disk spooling in syslog-ng is an independent feature for all flow-controllable destinations: tcp(), unix-stream(), pipe(), program() and also sql().

Adding a disk buffer to a destination is as simple as specifying the buffer size in bytes:

destination d_tcp { tcp("logserver" log_disk_fifo_size(100000000)); };

This means that a 100MB of space is allocated for disk-based spooling to store messages whenever "logserver" is not fast enough or is unavailable.

For more information read the syslog-ng documentation about the way this feature works.