Tuesday, April 24, 2007

Persistent disk-buffering in syslog-ng

Two blog spots in a row on two consecutive days, wow :) Things are happening fast these times. Because of my company's efforts to create a commercial fork of syslog-ng, I have somewhat more time to do syslog-ng hacking. This time I've finished persistent disk-buffering, an often requested feature to be released in the commercial version.

It means that if a target server is down, then in addition to the in-memory buffers, syslog-ng is able to store messages in a disk-based queue until the connection is restored. What's more, this queue is persistent and syslog-ng keeps its contents accross restarts.

One less reason to keep logs locally. :)

Next on my list is native SQL support, combined with this disk-buffer feature to cover times when the database is too slow processing INSERTs. I'm thinking along the lines of:

destination d_sql {
sql(type(pgsql) host("loghost") user("syslog-ng") password("secret-password")
database("logs")
table("messages_
${HOST}_${R_YEAR}${R_MONTH}${R_DAY}")
columns("date", "host", "program", "pid", "message")
values("$R_DATE", "$HOST", "$PROGRAM", "$PID", "$MSGONLY"));


};

Tables would be created/altered automatically on-demand, just like destination files. Values might refer to builtin macros, or matches within the message (like in sed, e.g. $1 refers to the first match).

The actual implementation might be somewhat different though.

No comments: