About disk queue files

Normal and reliable queue files

The key difference between disk queue files that employ the reliable(yes) option and not is the strategy they employ. Reliable disk queues guarantee that all the messages passing through them are written to disk first, and removed from the queue only after the destination has confirmed that the message has been successfully received. This prevents message loss, for example, due to AxoSyslog crashes. Of course, using the reliable(yes) option introduces a significant performance penalty as well.

Both reliable and normal disk-buffers employ an in-memory output queue (set in quot-size()) and an in-memory overflow queue (set in flow-control-window-bytes() for reliable disk-buffers, or flow-control-window-size() for normal disk-buffers). The difference between reliable and normal disk-buffers is that when the reliable disk-buffer uses one of its in-memory queues, it also stores the message on the disk, whereas the normal disk-buffer stores the message only in memory. The normal disk-buffer only uses the disk if the in-memory output buffer is filled up completely. This approach has better performance (due to fewer disk I/O operations), but also carries the risk of losing a maximum of quot-size() plus flow-control-window-size() number of messages in case of an unexpected power failure or application crash.

Size of the queue files

Disk queue files grow. Each may take up to capacity-bytes() bytes on the disk. Due to the nature of reliable queue files, all the messages traversing the queue are written to disk, constantly increasing the size of the queue file.

The disk-buffer file’s size should be considered as the configured capacity-bytes() at any point of time, even if it does not have messages in it. Truncating the disk-buffer file can slow down disk I/O operations, so AxoSyslog does not always truncate the file when it would be possible (see the truncate-size-ratio() option). If a large disk-buffer file is not desirable, you should set the capacity-bytes() option to a smaller value. Note that AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default.

Starting with AxoSyslog version 4.0, you can preallocate disk-buffer files.

Preallocating disk-buffer files

By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).

In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.

If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).