Using disk-based and memory buffering

• 1: Enabling reliable disk-based buffering
• 2: Enabling normal disk-based buffering
• 3: How to get information about disk-buffer files
• 3.1: Information about disk-buffer files
• 3.2: Getting the status information of disk-buffer files
• 3.3: Getting the list of disk-buffer files
• 3.4: Printing the content of disk-buffer files
• 3.5: Orphan disk-buffer files
• 3.6: Process orphan disk-buffer files
• 4: Enabling memory buffering
• 5: About disk queue files

1 - Enabling reliable disk-based buffering

Every destination driver supports the disk-buffer() option. The network(), syslog(), tcp(), and tcp6() destination drivers can also use the disk-buffer option, except when using the udp transport method.

To enable reliable disk-based buffering, use the disk-buffer(reliable(yes)) parameter in the destination. Use reliable disk-based buffering if you do not want to lose logs in case of reload/restart, unreachable destination or AxoSyslog crash. This solution provides a slower, but reliable disk-buffer option. It is created and initialized at startup and gradually grows as new messages arrive. The filename of the reliable disk buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.rqf.

Example: Example for using reliable disk-based buffering

   destination d_BSD {
        network("127.0.0.1"
            port(3333)
            disk-buffer(
                flow-control-window-bytes(10000)
                capacity-bytes(2000000)
                reliable(yes)
            )
        );
    }; 

For details on the differences between normal and reliable disk-based buffering, see also About disk queue files.

2 - Enabling normal disk-based buffering

Every destination driver supports the disk-buffer() option. The network(), syslog(), tcp(), and tcp6() destination drivers can also use the disk-buffer option, except when using the udp transport method.

If the reliable() option is not set, by default a normal disk-buffer is created. To explicitly enable the normal disk-buffer option, use the disk-buffer(reliable(no)) parameter in the destination. Use the normal disk-buffer option if you want a solution that is faster than the reliable disk-buffer option. In this case, the process will be less reliable and it is possible to lose logs in case of AxoSyslog crash. The filename of the normal disk-buffer file is the following: <syslog-ng path>/var/syslog-ng-00000.qf.

Example: Example for using normal disk-based buffering

When using the disk-buffer plugin:

   destination d_BSD {
        network("127.0.0.1"
            port(3333)
            disk-buffer(
                flow-control-window-size(10000)
                capacity-bytes(2000000)
                reliable(no)
            )
        );
    }; 

For details on the differences between normal and reliable disk-based buffering, see also About disk queue files.

3 - How to get information about disk-buffer files

Purpose

This section describes how to get information about disk-buffer files used in AxoSyslog.

3.1 - Information about disk-buffer files

This section describes information about disk-buffer files used in AxoSyslog.

The following list contains information about how disk-buffer files are used in AxoSyslog:

• You can configure disk-buffer() for a remote destination in the destination() statement.

  For more information about an example of configuring disk-buffer() for a remote destination in the destination() statement, see disk-buffer().

• By default, AxoSyslog creates disk-buffer files under /opt/syslog-ng/var directory, unless dir() option is set in disk-buffer().

• The filenames are generated automatically by AxoSyslog with the extensions .qf for a normal disk-buffer and .rqf for a reliable disk-buffer.

• The disk-buffer file stores processed log messages in the format in which they would have been sent out to the destination, but doesn’t store information about the destination.

3.2 - Getting the status information of disk-buffer files

This section describes how to get status information of the disk-buffer files used in AxoSyslog.

Command syntax

The basic command syntax for getting the status information of the disk-buffer files used in AxoSyslog looks like the following:

/opt/syslog-ng/bin/dqtool info <DISK-BUFFER_FILE>

Example commands and outputs

• Empty, normal disk-buffer file

  /opt/syslog-ng/bin/dqtool info /opt/syslog-ng/var/syslog-ng-00000.qf/var/lib/syslog-ng/syslog-ng-00000.qf
  

  Example output:

  Disk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.qf/var/lib/syslog-ng/syslog-ng-00000.qf', number_of_messages='0'
  

• Non-empty, reliable disk-buffer queue file

  /opt/syslog-ng/bin/dqtool info /opt/syslog-ng/var/syslog-ng-00000.rqf
  

  Reliable disk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.rqf', number_of_messages='10'
  

List disk-buffer state in the default directory

You can use the following one-liner command to get the state of all disk-buffer files that are stored in the default directory:

for qfile in /opt/syslog-ng/var/*.?(r)qf ; do /opt/syslog-ng/bin/dqtool info $qfile 2>&1 ; done

3.3 - Getting the list of disk-buffer files

Purpose

This section describes getting the list of disk-buffer files used in AxoSyslog.

The AxoSyslog application stores information (namely, the IP:PORT or DNS:PORT of the destinations, and the name of the disk-buffer file) about disk-buffer files in its persist file.

Example: command for listing the disk-buffer files in use

The following command will list the disk-buffer files in use:

   /opt/syslog-ng/bin/persist-tool dump /opt/syslog-ng/var/syslog-ng.persist/var/lib/syslog-ng/syslog-ng.persist | awk -F '["=]' '/(qfile\(|\.queue)/ { gsub(/[ \t]+/, "", $5); gsub(/^[0-9A-Fa-f]{8}/, "", $5); "echo "$5"|xxd -r -p"|& getline QUEUE; printf("%s ==> %s\n",$1,QUEUE)}'

The example output will look like the following:

   afsocket_dd_qfile(stream,10.21.10.20:601)  ==> /opt/syslog-ng/var/syslog-ng-00000.rqf

   xxd: command not found

3.4 - Printing the content of disk-buffer files

Purpose

This section describes printing the content of the disk-buffer files used in AxoSyslog.

Command syntax

The command syntax for printing the content of the disk-buffer files used in AxoSyslog looks like the following:

   /opt/syslog-ng/bin/dqtool cat DISK-BUFFER_FILE

Short example output for printed content

Example: short output that shows the printed content of the disk-buffer files used in AxoSyslog

The following short output example shows the printed content of the disk-buffer files used in AxoSyslog:

   /opt/syslog-ng/bin/dqtool cat /opt/syslog-ng/var/syslog-ng-00000.rqf
    
    Reliable disk-buffer state loaded; filename='/opt/syslog-ng/var/syslog-ng-00000.rqf', queue_length='2952', size='-437712'
    Jul 31 12:33:48.226 10.21.10.10 <382019-07-31T12:33:36 localhost prg00000[1234]: seq: 0000000838, thread: 0000, runid: 1564569216, stamp: 2019-07-31T12:33:36 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
    ...

3.5 - Orphan disk-buffer files

Purpose

This section describes orphan disk-buffer files used in AxoSyslog.

Orphan disk-buffer files

In certain situations (for example, after modifying the disk-buffer configuration or losing the persist information), AxoSyslog creates a new disk-buffer file instead of using the already existing one. In these situations, the already existing disk-buffer file becomes a so-called orphan disk-buffer file.

Discovering the new disk-buffer files (orphan disk-buffer files)

To discover orphan disk-buffer files, get the list of disk-buffer files from the persist file, then compare the list with the contents of the disk-buffer files’ saving directory.

For more information about how you can get the list of disk-buffer files from the persist file, see Getting the list of disk-buffer files).

Example: difference between the list of disk-buffer files from the persist file and the content of the disk-buffer files’ saving directory

The following examples show the difference between the list of disk-buffer files from the persist file and the content of the disk-buffer files’ saving directory.

Disk-buffer file list from persist file:

   afsocket_dd_qfile(stream,10.21.10.112:514) = { "queue_file": "/opt/syslog-ng/var/syslog-ng-00001.rqf" }

Disk-buffer files’ saving directory content:

   # ls -l /opt/syslog-ng/var//var/lib/syslog-ng/*qf
    -rw------- 1 root root 2986780 Jul 31 12:30 /opt/syslog-ng/var/syslog-ng-00000.qf/var/lib/syslog-ng/syslog-ng-00000.qf
    -rw------- 1 root root 2000080 Jul 31 12:31 /opt/syslog-ng/var/syslog-ng-00000.rqf
    -rw------- 1 root root    4096 Aug  1 11:09 /opt/syslog-ng/var/syslog-ng-00001.rqf

The disk-buffer files syslog-ng-00000.qf and syslog-ng-00000.rqf don’t exist in the persist file. These two files are the orphan disk-buffer files.

For more information about orphan disk-buffer files and how to process the messages in orphan disk-buffer files using a separate AxoSyslog instance, see Process orphan disk-buffer files.

3.6 - Process orphan disk-buffer files

Purpose

When AxoSyslog creates orphan disk-buffer files, you can start a separate AxoSyslog instance parallel to the AxoSyslog instance already running, and use the following procedure to process the messages in the orphan disk-buffer file.

Orphan disk-buffer files

In certain situations (for example, after modifying the disk-buffer configuration or losing the persist information), AxoSyslog creates a new disk-buffer file instead of using the already existing one. In these situations, the already existing disk-buffer file becomes a so-called orphan disk-buffer file.

Steps

1. Identify the orphan disk-buffer files and make a record of them. For more information, see How to get information about disk-buffer files.

   It is important to know the type of the disk-buffer file. Disk-buffer file types can be normal (.qf) or reliable (.rqf). In the examples during this process, the /opt/syslog-ng/var/syslog-ng-00005.rqf orphan reliable disk-buffer file is used.

2. Determine the destination of the logs. The content of the disk-buffer may help you determine the logs’ destination. For more information, see How to get information about disk-buffer files.

   In the examples the destination 10.21.10.20 is used with the standard network() port 514.

3. Create a directory for the temporary instance. In the examples during this process, the /tmp/qdisk directory is used.

   mkdir /tmp/qdisk
   

   Warning Make sure that there is sufficient disk space in the directory. The minimum recommended disk space in the directory is equal to the size of the orphan disk-buffer file.

4. Create the configuration file /tmp/qdisk/qdisk.conf for the temporary instance with the following content.

   @version:4.9.0
   @include "scl.conf"
   
   options {
       keep-hostname(yes);
       keep-timestamp(yes);
   };
   
   destination d_destination {
   #    ADD YOUR DESTINATION HERE
   
   };
   
   log {
       destination(d_destination);
   };
   

5. Add your destination statement with disk-buffer() to the configuration file. You can copy the destination statement from your running AxoSyslog configuration.

   Warning Add the dir() option and set the disk-buffer file’s destination directory to the temporary directory (that is, /tmp/qdisk) in your destination statement.

6. Start the temporary AxoSyslog instance in the foreground.

   syslog-ng -Fe -f /tmp/qdisk/qdisk.conf -R /tmp/qdisk/qdisk.persist -c /tmp/qdisk/qdisk.ctl
   

   The AxoSyslog application will log to the console, so you will see any potential error that may occur during startup.

   The following example output displays that an empty disk-buffer file has been created and the connection to the remote destination has been established.

   Follow-mode file source not found, deferring open; filename='/no_such_file_or.dir'
   Reliable disk-buffer state saved; filename='/tmp/qdisk/syslog-ng-00000.rqf', qdisk_length='0'
   No server license found, running in client mode;
   syslog-ng starting up; version='7.0.20', cfg-fingerprint='eaa03b9efb88b87d7c1b0ce7efd042ed8ac0c013', cfg-nonce-ndx='0', cfg-signature='c0327a7f7e6418ce0399a75089377dfb662bb072'
   FIPS information; FIPS-mode='disabled'
   Syslog connection established; fd='7', server='AF_INET(10.21.10.20:514)', local='AF_INET(0.0.0.0:0)'
   

7. To stop AxoSyslog, press CTRL+C.

8. Overwrite the empty disk-buffer file with the orphan disk-buffer file.

   mv /opt/syslog-ng/var/syslog-ng-00005.rqf /tmp/qdisk/syslog-ng-00000.rqf
   

9. Start AxoSyslog using the command used in Start the temporary AxoSyslog instance in the foreground step.

   syslog-ng -Fe -f /tmp/qdisk/qdisk.conf -R /tmp/qdisk/qdisk.persist -c /tmp/qdisk/qdisk.ctl
   

10. Open another terminal and check the progress by using one of the following methods.

    • Checking the number of stored logs in the disk-buffer (that is, the last number from the output).

      /opt/syslog-ng/sbin/syslog-ng-ctl stats -c /tmp/qdisk/qdisk.ctl | grep 'dst.*queued'
      

    • Checking the status of the disk-buffer file.

      /opt/syslog-ng/bin/dqtool info /tmp/qdisk/syslog-ng-00000.rqf
      

      An empty disk-buffer file will look similar to this:

      Reliable disk-buffer state loaded; filename='/tmp/qdisk/syslog-ng-00000.rqf', queue_length='0', size='0'
      

11. Press CTRL+C to stop AxoSyslog.

12. Check the state of the orphan disk-buffer file. For more information, see How to get information about disk-buffer files.

13. If you have more than one orphan disk-buffer file, repeat the steps following the AxoSyslog stop (that is, the steps beginning from overwriting the empty disk-buffer file with the orphan disk-buffer file) for each orphan disk-buffer file.

14. Remove the temporary directory.

    rm -rf /tmp/qdisk
    

4 - Enabling memory buffering

To enable memory buffering, use the log-fifo-size() parameter in the destination. All destination drivers can use memory buffering. Use memory buffering if you want to send logs to destinations where disk-based buffering is not available. Or if you want the fastest solution, and if AxoSyslog crash or network downtime is never expected. In these cases, losing logs is possible. This solution does not use disk-based buffering, logs are stored only in the memory.

Example: Example for using memory buffering

   destination d_BSD {
        network("127.0.0.1"
            port(3333)
            log-fifo-size(10000)
        );
    };

5 - About disk queue files

Normal and reliable queue files

The key difference between disk queue files that employ the reliable(yes) option and not is the strategy they employ. Reliable disk queues guarantee that all the messages passing through them are written to disk first, and removed from the queue only after the destination has confirmed that the message has been successfully received. This prevents message loss, for example, due to AxoSyslog crashes. Of course, using the reliable(yes) option introduces a significant performance penalty as well.

Both reliable and normal disk-buffers employ an in-memory output queue (set in quot-size()) and an in-memory overflow queue (set in flow-control-window-bytes() for reliable disk-buffers, or flow-control-window-size() for normal disk-buffers). The difference between reliable and normal disk-buffers is that when the reliable disk-buffer uses one of its in-memory queues, it also stores the message on the disk, whereas the normal disk-buffer stores the message only in memory. The normal disk-buffer only uses the disk if the in-memory output buffer is filled up completely. This approach has better performance (due to fewer disk I/O operations), but also carries the risk of losing a maximum of quot-size() plus flow-control-window-size() number of messages in case of an unexpected power failure or application crash.

Size of the queue files

Disk queue files grow. Each may take up to capacity-bytes() bytes on the disk. Due to the nature of reliable queue files, all the messages traversing the queue are written to disk, constantly increasing the size of the queue file.

The disk-buffer file’s size should be considered as the configured capacity-bytes() at any point of time, even if it does not have messages in it. Truncating the disk-buffer file can slow down disk I/O operations, so AxoSyslog does not always truncate the file when it would be possible (see the truncate-size-ratio() option). If a large disk-buffer file is not desirable, you should set the capacity-bytes() option to a smaller value. Note that AxoSyslog version 4.0 and later doesn’t truncate disk-buffer files by default.

Starting with AxoSyslog version 4.0, you can preallocate disk-buffer files.

Preallocating disk-buffer files

By default, AxoSyslog doesn’t reserve the disk space for the disk-buffer file, since in a properly configured and sized environment the disk-buffer is practically empty, so a large preallocated disk-buffer file is just a waste of disk space. But a preallocated buffer can prevent other data from using the intended buffer space (and elicit a warning from the OS if disk space is low), preventing message loss if the buffer is actually needed. To avoid this problem, when using AxoSyslog 4.0 or later, you can preallocate the space for your disk-buffer files by setting prealloc(yes).

In addition to making sure that the required disk space is available when needed, preallocated disk-buffer files provide radically better (3-4x) performance as well: in case of an outage the amount of messages stored in the disk-buffer is continuously growing, and using large continuous files is faster, than constantly waiting on a file to change its size.

If you are running AxoSyslog on a dedicated host (always recommended for any high-volume settings), use prealloc(yes).