kafka: Publish messages to Apache Kafka (Java implementation)

Starting with version 3.7, AxoSyslog can directly publish log messages to the Apache Kafka message bus, where subscribers can access them.

• Since AxoSyslog uses the official Java Kafka producer, the kafka destination has significant memory usage.

Declaration:

   @include "scl.conf"
    
    kafka(
        client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/:<path-to-preinstalled-kafka-libraries>")
        kafka-bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
        topic("${HOST}")
    
    );

Example: Sending log data to Apache Kafka

The following example defines a kafka destination, using only the required parameters.

   @include "scl.conf"
    
    destination d_kafka {
      kafka(
        client-lib-dir("/opt/syslog-ng/lib/syslog-ng/java-modules/KafkaDestination.jar:/usr/share/kafka/lib/")
        kafka-bootstrap-servers("1.2.3.4:9092,192.168.0.2:9092")
        topic("${HOST}")
      );
    };

• To install the software required for the kafka destination, see Prerequisites.

• For details on how the kafka destination works, see How AxoSyslog interacts with Apache Kafka.

• For the list of options, see Kafka destination options.

The kafka() driver is actually a reusable configuration snippet configured to receive log messages using the Java language-binding of AxoSyslog. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of the kafka configuration snippet on GitHub.