Kerberos authentication with the hdfs() destination
Version 3.10 and later supports Kerberos authentication to authenticate the connection to your Hadoop cluster. AxoSyslog assumes that you already have a Hadoop and Kerberos infrastructure.
If you configure Kerberos authentication for a hdfs() destination, it affects all hdfs() destinations. Kerberos and non-Kerberos hdfs() destinations cannot be mixed in a AxoSyslog configuration. This means that if one hdfs() destination uses Kerberos authentication, you have to configure all other hdfs() destinations to use Kerberos authentication too.
Failing to do so results in non-Kerberos hdfs() destinations being unable to authenticate to the HDFS server.
hdfs() destination to stop using Kerberos authentication, namely, to remove Kerberos-related options from the hdfs() destination configuration, make sure to restart AxoSyslog for the changes to take effect.
Prerequisites:
-
You have configured your Hadoop infrastructure to use Kerberos authentication.
-
You have a keytab file and a principal for the host running AxoSyslog. For details, see the Kerberos documentation.
-
You have installed and configured the Kerberos client packages on the host running AxoSyslog. (That is, Kerberos authentication works for the host, for example, from the command line using the
kinit user@REALM -k -t <keytab_file>command.)
destination d_hdfs {
hdfs(client-lib-dir("/hdfs-libs/lib")
hdfs-uri("hdfs://hdp-kerberos.syslog-ng.example:8020")
kerberos-keytab-file("/opt/syslog-ng/etc/hdfs.headless.keytab")
kerberos-principal("hdfs-hdpkerberos@MYREALM")
hdfs-file("/var/hdfs/test.log"));
};