1 - Configuring AxoSyslog on client hosts
Purpose:
To configure AxoSyslog on a client host, complete the following steps.
Steps:
-
Install the AxoSyslog application on the host. For details installing AxoSyslog on specific operating systems, see Install AxoSyslog.
-
Configure the local sources to collect the log messages of the host. Starting with version 3.2, AxoSyslog automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collect the system-specific log messages of a platform.
To configure AxoSyslog, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on the platform you are running AxoSyslog, and how you have installed AxoSyslog it.
- Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the
/etc/syslog-ng/ directory.
- Containers: When running AxoSyslog in a container, typically you map an external file to the
/etc/syslog-ng/syslog-ng.conf file within the container. Check the mapped volumes of your container, and edit the external file.
- Kubernetes: If you’re running AxoSyslog in Kubernetes and have installed it with helm, usually you configure AxoSyslog by editing a
values.yaml file, and redeploying AxoSyslog. Often the syslog-ng.conf part is under the config.raw section in the values.yaml file. For details, see Parameters of the AxoSyslog Helm chart.
Add sources to collect the messages from your log files. File sources look like this:
source s_myfilesource {
file("/var/log/myapplication.log" follow-freq(1));
};
Name every source uniquely. For details on configuring file sources, see file: Collect messages from text files.
Note
Many applications send log messages to logfiles by default (for example, the Roundcube webmail client, or the ProFTPD FTP server), but can be configured to send them to syslog instead. If possible, it is recommended to reconfigure the application that way.
Note
The default configuration file of AxoSyslog collects platform-specific log messages and the internal log messages of AxoSyslog.
source s_local {
system();
internal();
};
-
Create a network destination that points directly to the AxoSyslog server, or to a local relay. The network destination greatly depends on the protocol that your log server or relay accepts messages. Many systems still use the legacy BSD-syslog protocol (RFC3162) over the unreliable UDP transport:
destination d_network { network("10.1.2.3" transport("udp")); };
However, if possible, use the much more reliable IETF-syslog protocol over TCP transport:
destination d_network {
syslog("10.1.2.3" transport("tcp"));
};
-
Create a log statement connecting the local sources to the AxoSyslog server or relay. For example:
log {
source(s_local); destination(d_network);
};
-
If the logs will also be stored locally on the host, create local file destinations.
Note
The default configuration of AxoSyslog places the collected messages into the /var/log/messages file:
destination d_local {
file("/var/log/messages");
};
-
Create a log statement connecting the local sources to the file destination.
Note
The default configuration of AxoSyslog has only one log statement:
log {
source(s_local); destination(d_local);
};
-
Set filters, macros and other features and options (for example, TLS encryption) as necessary.
Example: The default configuration file of AxoSyslog
The following is the default configuration file of AxoSyslog4.10.0. It collects local log messages and the log messages of AxoSyslog and saves them in the /var/log/messages file.
@version: 4.10.0
@include "scl.conf"
source s_local {
system(); internal();
};
destination d_local {
file("/var/log/messages");
};
log {
source(s_local); destination(d_local);
};
Example: A simple configuration for clients
The following is a simple configuration file that collects local log messages and forwards them to a logserver using the IETF-syslog protocol.
@version: 4.10.0
@include "scl.conf"
source s_local {
system(); internal();
};
destination d_syslog_tcp {
syslog("192.168.1.1" transport("tcp") port(2010));
};
log {
source(s_local);destination(d_syslog_tcp);
};
2 - Configuring AxoSyslog on server hosts
Purpose:
To configure AxoSyslog on a server host, complete the following steps.
Steps:
-
Install the AxoSyslog application on the host. For details installing AxoSyslog on specific operating systems, see Install AxoSyslog.
-
Starting with version 3.2, AxoSyslog automatically collects the log messages that use the native system logging method of the platform, for example, messages from /dev/log on Linux, or /dev/klog on FreeBSD. For a complete list of messages that are collected automatically, see system: Collect the system-specific log messages of a platform.
-
To configure AxoSyslog, edit the syslog-ng.conf file with any regular text editor application. The location of the configuration file depends on the platform you are running AxoSyslog, and how you have installed AxoSyslog it.
- Native packages of a platform (like the ones downloaded from Linux repositories) typically place the configuration file under the
/etc/syslog-ng/ directory.
- Containers: When running AxoSyslog in a container, typically you map an external file to the
/etc/syslog-ng/syslog-ng.conf file within the container. Check the mapped volumes of your container, and edit the external file.
- Kubernetes: If you’re running AxoSyslog in Kubernetes and have installed it with helm, usually you configure AxoSyslog by editing a
values.yaml file, and redeploying AxoSyslog. Often the syslog-ng.conf part is under the config.raw section in the values.yaml file. For details, see Parameters of the AxoSyslog Helm chart.
Configure the network sources that collect the log messages sent by the clients and relays. How the network sources should be configured depends also on the capabilities of your client hosts: many older networking devices support only the legacy BSD-syslog protocol (RFC3164) using UDP transport:
source s_network {
syslog(ip(10.1.2.3) transport("udp"));
};
However, if possible, use the much more reliable TCP transport:
source s_network {
syslog(ip(10.1.2.3) transport("tcp"));
};
For other options, see syslog: Collect messages using the IETF-syslog protocol and tcp, tcp6, udp, udp6: OBSOLETE - Collect messages from remote hosts using the BSD syslog protocol.
Note
Starting with AxoSyslog version 3.2, the syslog() source driver can handle both BSD-syslog (RFC 3164) and IETF-syslog (RFC 5424-26) messages.
-
Create local destinations that will store the log messages, for example, file- or program destinations. The default configuration of AxoSyslog places the collected messages into the /var/log/messages file:
destination d_local {
file("/var/log/messages");
};
If you want to create separate logfiles for every client host, use the ${HOST} macro when specifying the filename, for example:
destination d_local {
file("/var/log/messages_${HOST}");
};
For details on further macros and how to use them, see template and rewrite: Format, modify, and manipulate log messages.
-
Create a log statement connecting the sources to the local destinations.
log {
source(s_local); source(s_network); destination(d_local);
};
-
Set filters, options (for example, TLS encryption) and other advanced features as necessary.
Note
By default, the AxoSyslog server treats the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the AxoSyslog server, use the keep-hostname(yes) option both on the AxoSyslog relay and the AxoSyslog server. This option can be set individually for every source if needed.
If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.
Example: A simple configuration for servers
The following is a simple configuration file for AxoSyslog that collects incoming log messages and stores them in a text file.
@version: 4.10.0
@include "scl.conf"
options {
time-reap(30);
mark-freq(10);
keep-hostname(yes);
};
source s_local {
system(); internal();
};
source s_network {
syslog(transport(tcp));
};
destination d_logs {
file(
"/var/log/syslog-ng/logs.txt"
owner("root")
group("root")
perm(0777)
);
};
log {
source(s_local); source(s_network); destination(d_logs);
};
3 - Configuring AxoSyslog relays
This section describes how to configure AxoSyslog as a relay.
3.1 - Configuring AxoSyslog on relay hosts
To configure AxoSyslog on a relay host, complete the following steps:
-
Install the AxoSyslog application on the host. For details on installing AxoSyslog on specific operating systems, see Install AxoSyslog.
-
Configure the network sources that collect the log messages sent by the clients.
-
Create a network destination that points to the AxoSyslog server.
-
Create a log statement connecting the network sources to the AxoSyslog server.
-
Configure the local sources that collect the log messages of the relay host.
-
Create a log statement connecting the local sources to the AxoSyslog server.
-
Enable the keep-hostname() and disable the chain-hostnames() options. (For details on how these options work, see Global options.)
Note
It is recommended to use these options on your AxoSyslog server as well.
-
Set filters and options (for example, TLS encryption) as necessary.
Note
By default, the AxoSyslog server treats the relayed messages as if they were created by the relay host, not the host that originally sent them to the relay. In order to use the original hostname on the AxoSyslog server, use the keep-hostname(yes) option both on the AxoSyslog relay and the AxoSyslog server. This option can be set individually for every source if needed.
If you are relaying log messages and want to resolve IP addresses to hostnames, configure the first relay to do the name resolution.
Example: A simple configuration for relays
The following is a simple configuration file that collects local and incoming log messages and forwards them to a logserver using the IETF-syslog protocol.
@version: 4.10.0
@include "scl.conf"
options {
time-reap(30);
mark-freq(10);
keep-hostname(yes);
chain-hostnames(no);
};
source s_local {
system(); internal();
};
source s_network {
syslog(transport(tcp));
};
destination d_syslog_tcp {
syslog("192.168.1.5" transport("tcp") port(2010));
};
log {
source(s_local); source(s_network);
destination(d_syslog_tcp);
};
3.2 - How relaying log messages works
Depending on your exact needs about relaying log messages, there are many scenarios and AxoSyslog options that influence how the log message will look like on the logserver. Some of the most common cases are summarized in the following example:
Consider the following example: client-host > relay > server, where the IP address of client-host is 192.168.1.2. The client-host device sends a syslog message to relay. Depending on the settings of relay, the following can happen.
-
By default, the keep-hostname() option is disabled, so relay writes the IP address of the sender host (in this case, 192.168.1.2) to the HOST field of the syslog message, discarding any IP address or hostname that was originally in the message.
-
If the keep-hostname() option is enabled on relay, but name resolution is disabled (the use-dns() option is set to no), relay uses the HOST field of the message as-is, which is probably 192.168.1.2.
-
To resolve the 192.168.1.2 IP address to a hostname on relay using a DNS server, use the keep-hostname(no) and use-dns(yes) options. If the DNS server is properly configured and reverse DNS lookup is available for the 192.168.1.2 address, AxoSyslog will rewrite the HOST field of the log message to client-host.
Note
It is also possible to resolve IP addresses locally, without relying on the DNS server. For details on local name resolution, see
Resolving hostnames locally.
-
The above points apply to the AxoSyslog server (server) as well, so if relay is configured properly, use the keep-hostname(yes) option on server to retain the proper HOST field. Setting keep-hostname(no) on server would result in AxoSyslog rewriting the HOST field to the address of the host that sent the message to server, which is relay in this case.
-
If you cannot or do not want to resolve the 192.168.1.2 IP address on relay, but want to store your log messages on server using the IP address of the original host (that is, client-host), you can enable the spoof-source() option on relay. However, spoof-source() works only under the following conditions:
- The
syslog-ng binary has been compiled with the --enable-spoof-source option.
- The log messages are sent using the highly unreliable UDP transport protocol. (Extremely unrecommended.)
4 - Sending Kubernetes logs to OpenSearch
The following tutorial shows you how to install AxoSyslog on Kubernetes, deploy OpenSearch to the same cluster, and send Kubernetes logs to OpenSearch.
Prerequisites
You need:
-
A Kubernetes cluster. We used minikube with docker driver and Helm. We used a Ubuntu 22.04 (amd64) machine, but it should work on any system that can run minikube (2 CPUs, 2GB of free memory, 20GB of free disk space).
The OpenSearch service needs a large mmap count setting, so set it to at least 262144, for example:
sysctl -w vm.max_map_count=262144
-
Helm and kubectl installed.
Generate logs
If you don’t already have an application that generates logs deployed to the Kubernetes cluster, install kube-logging/log-generator to generate sample logs. Complete the following steps.
-
Add the kube-logging repository to Helm.
helm repo add kube-logging https://kube-logging.github.io/helm-charts
Expected output:
"kube-logging" has been added to your repositories
-
Update your repositories.
The output should look like:
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "kube-logging" chart repository
Update Complete. ⎈Happy Helming!⎈
-
Install kube-logging/log-generator.
helm install --generate-name --wait kube-logging/log-generator
The output should look like:
NAME: log-generator-1684694629
LAST DEPLOYED: Sun May 21 20:43:49 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
-
Check that the log-generator is running:
The output should look like:
NAME READY STATUS RESTARTS AGE
log-generator-1681984863-5946c559b9-ftrrn 1/1 Running 0 8s
Set up OpenSearch
-
Install an OpenSearch cluster with Helm:
helm repo add opensearch https://opensearch-project.github.io/helm-charts/
Expected output:
"opensearch" has been added to your repositories
-
Update your repositories.
The output should look like:
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "opensearch" chart repository
Update Complete. ⎈Happy Helming!⎈
-
Install OpenSearch. This step can take a few minutes.
helm install --generate-name --wait opensearch/opensearch
-
Install the OpenSearch dashboards.
helm install --generate-name --wait opensearch/opensearch-dashboards
The output should look like:
NAME: opensearch-dashboards-1684695728
LAST DEPLOYED: Sun May 21 21:02:09 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=opensearch-dashboards,app.kubernetes.io/instance=opensearch-dashboards-1684695728" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
-
Now you should have 5 pods. Check that they exist:
The output should look like:
NAME READY STATUS RESTARTS AGE
log-generator-1681984863-5946c559b9-ftrrn 1/1 Running 0 3m39s
opensearch-cluster-master-0 1/1 Running 0 81s
opensearch-cluster-master-1 1/1 Running 0 81s
opensearch-cluster-master-2 1/1 Running 0 81s
opensearch-dashboards-1681999620-59f64f98f7-bjwwh 1/1 Running 0 44s
-
Forward the 5601 port of the OpenSearch Dashboards service (replace the name of the pod with your pod).
kubectl port-forward <name-of-your-opensearch-dashboards-pod> 8080:5601
The output should look like:
Forwarding from 127.0.0.1:8080 -> 5601
Forwarding from [::1]:8080 -> 5601
-
Log in to the dashboard at http://localhost:8080 with admin/admin. You will soon create an Index Pattern here, but first you have to send some logs from syslog-ng.
Set up axosyslog-collector
-
Add the AxoSyslog Helm repository:
helm repo add axosyslog https://axoflow.github.io/axosyslog-charts
helm repo update
-
Create a YAML file (called axoflow-demo.yaml in the examples) to configure the collector.
config:
sources:
kubernetes:
# Collect kubernetes logs
enabled: true
destinations:
# Send logs to OpenSearch
opensearch:
- address: "opensearch-cluster-master"
index: "test-axoflow-index"
user: "admin"
password: "admin"
tls:
# Do not validate the server's TLS certificate.
peerVerify: false
# Send the syslog fields + the metadata from .k8s.* in JSON format
template: "$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))"
-
Check how the syslog-ng.conf file looks with your custom values:
helm template -f axoflow-demo.yaml -s templates/config.yaml axosyslog/axosyslog-collector
The output should look like:
# Source: axosyslog-collector/templates/config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
helm.sh/chart: axosyslog-collector-0.3.0
app.kubernetes.io/name: axosyslog-collector
app.kubernetes.io/instance: release-name
app.kubernetes.io/version: "4.2.0"
app.kubernetes.io/managed-by: Helm
name: release-name-axosyslog-collector
data:
syslog-ng.conf: |
@version: current
@include "scl.conf"
options {
stats(
level(1)
);
};
log {
source { kubernetes(); };
destination {
elasticsearch-http(
url("https://opensearch-cluster-master:9200/_bulk")
index("test-axoflow-index")
type("")
template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))")
user("admin")
password("admin")
tls(
peer-verify(no)
)
);
};
};
-
Install the axosyslog-collector chart:
helm install --generate-name --wait -f axoflow-demo.yaml axosyslog/axosyslog-collector
The output should look like:
NAME: axosyslog-collector-1682002179
LAST DEPLOYED: Thu Apr 20 16:49:39 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Watch the axosyslog-collector-1682002179 container start.
-
Check your pods:
kubectl get pods --namespace=default -l app=axosyslog-collector-1682002179 -w
kubectl get pods
The output should look like:
NAME READY STATUS RESTARTS AGE
log-generator-1681984863-5946c559b9-ftrrn 1/1 Running 0 13m
opensearch-cluster-master-0 1/1 Running 0 11m
opensearch-cluster-master-1 1/1 Running 0 11m
opensearch-cluster-master-2 1/1 Running 0 11m
opensearch-dashboards-1681999620-59f64f98f7-bjwwh 1/1 Running 0 10m
axosyslog-collector-1682002179-pjlkn 1/1 Running 0 6s
Check the logs in OpenSearch
-
Open OpenSearch dashboard at http://localhost:8080/app/management/opensearch-dashboards/.
-
Create an Index Pattern called test-axoflow-index: http://localhost:8080/app/management/opensearch-dashboards/indexPatterns. At Step 2, set the Time field to @timestamp.
-
Now you can see your logs on the Discover view at http://localhost:8080/app/discover. Opening the detailed view for a log entry shows you the fields sent to OpenSearch.
5 - Managing and checking the syslog-ng service on Linux
This section describes how to start, stop and check the status of AxoSyslog service on Linux.
Starting AxoSyslog
To start AxoSyslog, execute the following command as root. For example:
systemctl start syslog-ng
If the service starts successfully, no output will be displayed.
The following message indicates that AxoSyslog can not start (see Checking AxoSyslog status):
Job for syslog-ng.service failed because the control process exited with error code. See `systemctl status syslog-ng.service` and `journalctl -xe` for details.
Stop AxoSyslog
To stop AxoSyslog
-
Execute the following command as root.
systemctl stop syslog-ng
-
Check the status of AxoSyslog service (see Checking AxoSyslog status).
Restart AxoSyslog
To restart AxoSyslog, execute the following command as root.
systemctl restart syslog-ng
Reload configuration file without restarting AxoSyslog
To reload the configuration file without restarting AxoSyslog, execute the following command as root.
systemctl reload syslog-ng
Check AxoSyslog status
To check the following status-related components, observe the suggestions below.
Check the status of AxoSyslog service
To check the status of AxoSyslog service
-
Execute the following command as root.
systemctl --no-pager status syslog-ng
-
Check the Active: field, which shows the status of AxoSyslog service. The following statuses are possible:
Check the process of AxoSyslog
To check the process of AxoSyslog, execute one of the following commands.
Check the internal logs of AxoSyslog
The internal logs of AxoSyslog contains informal, warning and error messages.
By default, AxoSyslog log messages (generated on the internal() source) are written to /var/log/messages.
Check the internal logs of AxoSyslog for any issue.
Message processing
The AxoSyslog application collects statistics about the number of processed messages on the different sources and destinations.
Note
When using syslog-ng-ctl stats, consider that while the output is generally consistent, there is no explicit ordering behind the command. Consequently, Axoflow does not recommend creating parsers that depend on a fix output order.
If needed, you can sort the output with an external application, for example, | sort.
Central statistics
To check the central statistics, execute the following command to see the number of received and queued (sent) messages by AxoSyslog.
watch "/opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^center"
The output will be updated in every 2 seconds. If the numbers are changing, AxoSyslog is processing the messages. Output example:
Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^center Tue Jun 25 10:33:25 2019
center;;queued;a;processed;112
center;;received;a;processed;28
Source statistics
To check the source statistics, execute the following command to see the number of received messages on the configured sources.
watch "/opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^source"
The output will be updated in every 2 seconds. If the numbers are changing, AxoSyslog is receiving messages on the sources. Output example:
Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^source Tue Jun 25 10:40:50 2019
source;s_null;;a;processed;0
source;s_net;;a;processed;0
source;s_local;;a;processed;90
Destination statistics
To check the source statistics, execute the following command to see the number of received messages on the configured sources.
watch "/opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^source"
The output will be updated in every 2 seconds. If the numbers are changing, AxoSyslog is receiving messages on the sources. Output example:
Every 2.0s: /opt/syslog-ng/sbin/syslog-ng-ctl stats | grep ^destination Tue Jun 25 10:41:02 2019
destination;d_logserver2;;a;processed;90
destination;d_messages;;a;processed;180
destination;d_logserver;;a;processed;90
destination;d_null;;a;processed;0
Note
If you find error messages in the internal logs, messages are not processed by AxoSyslog or you encounter any issue, you have the following options: