AxoSyslog documentation

• 1: Install syslog-ng with AxoSyslog
• 1.1: Install AxoSyslog with Docker
• 1.2: Install AxoSyslog with Podman
• 1.3: Install AxoSyslog with Helm
• 1.3.1: Parameters of the AxoSyslog collector Helm chart
• 2: Getting started
• 2.1: Sending Kubernetes logs to OpenSearch
• 3: Support
• 4: Legal notice

1 - Install syslog-ng with AxoSyslog

1.1 - Install AxoSyslog with Docker

AxoSyslog provides cloud-ready syslog-ng images. These images differ from the upstream syslog-ng images, because:

• They’re based on Alpine Linux, instead of Debian testing for reliability and smaller size (thus smaller attack surface).
• They incorporate cloud-native features and settings, such as the Kubernetes source.
• They incorporate container-level optimizations for better performance and improved security. For example, they use an alternative malloc library.
• They support the ARM architecture.

The AxoSyslog images support the following architectures:

• amd64
• arm/v7
• arm64

Install the AxoSyslog images

You can find the list of tagged versions at https://github.com/axoflow/axosyslog-docker/pkgs/container/axosyslog.

To install the latest stable version, run:

docker pull ghcr.io/axoflow/axosyslog:latest

You can also use it as a base image in your Dockerfile:

FROM ghcr.io/axoflow/axosyslog:latest

If you want to test a development version, you can use the nightly builds:

docker pull ghcr.io/axoflow/axosyslog:nightly

Note: These named packages are automatically updated when a new package is released. To install a specific version, run docker pull ghcr.io/axoflow/axosyslog:<version-number>, for example:

docker pull ghcr.io/axoflow/axosyslog:4.2.0

Customize the configuration

The AxoSyslog container image stores the configuration file at /etc/syslog-ng/syslog-ng.conf. By default, AxoSyslog collects the local system logs and logs received from the network into the /var/log/messages and /var/log/messages-kv.log files using this configuration file from the syslog-ng repository.

To customize the configuration, create your own configuration file and override the file in the container image with it, for example:

docker run --rm --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest

Contribute

If you have fixed a bug or would like to contribute your improvements to these images, open a pull request. We truly appreciate your help.

1.2 - Install AxoSyslog with Podman

AxoSyslog provides cloud-ready syslog-ng images. These images differ from the upstream syslog-ng images, because:

• They’re based on Alpine Linux, instead of Debian testing for reliability and smaller size (thus smaller attack surface).
• They incorporate cloud-native features and settings, such as the Kubernetes source.
• They incorporate container-level optimizations for better performance and improved security. For example, they use an alternative malloc library.
• They support the ARM architecture.

The AxoSyslog images support the following architectures:

• amd64
• arm/v7
• arm64

Install the AxoSyslog images

You can find the list of tagged versions at https://github.com/axoflow/axosyslog-docker/pkgs/container/axosyslog.

To install the latest stable version, run:

podman pull ghcr.io/axoflow/axosyslog:latest

You can also use it as a base image in your Dockerfile:

FROM ghcr.io/axoflow/axosyslog:latest

If you want to test a development version, you can use the nightly builds:

podman pull ghcr.io/axoflow/axosyslog:nightly

Note: These named packages are automatically updated when a new package is released. To install a specific version, run podman pull ghcr.io/axoflow/axosyslog:<version-number>, for example:

podman pull ghcr.io/axoflow/axosyslog:4.2.0

Customize the configuration

The AxoSyslog container image stores the configuration file at /etc/syslog-ng/syslog-ng.conf. By default, AxoSyslog collects the local system logs and logs received from the network into the /var/log/messages and /var/log/messages-kv.log files using this configuration file from the syslog-ng repository.

To customize the configuration, create your own configuration file and override the file in the container image with it, for example:

podman run --rm --volume <path-to-your/syslog-ng.conf>:/etc/syslog-ng/syslog-ng.conf ghcr.io/axoflow/axosyslog:latest

Contribute

If you have fixed a bug or would like to contribute your improvements to these images, open a pull request. We truly appreciate your help.

1.3 - Install AxoSyslog with Helm

AxoSyslog provides Helm charts for syslog-ng. You can use these charts to install cloud-ready syslog-ng images created and maintained by Axoflow.

Prerequisites

You must have Helm 3.0 or newer installed to use these charts. Refer to the official Helm documentation for details.

Limitations

The chart provides parameters that make it easy to:

• collect logs using the kubernetes() source, and
• forward the logs using the network() and opensearch() destinations.

To use other sources and destinations, use the config.raw parameter. For the list of configurable parameters and their default values, see Parameters of the AxoSyslog collector Helm chart.

Install

To install the axosyslog-collector charts, complete the following steps.

1. Clone the chart repository.

   helm repo add axosyslog https://axoflow.github.io/axosyslog-charts
   helm repo update
   

2. Install the chart. The following command installs axosyslog-collector into the default namespace. For the list of configurable parameters and their default values, see Parameters of the AxoSyslog collector Helm chart.

   helm install --generate-name axosyslog/axosyslog-collector
   

   NAME: axosyslog-collector-1683469360
   LAST DEPLOYED: Sun May  7 16:22:40 2023
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   NOTES:
   1. Watch the axosyslog-collector-1683469360 container start.
     $ kubectl get pods --namespace=default -l app=axosyslog-collector-1683469360 -w
   

3. Check that the pod is running.

   kubectl get pods
   

   The output should look like:

   NAME                                   READY   STATUS    RESTARTS   AGE
   axosyslog-collector-1683469360-tptfb   1/1     Running   0          28s
   

Uninstall

Tip: List all installed releases using helm list.

To uninstall a chart release, run:

helm delete <name-of-the-release-to-delete>

Contribute

If you have fixed a bug or would like to contribute your improvements to these charts, open a pull request. We truly appreciate your help.

1.3.1 - Parameters of the AxoSyslog collector Helm chart

The following table lists the configurable parameters of the AxoSyslog collector chart and their default values. For details on installing the chart, see Install AxoSyslog with Helm.

Parameters for syslog-ng configuration

The following example uses the config.raw parameter to configure a custom destination:

config:
  raw: |
    @version: 4.3
    @include "scl.conf"

    log {
      source {
        syslog(port(12345));
      };

      destination {
        logscale(
          token("your-secret-humio-ingest-token")
        );
      };

      flags(flow-control);
    };

daemonset:
  hostNetworking: true

Network destination

Send logs over the network, conforming to RFC3164 using the network() destination.

For example:

config:
  destinations:
    network:
      - transport: tcp
        address: localhost
        port: 12345
        template: "$(format-json .*)"

OpenSearch destination

Send logs to OpenSearch over HTTP or HTTPS.

For example:

config:
  destinations:
    opensearch:
      - address: 10.104.232.94
        index: "test-axoflow-index"
        tls:
          CAFile: "/path/to/CAFile.pem"
          CADir: "/path/to/CADir/"
          Cert: "/path/to/Cert.pem"
          Key: "/path/to/Key.pem"
          peerVerify: true
          template: "$(format-json .*)"

Generic parameters

2 - Getting started

2.1 - Sending Kubernetes logs to OpenSearch

The following tutorial shows you how to send Kubernetes logs to OpenSearch.

Prerequisites

You need a Kubernetes cluster. We used minikube with docker driver and Helm. We used a Ubuntu 22.04 (amd64) machine, but it should work on any system that can run minikube (2 CPUs, 2GB of free memory, 20GB of free disk space).

The OpenSearch service needs a large mmap count setting, so set it to at least 262144, for example:

sysctl -w vm.max_map_count=262144

You must have Helm and kubectl installed.

Generate logs

Install kube-logging/log-generator to generate logs. Complete the following steps.

1. Add the kube-logging repository to Helm.

   helm repo add kube-logging https://kube-logging.github.io/helm-charts
   

   Expected output:

   "kube-logging" has been added to your repositories
   

2. Update your repositories.

   helm repo update
   

   The output should look like:

   Hang tight while we grab the latest from your chart repositories...
   ...Successfully got an update from the "kube-logging" chart repository
   Update Complete. ⎈Happy Helming!⎈
   

3. Install kube-logging/log-generator.

   helm install --generate-name --wait kube-logging/log-generator
   

   The output should look like:

   NAME: log-generator-1684694629
   LAST DEPLOYED: Sun May 21 20:43:49 2023
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   

4. Check that the log-generator is running:

   kubectl get pods
   

   The output should look like:

   NAME                                        READY   STATUS        RESTARTS       AGE
   log-generator-1681984863-5946c559b9-ftrrn   1/1     Running       0              8s
   

Set up OpenSearch

1. Install an OpenSearch cluster with Helm:

   helm repo add opensearch https://opensearch-project.github.io/helm-charts/
   

   Expected output:

   "opensearch" has been added to your repositories
   

2. Update your repositories.

   helm repo update
   

   The output should look like:

   Hang tight while we grab the latest from your chart repositories...
   ...Successfully got an update from the "opensearch" chart repository
   Update Complete. ⎈Happy Helming!⎈
   

3. Install OpenSearch. This step can take a few minutes.

   helm install --generate-name --wait opensearch/opensearch
   

4. Install the OpenSearch dashboards.

   helm install --generate-name --wait opensearch/opensearch-dashboards
   

   The output should look like:

   NAME: opensearch-dashboards-1684695728
   LAST DEPLOYED: Sun May 21 21:02:09 2023
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   NOTES:
   1. Get the application URL by running these commands:
     export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=opensearch-dashboards,app.kubernetes.io/instance=opensearch-dashboards-1684695728" -o jsonpath="{.items[0].metadata.name}")
     export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
     echo "Visit http://127.0.0.1:8080 to use your application"
     kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
   

5. Now you should have 5 pods. Check that they exist:

   kubectl get pods
   

   The output should look like:

   NAME                                                READY   STATUS    RESTARTS   AGE
   log-generator-1681984863-5946c559b9-ftrrn           1/1     Running   0          3m39s
   opensearch-cluster-master-0                         1/1     Running   0          81s
   opensearch-cluster-master-1                         1/1     Running   0          81s
   opensearch-cluster-master-2                         1/1     Running   0          81s
   opensearch-dashboards-1681999620-59f64f98f7-bjwwh   1/1     Running   0          44s
   

6. Forward the 5601 port of the OpenSearch Dashboards service (replace the name of the pod with your pod).

   kubectl port-forward opensearch-dashboards-1681999620-59f64f98f7-bjwwh 8080:5601
   

   The output should look like:

   Forwarding from 127.0.0.1:8080 -> 5601
   Forwarding from [::1]:8080 -> 5601
   

7. Log in to the dashboard at http://localhost:8080 with admin/admin. You will soon create an Index Pattern here, but first you have to send some logs from syslog-ng.

Set up axosyslog-collector

1. Add the AxoSyslog Helm repository:

   helm repo add axosyslog https://axoflow.github.io/axosyslog-charts
   helm repo update
   

2. Create a YAML file (called axoflow-demo.yaml in the examples) to configure the collector.

   config:
     sources:
       kubernetes:
         # Collect kubernetes logs
         enabled: true
     destinations:
       # Send logs to OpenSearch
       opensearch:
         - address: "opensearch-cluster-master"
           index: "test-axoflow-index"
           user: "admin"
           password: "admin"
           tls:
             # Do not validate the server's TLS certificate.
             peerVerify: false
           # Send the syslog fields + the metadata from .k8s.* in JSON format
           template: "$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))"
   

3. Check how the syslog-ng.conf file looks with your custom values:

   helm template -f axoflow-demo.yaml -s templates/config.yaml axosyslog/axosyslog-collector
   

   The output should look like:

   # Source: axosyslog-collector/templates/config.yaml
   apiVersion: v1
   kind: ConfigMap
   metadata:
     labels:
       helm.sh/chart: axosyslog-collector-0.3.0
       app.kubernetes.io/name: axosyslog-collector
       app.kubernetes.io/instance: release-name
       app.kubernetes.io/version: "4.2.0"
       app.kubernetes.io/managed-by: Helm
     name: release-name-axosyslog-collector
   data:
     syslog-ng.conf: |
       @version: current
       @include "scl.conf"
   
       options {
         stats(
           level(1)
         );
       };
   
       log {
         source { kubernetes(); };
         destination {
           elasticsearch-http(
             url("https://opensearch-cluster-master:9200/_bulk")
             index("test-axoflow-index")
             type("")
             template("$(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE} k8s=$(format-json .k8s.* --shift-levels 2 --exclude .k8s.log))")
             user("admin")
             password("admin")
             tls(
               peer-verify(no)
             )
           );
         };
       };    
   

4. Install the axosyslog-collector chart:

   helm install --generate-name --wait -f axoflow-demo.yaml axosyslog/axosyslog-collector
   

   The output should look like:

   NAME: axosyslog-collector-1682002179
   LAST DEPLOYED: Thu Apr 20 16:49:39 2023
   NAMESPACE: default
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   NOTES:
   1. Watch the axosyslog-collector-1682002179 container start.
   

5. Check your pods:

   kubectl get pods --namespace=default -l app=axosyslog-collector-1682002179 -w
   kubectl get pods
   

   The output should look like:

   NAME                                                READY   STATUS    RESTARTS   AGE
   log-generator-1681984863-5946c559b9-ftrrn           1/1     Running   0          13m
   opensearch-cluster-master-0                         1/1     Running   0          11m
   opensearch-cluster-master-1                         1/1     Running   0          11m
   opensearch-cluster-master-2                         1/1     Running   0          11m
   opensearch-dashboards-1681999620-59f64f98f7-bjwwh   1/1     Running   0          10m
   axosyslog-collector-1682002179-pjlkn                1/1     Running   0          6s
   

Check the logs in OpenSearch

1. Open OpenSearch dashboard at http://localhost:8080/app/management/opensearch-dashboards/.

2. Create an Index Pattern called test-axoflow-index: http://localhost:8080/app/management/opensearch-dashboards/indexPatterns. At Step 2, set the Time field to @timestamp.

   

3. Now you can see your logs on the Discover view at http://localhost:8080/app/discover. Opening the detailed view for a log entry shows you the fields sent to OpenSearch.

   

3 - Support

In case you need help with any of the AxoSyslog projects, or directly with syslog-ng, you have several ways to contact us:

• Open a GitHub issue in the relevant repository.
• Chat with us in the syslog-ng channel of the Axoflow Discord server.
• Fill the Axoflow contact form.

We also provide consulting and professional services for logging and observability related projects. Contact us if you need our help!

4 - Legal notice

Copyright © 2023 Axoflow Zrt.

This documentation is licensed under the Apache License 2.0 (https://www.apache.org/licenses/LICENSE-2.0).

All logos and mentioned trademarks are the property of their respective owners.