s3: Amazon S3

Available in AxoSyslog version 4.4 and later.

The s3() destination sends log messages to the Amazon Simple Storage Service (Amazon S3) object storage service. Messages are normally sent encrypted with TLS (HTTPS), but you can specify a custom unencrypted HTTP endpoint.

Prerequisites

• An existing S3 bucket configured for programmatic access, and the related ACCESS_KEY and SECRET_KEY of a user that can access it. The user needs to have the following permissions:

  • s3:ListBucket
  • s3:ListBucketMultipartUploads
  • s3:AbortMultipartUpload
  • s3:ListMultipartUploadParts
  • s3:PutObject

  The following kms-related permissions are needed to use the aws:kms encryption. The AWS Role or User must have the following permissions on the given key:

  • kms:Decrypt (For details on why the kms:Decrypt is mandatory, check this AWS Knowledge Center entry.)
  • kms:Encrypt
  • kms:GenerateDataKey

• If you are not using the venv (/usr/bin/syslog-ng-update-virtualenv) created by AxoSyslog, you must install the boto3 and/or botocore Python dependencies.

• To use the s3() driver, the scl.conf file must be included in your AxoSyslog configuration:

  @include "scl.conf"
  

The s3() driver is actually a reusable configuration snippet. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.

Declaration

s3(
    region("us-east-2")
    url("http://localhost:9000")
    bucket("syslog-ng")
    access-key("my-access-key")
    secret-key("my-secret-key")
    object-key("${HOST}/my-logs")
    template("${MESSAGE}\n")
);

Creating objects

AxoSyslog can create a new object based on the following strategies:

• Based on object size: The max-object-size() option configures AxoSyslog to finish an object if it reaches a certain size. AxoSyslog appends an index ("-1", “-2”, …) to the end of the object key, then starts a new object.
• Based on timestamp: The object-key-timestamp() option can be used to set a datetime-related template, which is appended to the end of the object, for example: "${R_MONTH_ABBREV}${R_DAY}". When a log message arrives with a newer timestamp template resolution, the previous timestamped object gets finished and a new one is started with the new timestamp. If an older message arrives, it doesn`t reopen the old object, but starts a new object with the key having an index appended to the old object.
• Based on timeout: The flush-grace-period() option sets the number of minutes to wait for new messages to arrive after the last one. If the timeout expires, AxoSyslog closes the object, and opens a new object (with an appended index) when a new message arrives.

All of these strategies can be used individually, or together.

Upload options

AxoSyslog uploads objects using the multipart upload API. AxoSyslog composes chunks locally. When a chunk reaches the size set in chunk-size() (by default 5 MiB), the chunk is uploaded. When an object is finished, the multipart upload is completed and S3 merges the chunks.

You can influence the upload via the chunk-size(), upload-threads(), and the max-pending-uploads() options.

Options

The following options are specific to the s3() destination.

access-key()

Description: The ACCESS_KEY of the service account used to access the S3 bucket. (Together with secret-key().)

Starting with version 4.7, you can use the AWS_... environment variables or credentials files from the ~/.aws/ directory instead of this option. For details, see the official documentation.

bucket()

Description: The name of the S3 bucket, for example, my-bucket. Note that the bucket must already exist.

canned-acl()

Description: The ACL assigned to the object, if specified, for example, bucket-owner-read. The following values are valid:

authenticated-read, aws-exec-read, bucket-owner-full-control, bucket-owner-read, log-delivery-write, private, public-read, public-read-write

If you configure an invalid value, the default is used.

chunk-size()

Description: The size of log messages that AxoSyslog writes to the S3 object in a batch. If compression is enabled, the chunk-size() refers to the compressed size.

compression()

Description: Setting compression(yes) enables gzip compression, and implicitly adds a .gz suffix to the created object’s key. You can set the level of the compression using the compresslevel() option (0-9).

compresslevel()

Description: Only has effect if compression() is set to yes. You can set the level of the compression using the compresslevel() option (0-9).

flush-grace-period()

Description: After the grace period expires and no new messages are routed to the destination, AxoSyslog flushes the contents of the buffer to the S3 object even if the volume of the messages in the buffer is lower than chunk-size().

log-fifo-size()

Description: The number of messages that the output queue can store.

kms-key()

Available in AxoSyslog 4.8 and later.

Description: The kms-key() used for server-side encryption. The value of the kms-key() parameter must be one of the following:

• The ID of a key.
• An alias of a key. In that case, make sure to add the alias/prefix, for example: kms-key("alias/log-archive")
• The ARN of a key.

max-object-size()

Description: The maximal size of the S3 object. If an object reaches this size, AxoSyslog appends an index ("-1", “-2”, …) to the end of the object key and starts a new object after rotation.

max-pending-uploads()

Description: The max-pending-uploads() and upload-threads() options configure the upload of the chunks. Uploading happens in multiple threads to minimize network overhead.

• upload-threads() limits the maximum number of parallel uploads.
• max-pending-uploads() limits the number of chunks that are waiting in the work queue of the upload threads to get uploaded.

object-key()

Description: The object key (or key name), which uniquely identifies the object in an Amazon S3 bucket. Note that a suffix may be appended to this object key depending on the naming strategies used. Example: my-logs/${HOSTNAME}/.

object-key-timestamp()

Description: The object-key-timestamp() option can be used to set a datetime-related template, which is appended to the end of the object key, for example: "${R_MONTH_ABBREV}${R_DAY}". When a log message arrives with a newer timestamp template resolution, the previous timestamped object gets finished and a new one is started with the new timestamp. If an older message arrives, it doesn`t reopen the old object, but starts a new object with the key having an index appended to the old object.

persist-name()

Description: If you receive the following error message during AxoSyslog startup, set the persist-name() option of the duplicate drivers:

Error checking the uniqueness of the persist names, please override it with persist-name option. Shutting down.

This error happens if you use identical drivers in multiple sources, for example, if you configure two file sources to read from the same file. In this case, set the persist-name() of the drivers to a custom string, for example, persist-name("example-persist-name1").

region()

Description: The AWS region to use when writing the bucket. This should normally be the same region where the bucket is created. This option implies an API endpoint url(). For providers other than AWS, or for custom API endpoints, use the url() option.

secret-key()

Description: The SECRET_KEY of the service account used to access the S3 bucket. (Together with access-key().)

Starting with version 4.7, you can use the AWS_... environment variables or credentials files from the ~/.aws/ directory instead of this option. For details, see the official documentation.

server-side-encryption()

Available in AxoSyslog 4.8 and later.

Description: You can use the server-side-encryption() and kms-key() options to configure encryption. Currently only server-side-encryption("aws:kms") is supported.

destination d_s3 {
  s3(
    bucket("log-archive-bucket")
    object-key("logs/syslog")
    server-side-encryption("aws:kms")
    kms-key("alias/log-archive")
  );

For details on using KMS keys, see the official S3 documentation.

storage-class()

Description: The storage class of the object, for example, REDUCED_REDUNDANCY. The following values are valid:

DEEP_ARCHIVE, GLACIER, GLACIER_IR, INTELLIGENT_TIERING, ONEZONE_IA, OUTPOSTS, REDUCED_REDUNDANCY, SNOW, STANDARD, STANDARD_IA

If you configure an invalid value, the default is used.

upload-threads()

Description: The number of AxoSyslog worker threads that are used to upload data to S3 from this destination.

template()

Description: The message as written to the Amazon S3 object. You can use templates and template functions to format the message.

url()

Description: The API endpoint URL for writing to the S3 bucket, for example https://s3.us-west-2.amazonaws.com, http://minio.local:9000, or https://storage.googleapis.com.