template and rewrite: Format, modify, and manipulate log messages

• 1: Customize message format using macros and templates
• 1.1: Formatting messages, filenames, directories, and tablenames
• 1.2: Templates and macros
• 1.3: Date-related macros
• 1.4: Hard versus soft macros
• 1.5: Macros of AxoSyslog
• 1.5.1: Example use case: using the $DESTIP, the $DESTPORT, and the $PROTO macros
• 1.6: Using template functions
• 1.7: Template functions of AxoSyslog
• 1.8: Modifying the on-the-wire message format
• 2: Modifying messages using rewrite rules
• 2.1: Replacing message parts
• 2.2: Setting message fields to specific values
• 2.3: Setting severity with the set-severity() rewrite function
• 2.4: Setting the facility field with the set-facility() rewrite function
• 2.5: Setting the priority of a message with the set-pri() rewrite function
• 2.6: Setting match variables with the set-matches() rewrite rule
• 2.7: Unsetting message fields
• 2.8: Renaming message fields
• 2.9: Creating custom SDATA fields
• 2.10: Setting multiple message fields to specific values
• 2.11: map-value-pairs: Rename value-pairs to normalize logs
• 2.12: Conditional rewrites
• 2.13: Adding and deleting tags
• 2.14: Rewrite the timezone of a message
• 2.15: Anonymizing credit card numbers
• 3: Regular expressions
• 3.1: Options of regular expressions
• 3.1.1: The type() options of regular expressions
• 3.1.2: The flags() options of regular expressions
• 3.1.2.1: Perl Compatible Regular Expressions (PCRE)
• 3.1.2.2: Literal string searches
• 3.1.2.3: Glob patterns without regular expression support
• 3.2: Optimizing regular expressions

1 - Customize message format using macros and templates

The following sections describe how to customize the names of logfiles, and also how to use templates, macros, and template functions.

• Formatting messages, filenames, directories, and tablenames explains how macros work.

• Modifying messages using rewrite rules describes how to use macros and templates to format log messages or change the names of logfiles and database tables.

• Macros of AxoSyslog lists the different types of macros available in AxoSyslog.

• Using template functions explains what template functions are and how to use them.

• Template functions of AxoSyslog lists the template functions available in AxoSyslog.

1.1 - Formatting messages, filenames, directories, and tablenames

The AxoSyslog application can dynamically create filenames, directories, or names of database tables using macros that help you organize your log messages. Macros refer to a property or a part of the log message, for example, the ${HOST} macro refers to the name or IP address of the client that sent the log message, while ${DAY} is the day of the month when the message was received. Using these macros in the path of the destination log files allows you for example, to collect the logs of every host into separate files for every day.

A set of macros can be defined as a template object and used in multiple destinations.

Another use of macros and templates is to customize the format of the syslog message, for example, to add elements of the message header to the message text.

• For details on using templates and macros, see Templates and macros.

• For a list and description of the macros available in AxoSyslog, see Macros of AxoSyslog.

• For details on using custom macros created with CSV parsers and pattern databases, see parser: Parse and segment structured messages and Using parser results in filters and templates, respectively.

1.2 - Templates and macros

The AxoSyslog application allows you to define message templates, and reference them from every object that can use a template. Templates can include strings, macros (for example, date, the hostname, and so on), and template functions. For example, you can use templates to create standard message formats or filenames. For a list of macros available in AxoSyslog, see Macros of AxoSyslog. Fields from the structured data (SD) part of messages using the new IETF-syslog standard can also be used as macros.

Declaration:

   template <template-name> {
        template("<template-expression>") <template-escape(yes)>;
    };

Template objects have a single option called template-escape(), which is disabled by default (template-escape(no)). This behavior is useful when the messages are passed to an application that cannot handle escaped characters properly. Enabling template escaping (template-escape(yes)) causes AxoSyslog to escape the ', ", and backslash characters from the messages.

Note: Starting with AxoSyslog version 4.5, template-escape(yes) escapes the top-level template function in case of nested template functions.

If you do not want to enable the template-escape() option (which is rarely needed), you can define the template without the enclosing braces.

   template <template-name> "<template-expression>";

You can also refer to an existing template from within a template. The result of the referred template will be pasted into the second template.

   template first-template "sample-text";
    template second-template "The result of the first-template is: $(template first-template)";

If you want to use a template only once, you can define the template inline, for example:

   destination d_file {
        file ("/var/log/messages" template("${ISODATE} ${HOST} ${MESSAGE}\n") );
    };

Macros can be included by prefixing the macro name with a $ sign, just like in Bourne compatible shells. Although using braces around macro names is not mandatory, and the "$MESSAGE" and "${MESSAGE}" formats are equivalent, using the "${MESSAGE}" format is recommended for clarity.

Macro names are case-sensitive, that is, "$message" and "$MESSAGE" are not the same.

To use a literal $ character in a template, you have to escape it. In AxoSyslog versions 3.4 and earlier, use a backslash \$. In version 3.5 and later, use $$.

Default values for macros can also be specified by appending the :- characters and the default value of the macro. If a message does not contain the field referred to by the macro, or it is empty, the default value will be used when expanding the macro. For example, if a message does not contain a hostname, the following macro can specify a default hostname.

${HOST:-default_hostname}

By default, AxoSyslog sends messages using the following template: ${ISODATE} ${HOST} ${MSGHDR}${MESSAGE}\\n. (The ${MSGHDR}${MESSAGE} part is written together because the ${MSGHDR} macro includes a trailing whitespace.)

Example: Using templates and macros

The following template (t_demo_filetemplate) adds the date of the message and the name of the host sending the message to the beginning of the message text. The template is then used in a file destination: messages sent to this destination (d_file) will use the message format defined in the template.

   template t_demo_filetemplate {
        template("${ISODATE} ${HOST} ${MESSAGE}\n");
    };
    destination d_file {
        file("/var/log/messages" template(t_demo_filetemplate));
    };

If you do not want to enable the template-escape() option (which is rarely needed), you can define the template without the enclosing braces. The following two templates are equivalent.

   template t_demo_template-with-braces {
        template("${ISODATE} ${HOST} ${MESSAGE}\n");
    };
    template t_demo_template-without-braces "${ISODATE} ${HOST} ${MESSAGE}\n";

Templates can also be used inline, if they are used only at a single location. The following destination is equivalent with the previous example:

   destination d_file {
        file ("/var/log/messages" template("${ISODATE} ${HOST} ${MESSAGE}\n") );
    };

The following file destination uses macros to daily create separate logfiles for every client host.

   destination d_file {
            file("/var/log/${YEAR}.${MONTH}.${DAY}/${HOST}.log");
    };

1.3 - Date-related macros

The macros related to the date of the message (for example: ${ISODATE}, ${HOUR}, and so on) have three further variants each:

• S_ prefix, for example, ${S_DATE}: The ${S_DATE} macro represents the date found in the log message, that is, when the message was sent by the original application.

  Warning To use the S_ macros, the keep-timestamp() option must be enabled (this is the default behavior of AxoSyslog).

• R_ prefix, for example, ${R_DATE}: ${R_DATE} is the date when AxoSyslog has received the message.

• C_ prefix, for example, ${C_DATE}: ${C_DATE} is the current date, that is when AxoSyslog processes the message and resolves the macro.

The ${DATE} macro equals the ${S_DATE} macro.

The values of the date-related macros are calculated using the original timezone information of the message. To convert it to a different timezone, use the time-zone() option. You can set the time-zone() option as a global option, or per destination. For sources, it applies only if the original message does not contain timezone information. Alternatively, you can modify the timezone of the message using timezone-specific rewrite rules. For details, see Rewrite the timezone of a message.

Converting the timezone changes the values of the following date-related macros (macros MSEC and USEC are not changed):

• AMPM

• DATE

• DAY

• FULLDATE

• HOUR

• HOUR12

• ISODATE

• ISOWEEK

• MIN

• MONTH

• MONTH_ABBREV

• MONTH_NAME

• MONTH_WEEK

• SEC

• STAMP

• TZ

• TZOFFSET

• UNIXTIME

• WEEK

• WEEK_DAY

• WEEK_DAY_ABBREV

• WEEK_DAY_NAME

• YEAR

• YEAR_DAY

1.4 - Hard versus soft macros

Hard macros contain data that is directly derived from the log message, for example, the ${MONTH} macro derives its value from the timestamp. Hard macros are read-only. Soft macros (sometimes also called name-value pairs) are either built-in macros automatically generated from the log message (for example, ${HOST}), or custom user-created macros generated by using the pattern database or a CSV-parser. In contrast to hard macros, soft macros are writable and can be modified within AxoSyslog, for example, using rewrite rules.

Hard and soft macros are rather similar and often treated as equivalent. Macros are most commonly used in filters and templates, which does not modify the value of the macro, so both soft and hard macros can be used. However, it is not possible to change the values of hard macros in rewrite rules or via any other means.

The following macros in AxoSyslog are hard macros and cannot be modified: BSDTAG, CONTEXT_ID, DATE, DAY, FACILITY_NUM, FACILITY, FULLDATE, HOUR, IP-PROTO, ISODATE, ISOWEEK, LEVEL_NUM, LEVEL, MIN, MONTH_ABBREV, MONTH_NAME, MONTH, MONTH_WEEK, PRIORITY, PRI, RCPTID, SDATA, SEC, SEQNUM, SOURCEIP, STAMP, TAG, TAGS, TRANSPORT, TZOFFSET, TZ, UNIXTIME, WEEK_DAY_ABBREV, WEEK_DAY_NAME, WEEK_DAY, WEEK, YEAR_DAY, YEAR.

The following macros can be modified:FULLHOST_FROM, FULLHOST, HOST_FROM, HOST, LEGACY_MSGHDR, MESSAGE, MSG,MSGID, MSGONLY, PID, PROGRAM, SOURCE. Custom values created using rewrite rules or parsers can be modified as well, just like stored matches of regular expressions ($0 … $255).

Note that you can modify the timezone of the message, and change the timezone-related macros that way. For details, see Rewrite the timezone of a message.

1.5 - Macros of AxoSyslog

The following macros are available in AxoSyslog templates.

AMPM

Description: Typically used together with the ${HOUR12} macro, ${AMPM} returns the period of the day: AM for hours before mid day and PM for hours after mid day. In reference to a 24-hour clock format, AM is between 00:00-12:00 and PM is between 12:00-24:00. 12AM is midnight. Available in AxoSyslog 3.4 and later.

BSDTAG

Description: Facility/priority information in the format used by the FreeBSD syslogd: a priority number followed by a letter that indicates the facility. The priority number can range from 0 to 7. The facility letter can range from A to Y, where A corresponds to facility number zero (LOG_KERN), B corresponds to facility 1 (LOG_USER), and so on.

Custom macros

Description: CSV parsers and pattern databases can also define macros from the content of the messages, for example, a pattern database rule can extract the username from a login message and create a macro that references the username. For details on using custom macros created with CSV parsers and pattern databases, see parser: Parse and segment structured messages and Using parser results in filters and templates, respectively.

DATE, C_DATE, R_DATE, S_DATE

Description: Date of the message using the BSD-syslog style timestamp format (month/day/hour/minute/second, each expressed in two digits). This is the original syslog time stamp without year information, for example: Jun 13 15:58:00.

DAY, C_DAY, R_DAY, S_DAY

Description: The day the message was sent.

DESTIP

Description: When used, the output specifies the local IP address of the source from which the message originates.

For an example use case when using the macro is recommended, see Example use case: using the $DESTIP, the $DESTPORT, and the $PROTO macros.

DESTPORT

Description: When used, the output specifies the local port of the source from which the message originates.

For an example use case when using the macro is recommended, see Example use case: using the $DESTIP, the $DESTPORT, and the $PROTO macros.

FACILITY

Description: The name of the facility (for example, kern) that sent the message.

FACILITY_NUM

Description: The numerical code of the facility (for example, 0) that sent the message.

FILE_NAME

Description: Name of the log file (including its path) from where AxoSyslog received the message (only available if AxoSyslog received the message from a file or a wildcard-file source). If you need only the path or the filename, use the dirname and basename template functions.

FULLDATE, C_FULLDATE, R_FULLDATE, S_FULLDATE

Description: A nonstandard format for the date of the message using the same format as ${DATE}, but including the year as well, for example: 2006 Jun 13 15:58:00.

FULLHOST

Description: The name of the source host where the message originates from.

• If the message traverses several hosts and the chain-hostnames() option is on, the first host in the chain is used.

• If the keep-hostname() option is disabled (keep-hostname(no)), the value of the $FULLHOST macro will be the DNS hostname of the host that sent the message to AxoSyslog (that is, the DNS hostname of the last hop). In this case the $FULLHOST and $FULLHOST_FROM macros will have the same value.

• If the keep-hostname() option is enabled (keep-hostname(yes)), the value of the $FULLHOST macro will be the hostname retrieved from the log message. That way the name of the original sender host can be used, even if there are log relays between the sender and the server.

  Note The use-dns(), use-fqdn(), normalize-hostnames(), and dns-cache() options will have no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.

For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.

FULLHOST_FROM

Description: The FQDN of the host that sent the message to AxoSyslog as resolved by AxoSyslog using DNS. If the message traverses several hosts, this is the last host in the chain.

The AxoSyslog application uses the following procedure to determine the value of the $FULLHOST_FROM macro:

1. The AxoSyslog application takes the IP address of the host sending the message.

2. If the use-dns() option is enabled, AxoSyslog attempts to resolve the IP address to a hostname. If it succeeds, the returned hostname will be the value of the $FULLHOST_FROM macro. This value will be the FQDN of the host if the use-fqdn() option is enabled, but only the hostname if use-fqdn() is disabled.

3. If the use-dns() option is disabled, or the address resolution fails, the ${FULLHOST_FROM} macro will return the IP address of the sender host.

For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.

HOUR, C_HOUR, R_HOUR, S_HOUR

Description: The hour of day the message was sent.

HOUR12, C_HOUR12, R_HOUR12, S_HOUR12

Description: The hour of day the message was sent in 12-hour clock format. See also the ${AMPM} macro. 12AM is midnight. Available in AxoSyslog 3.4 and later.

HOST

Description: The name of the source host where the message originates from.

• If the message traverses several hosts and the chain-hostnames() option is on, the first host in the chain is used.

• If the keep-hostname() option is disabled (keep-hostname(no)), the value of the $HOST macro will be the DNS hostname of the host that sent the message to AxoSyslog (that is, the DNS hostname of the last hop). In this case the $HOST and $HOST_FROM macros will have the same value.

• If the keep-hostname() option is enabled (keep-hostname(yes)), the value of the $HOST macro will be the hostname retrieved from the log message. That way the name of the original sender host can be used, even if there are log relays between the sender and the server.

  Note The use-dns(), use-fqdn(), normalize-hostnames(), and dns-cache() options will have no effect if the keep-hostname() option is enabled (keep-hostname(yes)) and the message contains a hostname.

For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.

HOST_FROM

Description: The FQDN of the host that sent the message to AxoSyslog as resolved by AxoSyslog using DNS. If the message traverses several hosts, this is the last host in the chain.

The AxoSyslog application uses the following procedure to determine the value of the $HOST_FROM macro:

1. The AxoSyslog application takes the IP address of the host sending the message.

2. If the use-dns() option is enabled, AxoSyslog attempts to resolve the IP address to a hostname. If it succeeds, the returned hostname will be the value of the $HOST_FROM macro. This value will be the FQDN of the host if the use-fqdn() option is enabled, but only the hostname if use-fqdn() is disabled.

3. If the use-dns() option is disabled, or the address resolution fails, the ${HOST_FROM} macro will return the IP address of the sender host.

For details on using name resolution in AxoSyslog, see Using name resolution in syslog-ng.

IP-PROTO

Available in AxoSyslog version 4.5 and later.

The IP protocol version used to retrieve or receive the message. Contains either “4” to indicate IPv4 and “6” to indicate IPv6.

ISODATE, C_ISODATE, R_ISODATE, S_ISODATE

Description: Date of the message in the ISO 8601 compatible standard timestamp format (yyyy-mm-ddThh:mm:ss+-ZONE), for example: 2006-06-13T15:58:00.123+01:00. If possible, it is recommended to use ${ISODATE} for timestamping. Note that AxoSyslog can produce fractions of a second (for example, milliseconds) in the timestamp by using the frac-digits() global or per-destination option.

ISOWEEK, C_ISOWEEK, R_ISOWEEK, S_ISOWEEK

Description: The number of week according to the ISO 8601 standard. Note that the ${WEEK} macro that has been available in returns a non-standard week number that can differ from the value returned by the ${ISOWEEK} macro.

Available in 3.24 and later.

LEVEL_NUM

Description: The priority (also called severity) of the message, represented as a numeric value, for example, 3. For the textual representation of this value, use the ${LEVEL} macro. See PRIORITY or LEVEL for details.

LOGHOST

Description: The hostname of the computer running AxoSyslog.

• In version 3.24 and later: the ${LOGHOST} macro returns the fully-qualified domain name (FQDN) only if the use-fqdn() option is set to yes, and the hostname otherwise.

• In earlier versions: the ${LOGHOST} macro returns the fully-qualified domain name (FQDN).

MESSAGE

Description: Text contents of the log message without the program name and pid. The program name and the pid together are available in the ${MSGHDR} macro, and separately in the ${PROGRAM} and ${PID} macros.

If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.

The ${MSG} macro is an alias of the ${MESSAGE} macro: using ${MSG} in AxoSyslog is equivalent to ${MESSAGE}.

Note that before AxoSyslog version 3.0, the ${MESSAGE} macro included the program name and the pid. In AxoSyslog 3.0, the ${MESSAGE} macro became equivalent with the ${MSGONLY} macro.

MIN, C_MIN, R_MIN, S_MIN

Description: The minute the message was sent.

MONTH, C_MONTH, R_MONTH, S_MONTH

Description: The month the message was sent as a decimal value, prefixed with a zero if smaller than 10.

MONTH_ABBREV, C_MONTH_ABBREV, R_MONTH_ABBREV, S_MONTH_ABBREV

Description: The English abbreviation of the month name (3 letters).

MONTH_NAME, C_MONTH_NAME, R_MONTH_NAME, S_MONTH_NAME

Description: The English name of the month name.

MONTH_WEEK, C_MONTH_WEEK, R_MONTH_WEEK, S_MONTH_WEEK

Description: The number of the week in the given month (0-5). The week with numerical value 1 is the first week containing a Monday. The days of month before the first Monday are considered week 0. For example, if a 31-day month begins on a Sunday, then the 1st of the month is week 0, and the end of the month (the 30th and 31st) is week 5.

MSEC, C_MSEC, R_MSEC, S_MSEC

Description: The millisecond the message was sent.

Available in AxoSyslog version 3.4 and later.

MQTT_TOPIC

Description: The mqtt() source automatically sets the ${MQTT_TOPIC} name-value pair for the messages it receives. This is useful when the name of the topic contains MQTT wildcards ($, +, #).

Available in AxoSyslog version 4.7 and later.

MSG

The ${MSG} macro is an alias of the ${MESSAGE} macro, using ${MSG} in AxoSyslog is equivalent to ${MESSAGE}. For details on this macro, see MESSAGE.

MSGHDR

Description: The name and the PID of the program that sent the log message in PROGRAM[PID]: format. Includes a trailing whitespace. Note that the macro returns an empty value if both the PROGRAM and PID fields of the message are empty.

MSGFORMAT

Available in AxoSyslog version 4.8.1 and later.

Description: Stores the original format of the incoming message. Possible values:

• linux:devkmsg: Linux kernel message.
• linux:pacct: Linux process accounting log format.
• raw: AxoSyslog didn’t parse the message, for example, because the no-parse flag was set.
• syslog:rfc3164: Syslog message formatted as RFC3164.
• syslog:rfc5424: Syslog message formatted as RFC5424.

MSGID

Description: A string specifying the type of the message in IETF-syslog (RFC5424-formatted) messages. For example, a firewall might use the ${MSGID} “TCPIN” for incoming TCP traffic and the ${MSGID} “TCPOUT” for outgoing TCP traffic. By default, AxoSyslog does not specify this value, but uses a dash (-) character instead. If an incoming message includes the ${MSGID} value, it is retained and relayed without modification.

MSGONLY

Description: Message contents without the program name or pid. Starting with AxoSyslog 3.0, the following macros are equivalent: ${MSGONLY}, ${MSG}, ${MESSAGE}. For consistency, use the ${MESSAGE} macro. For details, see MESSAGE.

PID

Description: The PID of the program sending the message.

PRI

Description: The priority and facility encoded as a 2 or 3 digit decimal number as it is present in syslog messages.

PRIORITY or LEVEL

Description: The priority (also called severity) of the message, for example, error. For the textual representation of this value, use the ${LEVEL} macro. See PRIORITY or LEVEL for details.

PROGRAM

Description: The name of the program sending the message. Note that the content of the ${PROGRAM} variable may not be completely trusted as it is provided by the client program that constructed the message.

PROTO

Description: When used, the output specifies the protocol used on the source from which the message originates.

For an example use case when using the macro is recommended, see Example use case: using the $DESTIP, the $DESTPORT, and the $PROTO macros.

RAWMSG

Description: The original message as received from the client. Note that this macro is available only in 3.16 and later, and only if AxoSyslog received the message using the default-network-drivers() source, or the source receiving the message has the store-raw-message flag set.

RAWMSG_SIZE

Available in AxoSyslog version 4.2 and newer.

Description: The original size of the incoming message in bytes. Might not be available for every source driver.

RCPTID

Description: When the use-rcptid global option is set to yes, AxoSyslog automatically assigns a unique reception ID to every received message. You can access this ID and use it in templates via the ${RCPTID} macro. The reception ID is a monotonously increasing 48-bit integer number, that can never be zero (if the counter overflows, it restarts with 1).

RUNID

Description: An ID that changes its value every time AxoSyslog is restarted, but not when reloaded.

SDATA, .SDATA.SDID.SDNAME

Description: The AxoSyslog application automatically parses the STRUCTURED-DATA part of IETF-syslog messages, which can be referenced in macros. The ${SDATA} macro references the entire STRUCTURED-DATA part of the message, while structured data elements can be referenced using the ${.SDATA.SDID.SDNAME} macro.

Example: Using SDATA macros

For example, if a log message contains the following structured data: [exampleSDID@0 iut="3" eventSource="Application" eventID="1011"][examplePriority@0 class="high"] you can use macros like: ${.SDATA.exampleSDID@0.eventSource} — this would return the Application string in this case.

SEC, C_SEC, R_SEC, S_SEC

Description: The second the message was sent.

SEQNUM

Description: The ${SEQNUM} macro contains a sequence number for the log message. The value of the macro depends on the scenario, and can be one of the following:

• If AxoSyslog receives a message via the IETF-syslog protocol that includes a sequence ID, this ID is automatically available in the ${SEQNUM} macro.

• If the message is a Cisco IOS log message using the extended timestamp format, then AxoSyslog stores the sequence number from the message in this macro. If you forward this message the IETF-syslog protocol, AxoSyslog includes the sequence number received from the Cisco device in the ${.SDATA.meta.sequenceId} part of the message.

  Note To enable sequence numbering of log messages on Cisco devices, use the following command on the device (available in IOS 10.0 and later): service sequence-numbers. For details, see the manual of your Cisco device.

• For locally generated messages (that is, for messages that are received from a local source, and not from the network), AxoSyslog calculates a sequence number when sending the message to a destination (it is not calculated for relayed messages).

  • The sequence number is not global, but per-destination. Essentially, it counts the number of messages sent to the destination.
  • This sequence number increases by one for every message sent to the destination. It not lost when AxoSyslog is reloaded, but it is reset when AxoSyslog is restarted.
  • This sequence number is added to every message that uses the IETF-syslog protocol (${.SDATA.meta.sequenceId}), and can be added to BSD-syslog messages using the ${SEQNUM} macro.

SOURCE

Description: The identifier of the source statement in the AxoSyslog configuration file that received the message. For example, if AxoSyslog received the log message from the source s_local { internal(); }; source statement, the value of the ${SOURCE} macro is s_local. This macro is mainly useful for debugging and troubleshooting purposes.

SOURCEIP

Description: IP address of the host that sent the message to syslog-ng. (That is, the IP address of the host in the ${FULLHOST_FROM} macro.) Please note that when a message traverses several relays, this macro contains the IP of the last relay.

STAMP, R_STAMP, S_STAMP

Description: A timestamp formatted according to the ts-format() global or per-destination option.

SYSUPTIME

Description: The time elapsed since the AxoSyslog instance was started (that is, the uptime of the AxoSyslog process). The value of this macro is an integer containing the time in 1/100th of the second.

Available in AxoSyslog version 3.4 and later.

TAG

Description: The priority and facility encoded as a 2 digit hexadecimal number.

TAGS

Description: A comma-separated list of the tags assigned to the message.

TRANSPORT

Available in AxoSyslog version 4.5 and later.

AxoSyslog automatically populates this name-value pair with the “transport” mechanism used to retrieve or receive the message. The exact value depends on the source driver that received the message. Currently the following values are implemented:

• BSD syslog drivers tcp(), udp() & network()

  • rfc3164+tls
  • rfc3164+tcp
  • rfc3164+udp
  • rfc3164+proxied-tls
  • rfc3164+<custom logproto like altp>

• UNIX domain drivers unix-dgram(), unix-stream()

  • unix-stream
  • unix-dgram

• RFC5424-style syslog syslog():

  • rfc5426: syslog over udp
  • rfc5425: syslog over tls
  • rfc6587: syslog over tcp
  • rfc5424+<custom logproto like altp>: syslog over a logproto plugin

• Other drivers:

  • otlp: otel() driver
  • mqtt: mqtt() driver
  • hypr-api: hypr-audit-source() driver

• Locally created logs (in version 4.7 and newer):

  • local+unix-stream
  • local+unix-dgram
  • local+file
  • local+pipe
  • local+program
  • local+devkmsg
  • local+journal
  • local+afstreams
  • local+openbsd

TZ, C_TZ, R_TZ, S_TZ

Description: An alias of the ${TZOFFSET} macro.

TZOFFSET, C_TZOFFSET, R_TZOFFSET, S_TZOFFSET

Description: The time-zone as hour offset from GMT, for example: -07:00. In version 1.6.x this used to be -0700 but as ${ISODATE} requires the colon it was added to ${TZOFFSET} as well.

UNIXTIME, C_UNIXTIME, R_UNIXTIME, S_UNIXTIME

Description: Standard UNIX timestamp, represented as the number of seconds since 1970-01-01T00:00:00.

.tls.x509

Description: When using a transport that uses TLS, these macros contain information about the peer’s certificate. That way, you can use information from the client certificate in filenames, database values, or as other metadata. If you clients have their own certificates, then these values are unique per client, but unchangeable by the client. The following macros are available in AxoSyslog version 3.9 and later.

• .tls.x509_cn: The Common Name of the certificate.
• .tls.x509_o: The value of the Organization field.
• .tls.x509_ou: The value of the Organization Unit field.
• .tls.x509_fp: The key fingerprint of the peer, if the trusted-keys() option is used. Available in version 4.8.1 and later.

UNIQID

Description: A globally unique ID generated from the HOSTID and the RCPTID in the format of HOSTID@RCPTID. For details, see use-uniqid() and RCPTID.

Available in AxoSyslog version 3.7 and later.

USEC, C_USEC, R_USEC, S_USEC

Description: The microsecond the message was sent.

Available in AxoSyslog version 3.4 and later.

YEAR, C_YEAR, R_YEAR, S_YEAR

Description: The year the message was sent.

WEEK, C_WEEK, R_WEEK, S_WEEK

Description: The week number of the year, prefixed with a zero for the first nine weeks of the year. (The first Monday in the year marks the first week.)

See also ISOWEEK, C_ISOWEEK, R_ISOWEEK, S_ISOWEEK.

WEEK_DAY_ABBREV, C_WEEK_DAY_ABBREV, R_WEEK_DAY_ABBREV, S_WEEK_DAY_ABBREV

Description: The 3-letter English abbreviation of the name of the day the message was sent, for example, Thu.

WEEK_DAY, C_WEEK_DAY, R_WEEK_DAY, S_WEEK_DAY

Description: The day of the week as a numerical value (1-7).

WEEKDAY, C_WEEKDAY, R_WEEKDAY, S_WEEKDAY

Description: These macros are deprecated, use ${WEEK_DAY_ABBREV}, ${R_WEEK_DAY_ABBREV}, ${S_WEEK_DAY_ABBREV} instead. The 3-letter name of the day of week the message was sent, for example, Thu.

WEEK_DAY_NAME, C_WEEK_DAY_NAME, R_WEEK_DAY_NAME, S_WEEK_DAY_NAME

Description: The English name of the day.

1.5.1 - Example use case: using the $DESTIP, the $DESTPORT, and the $PROTO macros

This section describes scenarios when Axoflow recommends using the $DESTIP, the $DESTPORT, and the $PROTO macros.

Using the $DESTIP, the $DESTPORT, and the $PROTO macros is relevant when multiple sources are configured to receive messages on the AxoSyslog side. In this case, the hostname and IP address on the sender’s side and the AxoSyslog side is the same, and at a later point in the pipeline, AxoSyslog can not by default specify which source received the message. The $DESTIP, the $DESTPORT, and the $PROTO macros solve this issue by specifying the local IP address and local port of the original message source, and the protocol used on the original message source on the AxoSyslog side.

When to use the $DESTIP, the $DESTPORT, and the $PROTO macros

Axoflow recommends using the $DESTIP, the $DESTPORT, and the $PROTO macros in either of the following scenarios:

• Your appliance sends out log messages through both UDP and TCP.

• The format of the UDP log messages and the TCP log messages is different, and instead of using complex filters, you want to capture either of them, preferably with the simplest possible filter.

• The IP addresses on the sender’s side and the AxoSyslog side are the same, therefore the netmask() option doesn’t work in your configuration.

• The hostnames on the sender’s side and the AxoSyslog side are the same, therefore the host() option doesn’t work in your configuration.

Macros: $DESTIP, $DESTPORT, and $PROTO

To solve either of the challenges listed previously, AxoSyslog supports the following macros that you can include in your configuration:

• $DESTIP

• $DESTPORT

• $PROTO

Configuration and output

The following configuration example illustrates how you can use the $DESTIP, the $DESTPORT, and the $PROTO macros in your AxoSyslog configuration.

Example: using the $DESTIP, the $DESTPORT, and the $PROTO macros in your configuration

The $DESTIP, the $DESTPORT, and the $PROTO macros in your AxoSyslog configuration:

   log{ 
      source{ 
        network(localip(10.12.15.215) port(5555) transport(udp)); 
      };
    
    destination { 
      file("/dev/stdout" template("destip=$DESTIP destport=$DESTPORT proto=$PROTO\n")); 
      };
    };

With these configuration settings, the macros will specify the local IP, the local port, and the protocol information of the source from which the message originates as follows:

   destip=10.12.15.215 destport=5555 proto=17

1.6 - Using template functions

A template function is a transformation: it modifies the way macros or name-value pairs are expanded. Template functions can be used in template definitions, or when macros are used in the configuration of AxoSyslog. Template functions use the following syntax:

   $(function-name parameter1 parameter2 parameter3 ...)

For example, the $(echo) template function simply returns the value of the macro it receives as a parameter, thus $(echo ${HOST}) is equivalent to ${HOST}.

The parameters of template functions are separated by a whitespace character. A template function can have maximum 64 parameters. If you want to use a longer string or multiple macros as a single parameter, enclose the parameter in double-quotes or apostrophes. For example:

   $(echo "${HOST} ${PROGRAM} ${PID}")

Template functions can be nested into each other, so the parameter of a template function can be another template function, like:

   $(echo $(echo ${HOST}))

For details on the available template functions, see the descriptions of the individual template functions in Template functions of AxoSyslog.

You can define your own template function as a regular configuration object (for example, to reuse the same function in different places in your configuration).

Declaration:

   template-function <name-of-the-template-function> "<template-expression-using-strings-macros-template-functions>";

Example: Using custom template functions

The following template function can be used to reformat the message. It adds the length of the message to the message template.

   template-function my-template-function "${ISODATE} ${HOST} message-length=$(length "${MSG}") ${MESSAGE}";
    destination d_file {
        file("/tmp/mylogs.log" template("$(my-template-function)\n"));
    };

You can also refer to existing templates in your template function.

   template my-custom-header-template "${ISODATE} ${HOST_FROM} ${MSGHDR}";
    template-function my-template-function "$(my-custom-header-template) message-length=$(length "${MESSAGE}") ${MESSAGE}";

1.7 - Template functions of AxoSyslog

The following template functions are available in AxoSyslog.

base64-encode

Syntax:

$(base64-encode argument)

Description: You can use the base64-encode template function to base64-encode strings and macros. The template function can receive multiple parameters (maximum 64). In this case, AxoSyslog joins the parameters into a single string and encodes this string. For example, $(base64-encode string1 string2) is equivalent to $(base64-encode string1string2).

Available in AxoSyslog version 3.18 and later.

basename

Syntax:

$(basename argument)

Description: Returns the filename from an argument (for example, a macro: $(basename ${FILE_NAME})) that contains a filename with a path. For example, $(basename "/var/log/messages.log") returns messages.log. To extract the path, use the dirname template function.

Available in AxoSyslog version 3.10 and later.

ceil

Syntax:

$(ceil argument)

Description: Rounds a floating-point number upwards to the nearest integer. For example, $(ceil 1.5) is 2, $(ceil -1.5) is -1. See also the floor and round template functions.

context-lookup

Syntax:

$(context-lookup [option] condition value-to-select)

Description: The context-lookup template function can search a message context when correlating messages (for example, when you use a pattern database or the grouping-by parser). The context-lookup template function requires a condition (a filter or a string), and returns a specific macro or template of the matching messages (for example, the ${MESSAGE}) as a list. It works similarly to the $(grep) template function, but it escapes its output properly, so that the returned value is a list that can be processed with other template functions that work on lists, for example, $(list-slice).

Example: Using the context-lookup template function

The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the tags name-value pair.

   $(context-lookup ("${username}" == "root") ${tags})

To limit the number of matches that the template function returns, use the --max-count option, for example, $(context-lookup --max-count 5 ("${username}" == "root") ${tags}). If you do not want to limit the number of matches, use --max-count 0.

You can to specify multiple name-value pairs as parameters, separated with commas. If multiple messages match the condition of context-lookup, these will be returned also separated by commas. This can be used for example, to collect the email recipients from postfix messages.

Available in AxoSyslog version 3.10 and later.

context-values

Syntax:

$(context-values $name-value1 $name-value2 ...)

Description: The context-values template function returns a list of every occurrence of the specified name-value pairs from the entire context. For example, if the context contains multiple messages, the $(context-values ${HOST}) template function will return a comma-separated list of the ${HOST} values that appear in the context.

Available in AxoSyslog version 3.10 and later.

dirname

Syntax:

$(dirname argument)

Description: Returns the path (without the filename) from an argument (for example, a macro: $(basename ${FILE_NAME}) that contains a filename with a path. For example, $(dirname "/var/log/messages.log") returns /var/log path. To extract the filename, use the basename template function.

Available in AxoSyslog version 3.10 and later.

echo

Syntax:

$(echo argument)

Description: Returns the value of its argument. Using $(echo ${HOST}) is equivalent to ${HOST}.

env

Syntax:

$(env <environment-variable>)

Description: Returns the value of the specified environment variable. Available in AxoSyslog 3.5 and later.

explode

Syntax:

$(explode <separator> <string1> <string2> ...)

Description: Turns a string separated by a specific character into a list. You can also use the implode template function, which turns a list into a string combining the pieces together with a separator. Available in AxoSyslog 3.21 and later.

Example: Using the explode template function

The following configuration example turns strings into a list. If there are several strings, AxoSyslog looks for a separator within each individual string. For example, string 2 is separated as string, 2 in the example below:

Enclose the strings in double-quotes or apostrophes and string 2 is separated as shown below:

The following examples replace the separator ';' character with a ',' character:

filter

Syntax: $(filter <filter-expression> <list>)

Description: Runs the filter expression on each element of a given list, and returns only those list elements that meet the requirements of the filter expression. The current value is referred by $_, similarly to the map template function.

Available in AxoSyslog version 3.30 and later.

Example: using the filter template function in your configuration

When used in configuration as seen in the example, the filter template function filters even numbers from an input list of 0, 1, 2 and 3:

log {
    source { example-msg-generator(num(1) values(INPUT => "0,1,2,3")); };
    destination {
        file("/dev/stdout"
            template("$(filter ('$(% $_ 2)' eq '0') $INPUT)\n)")
        );
    };
};

The returned values are 0 and 2.

Parameters

• <filter-expression>: Mandatory parameter. The <filter-expression> parameter can be:

  • a comparison
  • a filter
  • a logical expression built from filters (using and, or, and not)
  Note

  • When using the <filter-expression> parameter, you can refer other template functions, or use macros.
  • To refer to the variable bound to the current element of the list, use $_ macro.

  The following examples illustrate several ways that you can use a single filter, or a logical expression built from several filters.

      ('1' == '1')
      ('$_' le '1')
      ('$(% $_ 2)' eq '0')
      ('$_' le '1') and ('$(% $_ 2)' eq '0')
  

• list: Mandatory parameter. A AxoSyslog list.

format-cef-extension

AxoSyslog version 3.8 includes a new template function (format-cef-extension) to format name-value pairs as ArcSight Common Event Format extensions. Note that the template function only formats the selected name-value pairs, it does not provide any mapping. There is no special support for creating the prefix part of a Common Event Format (CEF) message. Note that the order of the elements is random. For details on the CEF extension escaping rules format, see the ArcSight Common Event Format.

You can use the value-pairs that AxoSyslog stores about the log message as CEF fields. Using value-pairs, you can:

• select which value-pairs to use as CEF fields,

• add custom value-pairs as CEF fields,

• rename value-pairs, and so on.

For details, see Structuring macros, metadata, and other value-pairs. Note that the syntax of format-* template functions is different from the syntax of value-pairs(): these template functions use a syntax similar to command lines.

Using the format-cef-extension template function has the following prerequisites:

• Set the on-error global option to drop-property, otherwise if the name of a name-value pair includes an invalid character, AxoSyslog drops the entire message. (Key name in CEF extensions can contain only the A-Z, a-z and 0-9 characters.)

      options {
         on-error("drop-property");
      };
  

• The log messages must be encoded in UTF-8. Use the encoding() option or the validate-utf8 flag in the message source.

Example: Using the format-cef-extension template function

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).

   $(format-cef-extension --scope syslog,all_macros,selected_macros \
      --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
      --pair MSGHDR=\"$PROGRAM[$PID]: \")

The following example selects every value-pair that has a name beginning with .cef., but removes the .cef. prefix from the key names.

   template("$(format-cef-extension --subkeys .cef.)\n")

The following example shows how to use this template function to store log messages in CEF format:

   destination d_cef_extension {
        file("/var/log/messages.cef" template("${ISODATE} ${HOST} $(format-cef-extension --scope selected_macros --scope nv_pairs)\n"));
    };

format-cim

Syntax: $(format-cim)

Description: Formats the message into Splunk Common Information Model (CIM) format. Applications that can receive messages in CIM format include Kibana, logstash, and Splunk. Applications that can be configured to log into CIM format include nflog and the Suricata IDS engine.

   destination d_cim {
        network(
            "192.168.1.1"
            template("$(format-cim)\n")
        );
    };

You can find the exact source of this template function in the AxoSyslog GitHub repository.

format-date

Syntax: $(format-date [options] format-string [timestamp])

Available in AxoSyslog 4.1 and later.

Description: The $(format-date) template function takes a timestamp in the DATETIME representation and formats it according to an strftime() format string. The DATETIME representation is a UNIX timestamp formatted as a decimal number, with an optional fractional part, where the seconds and the fraction of seconds are separated by a dot.

If the timestamp argument is missing, the timestamp of the message is used.

Options: --time-zone <TZstring>: Override timezone of the original timestamp

For example: $(format-date --time-zone PST8PDT %Y-%m-%dT%H:%M:%S 1667500613) corresponds to the following format and sets the timezone to Pacific Standard Time, Daylight Saving: “2022-11-03T11:36:53”

format-ewmm

Syntax: $(format-ewmm)

Description: The format-ewmm template function converts the message into the Enterprise-wide message model (EWMM) format. Available in version 3.16 and later.

The following is a sample log message in EWMM format.

   <13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - -
    {"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on
    /dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}

@include "scl.conf"

format-flat-json

Syntax: $(format-flat-json parameters)

Description: The format-flat-json template function is identical to the format-json template function, but nested JSON objects are flattened in the output. If you have to forward your log messages in JSON format, but the receiving application cannot handle nested JSON objects, use the format-flat-json template function.

Example: Flattened JSON output

The following example shows the difference between nested and flattened JSON objects.

• The output of $(format-json a.b.c=1) is a nested JSON object (whitespace added for better readability):

      {
          "a": {
              "b": {
              "c": "1"
              }
          }
      }
  

• The output of $(format-flat-json a.b.c=1) is a flattened JSON object (whitespace added for better readability):

      {
          "a.b.c": "1"
      }
  

For details on formatting log messages into JSON format, see format-json.

format-gelf

Syntax: $(format-gelf)

Description: Available in AxoSyslog 3.13 and later.

You can use the Graylog Extended Log Format (GELF) template together with the graylog2() destination to send syslog messages to Graylog. GELF is the native data format of Graylog.

Example: Using the format-gelf template function

The following configuration example shows how you can use the format-gelf template:

   destination graylog2 {
        network(
            "127.0.0.1"
            port(12201)
            transport(tcp)
            template("$(format-gelf)")
        );
    };

format-json

Syntax: $(format-json parameters)

Description: The format-json template function receives value-pairs as parameters and converts them into JavaScript Object Notation (JSON) format. Including the template function in a message template allows you to store selected information about a log message (that is, its content, macros, or other metadata) in JSON format. Note that the input log message does not have to be in JSON format to use format-json, you can reformat any incoming message as JSON.

You can use the value-pairs that AxoSyslog stores about the log message as JSON fields. Using value-pairs, you can:

• select which value-pairs to use as JSON fields,
• add custom value-pairs as JSON fields,
• rename value-pairs, and so on.

If the value-pair includes type information format-json can propagate it to the next component of the log path. For details, see Structuring macros, metadata, and other value-pairs. Note that the syntax of format-json is different from the syntax of value-pairs(): format-json uses a syntax similar to command lines.

Example: Using the format-json template function

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).

   $(format-json --scope syslog,all_macros,selected_macros
      --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId
      --pair MSGHDR="$PROGRAM[$PID]: ")

The following example shows how to use this template function to store log messages in JSON format:

   destination d_json {
        file(
            "/var/log/messages.json"
            template("$(format-json --scope selected_macros --scope nv_pairs)\n")
        );
    };

$(format-json --leave-initial-dot .SDATA.meta.sequenceID)

If you have to forward your log messages in JSON format, but the receiving application cannot handle nested JSON objects, use the format-flat-json template function. For details, see format-flat-json.

format-welf

This template function converts value-pairs into the WebTrends Enhanced Log file Format (WELF). The WELF format is a comma-separated list of name=value elements. Note that the order of the elements is random. If the value contains whitespace, it is enclosed in double-quotes, for example, name="value". For details on the WELF format, see https://www3.trustwave.com/support/kb/article.aspx?id=10899.

To select which value-pairs to convert, use the command-line syntax of the value-pairs() option. For details on selecting value-pairs, see value-pairs().

Example: Using the format-welf() template function

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).

   $(format-welf --scope syslog,all_macros,selected_macros
      --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId
      --pair MSGHDR="$PROGRAM[$PID]: ")

The following example shows how to use this template function to store log messages in WELF format:

   destination d_welf {
        file(
            "/var/log/messages.welf"
            template("$(format-welf --scope selected_macros --scope nv_pairs)\n")
        );
    };

geoip (DEPRECATED)

This template function is deprecated. Use geoip2 instead.

Syntax: $(geoip <IPv4-address>)

Description: This template function returns the 2-letter country code of any IPv4 address or host. IPv6 addresses are not supported. Currently only the 2-letter codes are supported, and only from the default database. For example, $(geoip $HOST)

To retrieve additional GeoIP information, see Looking up GeoIP data from IP addresses (DEPRECATED).

geoip2

Syntax:

```bash
$(geoip2 --database <path-to-geoip2-database-file>
    [ --field "registered_country.names.ru" ] ${HOST})
```

Description: This template function extracts specific fields from the mmdb database using the --field parameter. If you omit this parameter, it returns the 2-letter country code of any IPv4/IPv6 address or host.

To retrieve additional GeoIP information, see Looking up GeoIP2 data from IP addresses.

Starting with version 3.24, AxoSyslog tries to automatically detect the location of the database. If that is successful, the database() option is not mandatory.

getent

Syntax: $(getent)

Description: Available in AxoSyslog 3.13 and later.

You can use the getent template function to look up entries from the Name Service Switch libraries, such as, passwd, services, or protocols.

The following databases are supported:

• passwd

  Use this database to query data related to a user. Specify the user by either username or user ID. You can query the following data: username, user ID, group ID, GECOS field, home directory, or user shell.

      $(getent passwd testuser name)
      $(getent passwd testuser uid)
      $(getent passwd testuser gid)
      $(getent passwd testuser gecos)
      $(getent passwd testuser dir)
      $(getent passwd testuser shell)
  

  or

      $(getent passwd 1000 name)
      $(getent passwd 1000 uid)
      $(getent passwd 1000 gid)
      $(getent passwd 1000 gecos)
      $(getent passwd 1000 dir)
      $(getent passwd 1000 shell)
  

  The queried data is optional. When you do not query any data, the default behavior applies, which is as follows: user ID is returned for username, or username is returned for user ID.

  • Username $(getent passwd testuser) returns user ID 1000.

  • User ID $(getent passwd 1000) returns username testuser.

• group

  Use this database to query group-related data. The group can be specified using either group ID or group name. You can query the following data: group name, group ID, and members.

      $(getent group adm name)
      $(getent group adm gid)
      $(getent group adm members)
  

  The queried data is optional. The default behavior is as follows: group ID is returned for group name, or group name is returned for user ID.

  • Group name $(getent group adm) returns group ID 4.

  • Group ID $(getent group 4) returns group name adm.

• protocols

  Use this database to translate protocol name to protocol ID, or protocol ID to protocol string.

      $(getent protocols tcp)
      $(getent protocols 6)
  

• services

  Use this database to translate service name to service ID, or service ID to service name.

      $(getent services http)
      $(getent services 80)
  

graphite-output

Syntax: $(graphite-output parameters)

Description: Available in AxoSyslog 3.6 and later. This template function converts value-pairs from the incoming message to the Graphite plain text protocol format.

For details on selecting value-pairs in AxoSyslog and for possibilities to specify which information to convert to Graphite plain text protocol format, see Structuring macros, metadata, and other value-pairs. Note that the syntax of graphite-output is different from the syntax of value-pairs(): graphite-output uses a the command-line syntax used in the format-json template function.

Example: Using the graphite-output template function

The following configuration example shows, how to send value-pairs with names starting with “vmstat.” to Graphite running on localhost, port 2003:

   destination d_graphite {
        network( host("localhost") port(2003) template("$(graphite-output --key vmstat.*)"));
    };

grep

Syntax: $(grep condition value-to-select)

Description: The grep template function can search a message context when correlating messages (for example, when you use a pattern database or the grouping-by parser). The context-lookup template function requires a condition (a filter or a string), and returns a specific macro or template of the matching message (for example, the ${MESSAGE} field of the message).

Example: Using the grep template function

The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the auth_method name-value pair.

   $(grep ("${username}" == "root") ${auth_method})

You can to specify multiple name-value pairs as parameters, separated with commas. If multiple messages match the condition of grep, these will be returned also separated by commas. This can be used for example, to collect the email recipients from postfix messages.

hash

Syntax: $(<method> [opts] $arg1 $arg2 $arg3...)

Options: --length N, -l N

Truncate the hash to the first N characters.

Description: Calculates a hash of the string or macro received as argument using the specified hashing method. If you specify multiple arguments, effectively you receive the hash of the first argument salted with the subsequent arguments.

<method> can be one of md5, md4, sha1, sha256, sha512 and “hash”, which is equivalent to md5. Macros are expected as arguments, and they are concatenated without the use of additional characters.

The md4 <method> is deprecated.

This template function can be used for anonymizing sensitive parts of the log message (for example, username) that were parsed out using PatternDB before storing or forwarding the message. This way, the ability of correlating messages along this value is retained.

Also, using this template, quasi-unique IDs can be generated for data, using the --length option. This way, IDs will be shorter than a regular hash, but there is a very small possibility of them not being as unique as a non-truncated hash.

Example: Using the $(hash) template function

The following example calculates the SHA1 hash of the hostname of the message:

   $(sha1 $HOST)

The following example calculates the SHA256 hash of the hostname, using the salted string to salt the result:

   $(sha1 $HOST salted)

To use shorter hashes, set the --length:

   $(sha1 --length 6 $HOST)

To replace the hostname with its hash, use a rewrite rule:

   rewrite r_rewrite_hostname{set("$(sha1 $HOST)", value("HOST"));};

Example: Anonymizing IP addresses

The following example replaces every IPv4 address in the MESSAGE part with its SHA-1 hash:

   rewrite pseudonymize_ip_addresses_in_message {subst ("((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])[.]){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))", "$(sha1 $0)", value("MESSAGE"));};

if

Syntax: $(if (<condition>) <true template> <false template>)

Description: Returns the value of the <true template> parameter if the <condition> is true. If the <condition> is false, the value of <false template> is returned.

Example: Using pattern databases and the if template function

The following example returns violation if the username name-value pair of a message is root, and system otherwise.

   $(if ("${username}" == "root") "violation" "system")

This can be used to set the class of a message in pattern database rules based on the condition.

   <value name="username">$(if ("${username}" == "root") "violation" "system")</value>

Since template functions can be embedded into each other, it is possible to use another template function as the template of the first one. For example, the following expression returns root if the username is root, admin if the username is joe, and normal user otherwise.

   <value name="username">
        $(if ("${username}" == "root")
            "root"
            $(if ("${username}" == "joe") "admin" "normal user"))</value>

implode

Syntax: $(implode <separator> <string1>, <string2>, ...)

Description: Turns a list into a string combining the pieces together with a separator. You can also use the explode template function, which turns a string separated by a specific character into a list. Available in AxoSyslog 3.21 and later.

Example: Using the implode template function

The following configuration example shows how you can use the implode template to turn a list into a string:

You can also use a $(list-*) template function to further manipulate the list. The following example returns the first three elements of the list:

indent-multi-line

Syntax: $(indent-multi-line parameter)

Description: This template function makes it possible to write multi-line log messages into a file. The first line is written like a regular message, subsequent lines are indented with a tab, in compliance with RFC822.

Example: Using the indent-multi-line template function

The following example writes multi-line messages into a text file.

   destination d_file {
        file (
            "/var/log/messages"
            template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n")
        );
    };

ipv4-to-int

Syntax: $(ipv4-to-int parameter)

Description: Converts the specified IPv4 address to its numeric representation. The numerical value of an IPv4 address is calculated by treating the IP address as a 4-byte hexadecimal value. For example, the 192.168.1.1 address equals to: 192=C0, 168=A8, 1=01, 1=01, or C0A80101, which is 3232235777 in decimal representation.

List manipulation

The list-* template functions allow you to manipulate comma-separated lists. Such lists represent a simple array type in AxoSyslog. Note the following about formatting lists:

• Values are separated by commas, for example, "item1","item2","item3". The single-element list is an element without a comma.

• You can use shell-like quotation to embed commas, for example, "item1","ite\\,m2","item3".

• Empty values are skipped (except if they are quoted)

These template functions return a well-formed list, properly encoding and quoting all elements. If a template function returns a single element, all quotation is decoded and the value contains the literal value.

Starting with AxoSyslog version 3.10, the following list-related template functions are available. Certain functions allow you to reference an element using its number: note that the list index starts with zero, so the index of the first element is 0, the second element is 1, and so on.

list-append

Syntax: $(list-append ${list} ${name-value-pair1} ${name-value-pair2} ... )

Description: Returns a list and appends the values of the specified name-value pairs to the end of the list. You can also append elements to an empty list, for example, $(list-append '' 'element-to-add')

list-concat

Syntax: $(list-concat ${name-value-pair1} ${name-value-pair2} ... )

The commas between the parameters are optional.

Description: This template function creates (concatenates) a list of the values it receives as parameter. The values can be single values (for example, ${HOST}) or lists.

For example, the value of the $(list-concat ${HOST}, ${PROGRAM}, ${PID}) is a comma-separated list.

You can concatenate existing lists into a single list using:

   $(list-concat ${list1} ${list2})

list-count

Syntax: $(list-count ${list} )

Description: Returns the number of elements in the list.

list-head

Syntax: $(list-head ${list} )

Description: Returns the first element of the list, unquoted.

list-nth

Syntax: $(list-nth <index-number> ${list} )

Description: Returns the nth element of the list, unquoted. Note that the list index starts with zero, so (list-nth 1 ${list} ) returns the second element, and so on.

list-search

Syntax: $(list-search [OPTIONS] <pattern> ${list})

Description: The list-search template function searches the elements of ${list} starting at the specified start_index, then returns the index of the first match of <pattern> within ${list}`.

Options:

• --mode MODE: Matching mode, with the following possible values: literal (default), prefix, substring, glob, pcre
• --start-index N: Skips N elements in the ${list}

list-slice

Syntax: $(list-slice <from>:<to> ${list} )

Description: Returns the specified subset of the list. Note that the list index starts with zero, for example, $(list-slice 1:2 ${list} ) returns the second and third element of the list, and so on.

You can omit the from or to index if you want to start the subset from the beginning or end of the list, for example: 3: returns the list starting with the 4th element, while :3 returns the first four elements.

Negative numbers select an element from the end of the list, for example, -3: returns the last three element of the list.

list-tail

Syntax: $(list-tail ${list} )

Description: Returns the list without the first element. For example, if the ${mylist} list contains the one, two, three elements, then $(list-tail ${mylist} ) returns two, three.

length

Syntax: $(length "<macro>")

Description: Returns the length of the macro in characters, for example, the length of the message. For example, the following filter selects messages that are shorter than 16 characters:

   f_short {
        match ('-', value ("$(if ($(length "${MESSAGE}") <= 16) "-" "+")"));
    };

lowercase

Syntax: $(lowercase "<macro>")

Description: Returns the lowercase version of the specified string or macro. For example, the following example uses the lowercase version of the hostname in a directory name:

   destination d_file {
        file ("/var/log/${MONTH}/${DAY}/$(lowercase "${HOST}")/messages");
    };

Available in AxoSyslog 3.5 and later.

map

Syntax: $(map template list)

Description: Returns with a list that contains the results of applying a template function for each elements of a list.

Available in AxoSyslog version 3.28 and later.

Parameters:

• template: Mandatory. This template function is applied for each elements of the list. Use $_ macro to refer to the current list element.

• list: Mandatory. A list, or template.

Example:

When used in configuration as seen in the example, the map template function adds one to each element of a list:

   log {
      source { example-msg-generator(num(1) values(LST => "0,1,2")); };
      destination {
        file("/dev/stdout"template('$(map "$(+ 1 $_)" $LST)')
        );
      };
    };

The returned values are 1, 2, and 3.

Numerical operations

Syntax: $(<operation> "<value1>" "<value2>")

Description: These template functions allow you to manipulate numbers, that is, to perform addition (+), substraction (-), multiplication (*), division (/), and modulus (%). All of them require two numeric arguments. The result is NaN (Not-a-Number) if the parameters are not numbers, cannot be parsed, or if a division by zero would occur. For example, to add the value of two macros, use the following template function:

   $(+ "${<MACRO1>}" "${<MACRO2>}");

Starting with AxoSyslog version 3.22 and later, the numerical operators support floating-point values. They behave like the operators in the C programming language:

• If both operands are integers, they return an integer.

• If any of the operands is a floating-point number, they return a floating-point result.

For example:

   $(/ 3 2) # Both operands are integers, so the result is 1
    # One of the operands is a floating point number, so the result is also floating-point
    $(/ 3.0 2) # = 1.500000
    $(/ 3 2.0) # = 1.500000
    $(/ 3.0 2.0) # = 1.500000

To round floating-point numbers, you can use the ceil, floor, and round template functions.

When you are correlating messages and a name-value pair contains numerical values in the messages, you can calculate the lowest (min), highest (max), total (sum), and mean (average) values. These calculations process every message of the correlation context. For details on message correlation, see Correlating log messages. For example, if the messages of the context have a .myfields.load name-value pair, you can find the highest load value using the following template function.

   $(max ${.myfields.load})

or

Syntax: $(or <macro1> <macro2>)

Description: This template function returns the first non-empty argument.

padding

Syntax:

 $(padding <macro> <width> <prepended-character-or-string>)

Description: This template function returns the value of its first parameter (a string or macro), prepended with a string. This string is <width> long, and repeats the character or string set in the third parameter. If you use a single character, it is added >times. If you use a string, it is repeated until its length reaches <width>>The default padding character is ’ ’ (space). For example:

Example: Using the padding template function

If the value of the ${MESSAGE} macro is mymessage, then the output of the padding() template function is the following:

   $(padding ${MESSAGE} 10 X)

Output: XXXXXXXXXXmymessage

   $(padding ${MESSAGE} 10 foo)

Output: foofoofoofmymessage

python

Syntax:

 $(python <name-of-the-python-method-to-use> <arguments-of-the-method>)

Description: This template function enables you to write a custom template function in Python. You can define a Python block in your AxoSyslog configuration file, define one or more Python functions in it, and use the methods as template functions. If you use a Python block, AxoSyslog embeds a Python interpreter to process the messages.

The following points apply to using Python blocks in AxoSyslog in general:

• Python parsers and template functions are available in AxoSyslog version 3.10 and later.

  Python destinations and sources are available in AxoSyslog version 3.18 and later.

• Supported Python versions: 2.7 and 3.4+ (if you are using pre-built binaries, check the dependencies of the package to find out which Python version it was compiled with).

• The Python block must be a top-level block in the AxoSyslog configuration file.

• If you store the Python code in a separate Python file and only include it in the AxoSyslog configuration file, make sure that the PYTHONPATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start AxoSyslog manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc.

  In production, when AxoSyslog starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting AxoSyslog. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc".

• The Python object is initiated every time when AxoSyslog is started or reloaded.

  Warning If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.

• The Python block can contain multiple Python functions.

• Using Python code in AxoSyslog can significantly decrease the performance of AxoSyslog, especially if the Python code is slow. In general, the features of AxoSyslog are implemented in C, and are faster than implementations of the same or similar features in Python.

• Validate and lint the Python code before using it. The AxoSyslog application does not do any of this.

• Python error messages are available in the internal() source of AxoSyslog.

• You can access the name-value pairs of AxoSyslog directly through a message object or a dictionary.

• To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of AxoSyslog. For details, see Logging from your Python code.

The following points apply to Python parsers.

• The first argument in the definition of the Python function is the actual log message. This is implicitly passed to the function, you do not have to use it in the template function.

• The value of the template function is return value of the Python function.

• To reference a name-value pair or a macro in the Python function, use the dot-notation. For example, if the first argument in the definition of the function is called log-message, the value of the HOST macro is log-message.HOST, and so on.

• You can define new name-value pairs in the Python function. For example, if the first argument in the definition of the function is called log-message, you can create a new name-value pair like this: log_message["new-macro-name"]="value". This is useful when you parse a part of the message from Python, or lookup a value based on data extracted from the log message.

Declaration:

   python {
    def <name_of_the_python_function>(<log_message>, <optional_other_arguments>):
        # <your-python-code>
        return <value_of_the_template_function>
    };
    
    template <template-name> {
        template($(python <name_of_the_python_function>));
    };

Example: Writing template functions in Python

The following example creates a Python template function called return_message that returns the MESSAGE part of the log message.

   @version: 4.9.0
    
    python {
    def return_message(log_message):
        return log_message.MESSAGE
    };
    
    destination d_local {
        file("/tmp/logs.txt" template("[$(python return_message)]\n"));
    };

The following example creates a Python template function called resolve_host that receives an IP address as an argument, and attempts to resolve it into a hostname.

   @version: 4.9.0
    
    python {
    import socket
    
    def resolve_host(log_message, hostname):
        try:
            return socket.gethostbyaddr(hostname)[0]
        except (socket.herror, socket.error):
            return 'unknown'
    };
    
    destination d_local {
        file(
            "/tmp/logs.txt"
            template("${ISODATE} $(python resolve_host ${SOURCE_IP}) ${MESSAGE}\n")
        );
    };

replace-delimiter

Syntax: $(replace-delimiter "<old-delimiters>" "<new-delimiter>" "<macro>")

Description: Replaces the delimiter character with a new one. For example, the following example replaces the tabulators (\\t) in the message with semicolons (;):

   $(replace-delimiter "\t" ";" "${MESSAGE}")

Available in AxoSyslog 3.5 and later.

round

Syntax: $(round argument)

Description: Rounds a floating-point number to the nearest integer. For example, $(round 1.5) is 2. See also the ceil and floor template functions.

This template function has an optional second argument that sets the precision of rounding. The default is 0 (output a natural number), but values up to 20 are accepted. For example, $(round 2.123456 4) is 2.1235.

sanitize

Syntax: $(sanitize <options> "<macro1>" "<macro2> ...")

Description: This file replaces the special characters in macro values, for example, it can replace the slash (/) characters in a filename with the underscore (_) character. If you specify multiple arguments, they will be concatenated using the / character, so they can be used as separate directory levels when used in filenames.

The function has the following options:

• --ctrl-chars or -c

• Filter control characters (characters that have an ASCII code of 32 or lower). This option is used by default.

• --invalid-chars <characterlist> or -i <characterlist>

• The list of characters to be replaced with underscores (_). The default list contains the / character. The following example replaces the \ and @ characters, so for example, fo\o@bar becomes foobar:

      $(sanitize -i \@ $PROGRAM)
  

• --no-ctrl-chars or -C

• Do not filter the control characters (characters that have an ASCII code of 32 or lower).

• --replacement <replacement-character> or -r <replacement-character>

• The character used to replace invalid characters. By default, this is the underscore (_). The following example replaces invalid characters with colons instead of underscores, so for example, foo/bar becomes foo;bar:

      $(sanitize -r ; $PROGRAM)
  

Example: Using the sanitize template function

The following example uses the sanitize function on two macros, and the results are used as directory names in a file destination.

   file("/var/log/$(sanitize $HOST $PROGRAM)/messages");

This is equivalent to file("/var/log/$HOST/$PROGRAM/messages");, but any slashes in the values of the $HOST and $PROGRAM macros are replaced with underscores.

stardate

Syntax: $(stardate [option] "<date-in-unixtime>")

Description: Converts a date in UNIXTIME (for example, ${UNIXTIME}) into stardate, displaying the year and the progress of the year in a number of digits (YYYY.NNN). You can set the number of digits using the --digits option, for example:

   $(stardate --digits 2 "${R_UNIXTIME}")

strip

Syntax: $(strip "<macro>")

Description: Deletes whitespaces from the beginning and the end of a macro. You can specify multiple macros separated with whitespace in a single template function, for example:

   $(strip "${MESSAGE}" "${PROGRAM}")

substr

Syntax: $(substr "<argument>" "<offset>" "<length>")

Description: This function extracts a substring of a string.

• argument The string to extract the substring from, for example, "${MESSAGE}"

• offset Specifies where the substring begins (in characters). 0 means to start from the beginning of the string, 5 means to skip the first 5 characters of the string, and so on. Use negative numbers to specify where to start from the end of the string, for example, -1 means the last character, -5 means to start five characters before the end of the string.

• length Optional parameter: The number of characters to extract. If not specified, the substring will be extracted from the offset to the end of the string. Use negative numbers to stop the substring before the end of the string, for example, -5 means the substring ends five characters before the end of the string.

Example: Using the substr template function

Skip the first 15 characters of the message, and select the rest:

   $(substr "${MESSAGE}" "15");

Select characters 16-30 of the message (15 characters with offset 15):

   $(substr "${MESSAGE}" "15" "15");

Select the last 15 characters of the message:

   $(substr "${MESSAGE}" "-15");

A template that converts the message to RFC3164 (BSD-syslog) format and truncates the messages to 1023 characters:

   template t_truncate_messages {
        template("$(substr \"<$PRI>$DATE $HOST $MSGHDR$MESSAGE\" \"0\" \"1023\")\n");
        template-escape(no);
    };

tag

Available in AxoSyslog version 4.6 and later.

Syntax:

$(tag <name-of-the-tag> <value-if-set> <value-if-unset>)

Description: Adds bit-like tags to the messages.

• If you do not set the value-if-set and value-if-unset arguments, the $(tag) template function acts as a boolean and expands to 0 or 1, depending on whether the message has the specified tag set.
• If the value-if-set and value-if-unset arguments are set, $(tag) returns a string: the second argument (<value-if-set>) if the message has <tag>, and the third argument (<value-if-unset>) if the message doesn’t have <tag>.

tags-head

Available in AxoSyslog version 4.7 and later.

Syntax:

$(tags-head <name-of-first-tag> <name-of-second-tag> ... )

Description: This template function accepts multiple tag names, and returns the first one that is set.

Example config:

# resolves to "bar" if "bar" tag is set, but "foo" is not
template("$(tags-head foo bar baz)")

template

Syntax:

$(template <template-name>)
$(template $<dynamic-template-name>)
$(template $<dynamic-template-name> '<optional-fallback-template>')

Description: This template function looks up the in the configuration and uses that to format its result. The referenced template can be static or dynamic. For static templates, AxoSyslog resolves the template when it starts, or when the configuration is reloaded. For dynamic templates, the results are resolved runtime (for dynamic templates, the template name contains at least one ‘$’ character). For example, the name of the template to be invoked can be extracted from the message, or from a name-value pair set using the add-contextual-data() feature.

For dynamic templates, you can set an optional second template. This second template will be the results of the template function if resolving the dynamic template fails for some reason. For example:

   $(template ${my-dynamic-template} '$DATE $HOST $MSGHDR$MSG\n')

Available in AxoSyslog 3.22 and later.

uppercase

Syntax: $(uppercase "<macro>")

Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:

   destination d_file {
        file ("/var/log/${MONTH}/${DAY}/$(uppercase ${HOST})/messages");
    };

Available in AxoSyslog 3.5 and later.

url-decode

Syntax: $(url-decode <string-pr-macro-1> <string-pr-macro-2> ... )

Description: You can use the url-decode template function to decode url-encoded strings and macros. For example, $(url-decode %3C%3E) yields <>. The url-decode` can receive multiple parameters (maximum 64). In this case, each parameter is decoded separately, and simply concatenated.

Available in AxoSyslog version 3.18 and later.

url-encode

Syntax: $(url-encode ${MESSAGE})

Description: The url-encode template function escapes strings for use as URI path or query parameter segments or form encoded HTTP POST data. All input characters that are not a-z, A-Z, 0-9, ‘-’, ‘.’, ‘_’ or ‘~’ are converted to their “URL escaped” version.

Available in AxoSyslog version 3.18 and later. (In version 3.16-3.17, this template function was called urlencode.)

uuid

Syntax: $(uuid)

Description: Generates a Universally Unique IDentifier (UUID) that complies with RFC4122. That way, an UUID can be added to the message soon after it is received, so messages stored in multiple destinations can be identified. For example, when storing messages in a database and also in files, the UUID can be used to find a particular message both in the database and the files.

To generate a UUID, you can use a rewrite rule to create a new value-pair for the message.

Example: Using Universally Unique Identifiers

The following example adds a value-pair called MESSAGE_UUID to the message using a rewrite rule and a template.

   rewrite r_add_uuid {
        set("$(uuid)" value("MESSAGE_UUID"));
    };
    
    destination d_file {
        file (
            "/var/log/messages"
            template("$MESSAGE_UUID $ISODATE $HOST $MSG\n")
            template-escape(no)
        );
    };
    
    log {
        source(s_network);
        rewrite(r_add_uuid);
        destination(d_file);
    };

1.8 - Modifying the on-the-wire message format

Macros, templates, and template functions allow you to fully customize the format of the message. This flexibility makes it possible to use AxoSyslog in some unexpected way if needed, for example, to emulate simple, plain-text protocols. The following example shows you how to send LPUSH commands to a Redis server.

The following template is a valid LPUSH command in accordance with the Redis protocol, and puts the $MESSAGE into a separate list for every $PROGRAM:

   template t_redis_lpush {
        template("*3\r\n$$5\r\nLPUSH\r\n$$$(length ${PROGRAM})\r\n${PROGRAM}\r\n$$$(length ${MESSAGE})\r\n${MESSAGE}\r\n");
    };

If you use this template in a network() destination, AxoSyslog formats the message according to the template, and sends it to the Redis server.

   destination d_redis_tcp {
        network("127.0.0.1" port(6379) template(t_redis_lpush));
    };

2 - Modifying messages using rewrite rules

The AxoSyslog application can rewrite parts of the messages using rewrite rules. Rewrite rules are global objects similar to parsers and filters and can be used in log paths. The AxoSyslog application has two methods to rewrite parts of the log messages: substituting (setting) a part of the message to a fix value, and a general search-and-replace mode.

• Substitution completely replaces a specific part of the message that is referenced using a built-in or user-defined macro.

• General rewriting searches for a string in the entire message (or only a part of the message specified by a macro) and replaces it with another string. Optionally, this replacement string can be a template that contains macros.

Rewriting messages is often used in conjunction with message parsing parser: Parse and segment structured messages.

Rewrite rules are similar to filters: they must be defined in the syslog-ng.conf configuration file and used in the log statement. You can also define the rewrite rule inline in the log path.

2.1 - Replacing message parts

To replace a part of the log message, you have to:

• define a string or regular expression to find the text to replace

• define a string to replace the original text (macros can be used as well)

• select the field of the message that the rewrite rule should process

Substitution rules can operate on any soft macros, for example, MESSAGE, PROGRAM, or any user-defined macros created using parsers. You can also rewrite the structured-data fields of messages complying to the RFC5424 (IETF-syslog) message format.

Substitution rules use the following syntax:

Declaration:

   rewrite <name_of_the_rule> {
        subst(
            "<string or regular expression to find>",
            "<replacement string>", value(<field name>), flags()
        );
    };

The type() and flags() options are optional. The type() specifies the type of regular expression to use, while the flags() are the flags of the regular expressions. For details on regular expressions, see Regular expressions.

A single substitution rule can include multiple substitutions that are applied sequentially to the message. Note that rewriting rules must be included in the log statement to have any effect.

Example: Using substitution rules

The following example replaces the IP in the text of the message with the string IP-Address.

   rewrite r_rewrite_subst{
        subst("IP", "IP-Address", value("MESSAGE"));
    };

To replace every occurrence, use:

   rewrite r_rewrite_subst{
        subst("IP", "IP-Address", value("MESSAGE"), flags("global"));
    };

Multiple substitution rules are applied sequentially. The following rules replace the first occurrence of the string IP with the string IP-Addresses.

   rewrite r_rewrite_subst{
        subst("IP", "IP-Address", value("MESSAGE"));
        subst("Address", "Addresses", value("MESSAGE"));
    };

Example: Anonymizing IP addresses

The following example replaces every IPv4 address in the MESSAGE part with its SHA-1 hash:

   rewrite pseudonymize_ip_addresses_in_message {subst ("((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])[.]){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))", "$(sha1 $0)", value("MESSAGE"));};

2.2 - Setting message fields to specific values

To set a field of the message to a specific value, you have to:

• define the string to include in the message, and
• select the field where it should be included.

• You can set the type of the field. Where you can use of templates in set() and groupset(), you can use type-casting, and the type information is properly promoted. For details, see Specifying data types in value-pairs.

You can set the value of available macros, for example, HOST, MESSAGE, PROGRAM, or any user-defined macros created using parsers (for details, see parser: Parse and segment structured messages and db-parser: Process message content with a pattern database (patterndb)). Note that the rewrite operation completely replaces any previous value of that field.

Use the following syntax:

Declaration:

   rewrite <name_of_the_rule> {
        set("<string to include>", value(<field name>));
    };

Example: Setting message fields to a particular value

The following example sets the HOST field of the message to myhost.

   rewrite r_rewrite_set{
        set("myhost", value("HOST"));
    };

The following example appends the “suffix” string to the MESSAGE field:

   rewrite r_rewrite_set{
        set("$MESSAGE suffix", value("MESSAGE"));
    };

For details on rewriting SDATA fields, see Creating custom SDATA fields.

You can also use the following options in rewrite rules that use the set() operator.

   rewrite <name_of_the_rule> {
        set("<string to include>", value(<field name>), on-error("fallback-to-string");
    };

2.3 - Setting severity with the set-severity() rewrite function

It is possible to configure the severity field with the set-severity() rewrite function. When configured, the set-severity() rewrite function will only rewrite the $SEVERITY field in the message to the first parameter value specified in the function.

Declaration

   rewrite <name_of_the_rule> {
        set-severity("severity string or number");
    };

Parameters

The set-severity() rewrite function has a single, mandatory parameter that can be defined as follows:

   `set-severity( "parameter1" );`

Accepted values

The set-severity() rewrite function accepts numeric values, named values, and aliases. Aliases are available in AxoSyslog version 4.6 and later.

Example usage for the set-severity() rewrite function

The following examples can be used in production for the set-severity() rewrite function.

Example using string:

   rewrite {
        set-severity("info");
    };

Example using numeric string:

   rewrite {
        set-severity("6");
    };

Example using template:

   rewrite {
        set-severity("${.json.severity}");
    };

2.4 - Setting the facility field with the set-facility() rewrite function

It is possible to set the facility field with the set-facility() rewrite function. When set, the set-facility() rewrite function will only rewrite the $PRIORITY field in the message to the first parameter value specified in the function.

Declaration

   log {
                    source { system(); };
                        if (program("postfix")) {
                          rewrite { set-facility("mail"); };
                        };
                        destination { file("/var/log/mail.log"); };
                        flags(flow-control);
                    };

Parameters

The set-facility() rewrite function has a single, mandatory parameter that can be defined as follows:

   `set-facility( "parameter1" );`

Accepted values

The set-facility() rewrite function accepts the following values:

• numeric strings: [0-7]
• named values: emerg, emergency, panic, alert, crit, critical, err, error, warning, warn, notice, info, informational, debug

Example usage for the set-facility() rewrite function

The following example can be used in production for the set-facility() rewrite function.

   rewrite {
    set-facility("info");
    set-facility("6");
    set-facility("${.json.severity}");};

2.5 - Setting the priority of a message with the set-pri() rewrite function

You can set the PRI value of a BSD or IETF syslog message with the set-pri() rewrite function by specifying a template string. This is useful, for example, if incoming messages do not have a PRI value specified by default, but a PRI value is required for filtering purposes.

When configured, the set-pri() function will only rewrite the PRI value of the message field.

Declaration

   rewrite <rule-name> {
        set-pri("template-string");
    };

Parameters

The set-pri() rewrite function expects a template string as its only parameter, for example:

• set-pri(“42”);

• set-pri("$.json.priority");

Accepted values

The template string specified for the set-pri() rewrite function must expand to a natural number in the interval of 0–1023, inclusive. This means that if you, for example, extract the value from a syslog <PRI> header (such as <42>), then you need to remove the opening and closing brackets (< >) in the specified template string.

Example: Temporarily raising the priority of an application

In the following example, the set-pri() rewrite function is used to temporarily raise the priority of the application myprogram:

   log {
      source { system(); };
      if (program("myprogram")){
      rewrite { set-pri("92"); };
      };
      destination { file("/var/log/mail.log"); };
      flags(flow-control);
    }

Example: Changing the priority of an application log message in JSON format

In the following example, an application sends log messages in the following JSON format:

   {
    "time": "2003-10-11T22:14:15.003Z",
    "host": "mymachine",
    "priority": "165",
    "message": "An application event log entry."
    }

You can parse these logs with the JSON parser function:

   {
    parser p_json {
    json-parser (prefix(".json."));
    }

As the application message contains a valid priority field, you can use the set-pri() rewrite function to modify the priority of the message:

   set-pri("$.json.priority");

2.6 - Setting match variables with the set-matches() rewrite rule

Match macros ($1, $2, ... $255) are temporary variables. You can use them for general purposes when operating with list-like items. For example, the match() filter stores capture group results in match variables when the store-matches flag is set, or the JSON parser produces match variables if the parsed JSON data is an array.

It is possible to set match variables in a single operation with the set-matches() rewrite function. set-matches() uses AxoSyslog list expressions to set $1, $2, ... $255, so it can be considered as a conversion function between AxoSyslog lists and match variables.

Declaration

   rewrite <name_of_the_rule> {
        set-matches("<list-expression or list-based template function>");
    };

Example usage for the set-matches() rewrite function

In the following two examples, $1, $2, and $3 will be set to foo, bar, and baz, respectively.

Example using string:

   rewrite {
        set-matches("foo,bar,baz");
    };

Example using a list template function:

   rewrite {
        set-matches("$(explode ':' 'foo:bar:baz')");
    };

2.7 - Unsetting message fields

You can unset macros or fields of the message, including any user-defined macros created using parsers (for details, see parser: Parse and segment structured messages and db-parser: Process message content with a pattern database (patterndb)). Note that the unset operation completely deletes any previous value of the field that you apply it on.

Use the following syntax:

Declaration:

   rewrite <name_of_the_rule> {
        unset(value("<field-name>"));
    };

Example: Unsetting a message field

The following example unsets the HOST field of the message.

   rewrite r_rewrite_unset{
        unset(value("HOST"));
    };

To unset a group of fields, you can use the groupunset() rewrite rule.

Declaration:

   rewrite <name_of_the_rule> {
        groupunset(values("<expression-for-field-names>"));
    };

Example: Unsetting a group of fields

The following rule clears all SDATA fields:

   rewrite r_rewrite_unset_SDATA{
        groupunset(values(".SDATA.*"));
    };

2.8 - Renaming message fields

If you want to change the name of a field of a message, you can use rename() rewrite rules. This can be also achieved via using set() and unset() but those require extra conditions and two operation instead of one.

The rename() rewrite rule uses positional arguments and they are both required. It supports condition rewrite. For more information, see Conditional rewrites.

Declaration

   rewrite r_rewrite_rename {
        rename("<string1>" "<string2>");
    };

Example usage for the rename() rewrite function

The following example renames the .app.name into .container if the .app.name exists. Otherwise, it does nothing.

   rewrite r_rewrite_rename {
        rename(".app.name" ".container");
    };

2.9 - Creating custom SDATA fields

If you use RFC5424-formatted (IETF-syslog) messages, you can also create custom fields in the SDATA part of the message (For details on the SDATA message part, see The STRUCTURED-DATA message part). According to RFC5424, the name of the field (its SD-ID) must not contain the @ character for reserved SD-IDs. Custom SDATA fields must be in the following format: .SDATA.group-name@<private enterprise number>.field-name, for example, .SDATA.mySDATA-field-group@18372.4.mySDATA-field. (18372.4 is the private enterprise number of Axoflow, the developer of AxoSyslog.)

Example: Rewriting custom SDATA fields

The following example sets the sequence ID field of the RFC5424-formatted (IETF-syslog) messages to a fixed value. This field is a predefined SDATA field with a reserved SD-ID, therefore its name does not contain the @ character.

   rewrite r_sd {
        set("55555" value(".SDATA.meta.sequenceId"));
    };

It is also possible to set the value of a field that does not exist yet, and create a new, custom name-value pair that is associated with the message. The following example creates the .SDATA.groupID.fieldID@18372.4 field and sets its value to yes. If you use the ${.SDATA.groupID.fieldID@18372.4} macro in a template or SQL table, its value will be yes for every message that was processed with this rewrite rule, and empty for every other message.

The next example creates a new SDATA field-group and field called custom and sourceip, respectively:

   rewrite r_rewrite_set {
        set("${SOURCEIP}" value(".SDATA.custom@18372.4.sourceip"));
    };

If you use the ${.SDATA.custom@18372.4.sourceip} macro in a template or SQL table, its value will be that of the SOURCEIP macro (as seen on the machine where the SDATA field was created) for every message that was processed with this rewrite rule, and empty for every other message.

You can verify whether or not the format is correct by looking at the actual network traffic. The SDATA field-group will be called custom@18372.4, and sourceip will become a field within that group. If you decide to set up several fields, they will be listed in consecutive order within the field-group’s SDATA block.

2.10 - Setting multiple message fields to specific values

The groupset() rewrite rule allows you to modify the value of multiple message fields at once, for example, to change the value of sensitive fields extracted using patterndb, or received in a JSON format. (If you want to modify the names of message fields, see map-value-pairs: Rename value-pairs to normalize logs.)

• The first parameter is the new value of the modified fields. This can be a simple string, a macro, or a template (which can include template functions as well).
• The second parameter (values()) specifies the fields to modify. You can explicitly list the macros or fields (a space-separated list with the values enclosed in double-quotes), or use wildcards and glob expressions to select multiple fields.
• Note that groupset() does not create new fields, it only modifies existing fields.
• You can refer to the old value of the field using the $_ macro. This is resolved to the value of the current field, and is available only in groupset() rules.

• You can set the type of the field. Where you can use of templates in set() and groupset(), you can use type-casting, and the type information is properly promoted. For details, see Specifying data types in value-pairs.

Declaration:

   rewrite <name_of_the_rule> {
        groupset("<new-value-of-the-fields>", values("<field-name-or-glob>" ["<another-field-name-or-glob>"]));
    };

Example: Using groupset rewrite rules

The following examples show how to change the values of multiple fields at the same time.

• Change the value of the HOST field to myhost.

      groupset ("myhost" values("HOST"))
  

• Change the value of the HOST and FULLHOST fields to myhost.

      groupset ("myhost" values("HOST" "FULLHOST"))
  

• Change the value of the HOST, FULLHOST and fields to lowercase.

      groupset ("$(lowercase "$_")" values("HOST" "FULLHOST"))
  

• Change the value of each field and macro that begins with .USER to nobody.

      groupset ("nobody" values(".USER.*"))
  

• Change the value of each field and macro that begins with .USER to its SHA-1 hash (truncated to 6 characters).

      groupset ("$(sha1 --length 6 $_)" values(".USER.*"))
  

2.11 - map-value-pairs: Rename value-pairs to normalize logs

The map-value-pairs() parser allows you to map existing name-value pairs to a different set of name-value pairs. You can rename them in bulk, making it easy to use for log normalization tasks (for example, when you parse information from different log messages, and want to convert them into a uniform naming scheme). You can use the normal value-pairs expressions, similarly to value-pairs based destinations. Using map-value-pairs() retains type data if available.

Available in AxoSyslog version 3.10 and later.

Declaration:

   parser parser_name {
        map-value-pairs(
            <list-of-value-pairs-options>
        );
    };

Example: Map name-value pairs

The following example creates a new name-value pair called username, adds the hashed value of the .apache.username to this new name-value pair, then adds the webserver prefix to the name of every name-value pair of the message that starts with .apache

   parser p_remap_name_values {
        map-value-pairs(
            pair("username", "'($sha1 $.apache.username)")
            key('.apache.*' rekey(add-prefix("webserver")))
        );
    };

2.12 - Conditional rewrites

Starting with 3.2, it is possible to apply a rewrite rule to a message only if certain conditions are met. The condition() option effectively embeds a filter expression into the rewrite rule: the message is modified only if the message passes the filter. If the condition is not met, the message is passed to the next element of the log path (that is, the element following the rewrite rule in the log statement, for example, the destination). Any filter expression normally used in filters can be used as a rewrite condition. Existing filter statements can be referenced using the filter() function within the condition. For details on filters, see Filters.

Using conditional rewrite

The following procedure summarizes how conditional rewrite rules (rewrite rules that have the condition() parameter set) work. The following configuration snippet is used to illustrate the procedure:

   rewrite r_rewrite_set{
        set(
            "myhost",
            value("HOST")
            condition(program("myapplication"))
        );
    };
    log {
        source(s1);
        rewrite(r_rewrite_set);
        destination(d1);
    };

To configure condtional rewrite

1. The log path receives a message from the source (s1).

2. The rewrite rule (r_rewrite_set) evaluates the condition. If the message matches the condition (the PROGRAM field of the message is “myapplication”), AxoSyslog rewrites the log message (sets the value of the HOST field to “myhost”), otherwise it is not modified.

3. The next element of the log path processes the message (d1).

Example: Using conditional rewriting

The following example sets the HOST field of the message to myhost only if the message was sent by the myapplication program.

rewrite r_rewrite_set{set("myhost", value("HOST") condition(program("myapplication")));};

The following example is identical to the previous one, except that the condition references an existing filter template.

filter f_rewritefilter {program("myapplication");};
rewrite r_rewrite_set{set("myhost", value("HOST") condition(filter(f_rewritefilter)));};

2.13 - Adding and deleting tags

To add or delete a tag, you can use rewrite rules. To add a tag, use the following syntax:

   rewrite <name_of_the_rule> {
        set-tag("<tag-to-add>");
    };

To delete a tag, use the following syntax:

   rewrite <name_of_the_rule> {
        clear-tag("<tag-to-delete>");
    };

Templates (macros, template functions) can be used when specifying tags, for example, set-tag("dyn::$HOST");.

2.14 - Rewrite the timezone of a message

Starting with version 3.24 of the AxoSyslog application, you can manipulate the timezone information of messages using rewrite rules. You can:

• Set a timezone to a specific value

• Fix a timezone if it was improperly parsed

• Assuming the sender is sending messages in near-real-time, AxoSyslog can guess the timezone

By default, these operations modify the date-related macros of the message that correspond to the date the message was sent (that is, the S_ macros). You can modify the dates when AxoSyslog has received the messages (that is, the R_ macros), but this is rarely needed. To do so, include the time-stamp(recvd) option in the operation, for example:

   rewrite { fix-time-zone("EST5EDT" time-stamp(recvd)); };

fix-time-zone()

Use the fix-time-zone() operation to correct the timezone of a message if it was parsed incorrectly for some reason, or if the client did not include any timezone information in the message. You can specify the new timezone as the name of a timezone, or as a template string. For example, use the following rewrite rule to set the timezone to EST5EDT:

   rewrite { fix-time-zone("EST5EDT"); };

If you have lots of clients that do not send timezone information in the log messages, you can create a database file that stores the timezone of the clients, and feed this data to AxoSyslog using the add-contextual-data() feature. For details, see Adding metadata from an external file.

guess-time-zone()

Use the guess-time-zone() operation attempts to set the timezone of the message automatically, using heuristics on the timestamps. Normally the AxoSyslog application performs this operation automatically when it parses the incoming message. Using this operation in a rewrite rule can be useful if you cannot parse the incoming message for some reason (and use the flags(no-parse) option in your source, but you want to set the timezone automatically later (for example, after you have preprocessed the message).

Using this operation is identical to using the flags(guess-timezone) flag in the source.

set-time-zone()

Use the set-time-zone() operation to set the timezone of the message to a specific value, that is to convert an existing timezone to a different one. This operation is identical to setting the time-zone() option in a destination or as a global option, but can be applied selectively to the messages using conditions.

2.15 - Anonymizing credit card numbers

Log messages of banking and e-commerce applications might include credit card numbers (Primary Account Number or PAN). According to privacy best practices and the requirements of the Payment Card Industry Data Security Standards (PCI-DSS), PAN must be rendered unreadable. The AxoSyslog application uses a regular expression to detect credit card numbers, and provides two ways to accomplish this: you can either mask the credit card numbers, or replace them with a hash. To mask the credit card numbers, use the credit-card-mask() or the credit-card-hash() rewrite rules in a log path.

Declaration:

   @include "scl/rewrite/cc-mask.conf"
    
    rewrite {
        credit-card-mask(value("<message-field-to-process>"));
    };

By default, these rewrite rules process the MESSAGE part of the log message.

credit-card-hash()

Description: Process the specified message field (by default, ${MESSAGE}), and replace any credit card numbers (Primary Account Number or PAN) with a 16-character-long hash. This hash is generated by calculating the SHA-1 hash of the credit card number, selecting the first 64 bits of this hash, and representing this 64 bits in 16 characters.

credit-card-mask()

Description: Process the specified message field (by default, ${MESSAGE}), and replace the 7-12th character of any credit card numbers (Primary Account Number or PAN) with asterisks (\*). For example, AxoSyslog replaces the number 5542043004559005 with 554204******9005.

3 - Regular expressions

Filters and substitution rewrite rules can use regular expressions. In regular expressions, the characters ()[].*?+^$|\\ are used as special symbols. Depending on how you want to use these characters and which quotation mark you use, these characters must be used differently, as summarized below.

• Strings between single quotes ('string') are treated literally and are not interpreted at all, you do not have to escape special characters. For example, the output of '\\x41' is \\x41 (characters as follows: backslash, x(letter), 4(number), 1(number)). This makes writing and reading regular expressions much more simple: it is recommended to use single quotes when writing regular expressions.

• When enclosing strings between double-quotes ("string"), the string is interpreted and you have to escape special characters, that is, to precede them with a backslash (\\) character if they are meant literally. For example, the output of the "\\x41" is simply the letter a. Therefore special characters like \\(backslash) or "(quotation mark) must be escaped (\\\\ and \\"). The following expressions are interpreted: \\a, \\n, \\r, \\t, \\v. For example, the \\$40 expression matches the $40 string. Backslashes have to be escaped as well if they are meant literally, for example, the \\\\d expression matches the \\d string.

  Note If you use single quotes, you do not need to escape the backslash, for example, match("\\\\.") is equivalent to match('\\.').

• Enclosing alphanumeric strings between double-quotes ("string") is not necessary, you can just omit the double-quotes. for example, when writing filters, match("sometext") and match(sometext) will both match for the sometext string.

  Note

  Only strings containing alphanumerical characters can be used without quotes or double quotes. If the string contains whitespace or any special characters (()[].*?+^$|\\ or ;:#), you must use quotes or double quotes.

  When using the ;:# characters, you must use quotes or double quotes, but escaping them is not required.

By default, all regular expressions are case sensitive. To disable the case sensitivity of the expression, add the flags(ignore-case) option to the regular expression.

   filter demo_regexp_insensitive {
        host("system" flags(ignore-case));
    };

The regular expressions can use up to 255 regexp matches (${1} ... ${255}), but only from the last filter and only if the flags("store-matches") flag was set for the filter. For case-insensitive searches, use the flags("ignore-case") option.

3.1 - Options of regular expressions

This chapter lists regular expressions supported by AxoSyslog and their available supported type() and flags() options.

By default, AxoSyslog uses PCRE-style regular expressions. To use other expression types, add the type() option after the regular expression.

The AxoSyslog application supports the following regular expression type() options:

• Perl Compatible Regular Expressions (pcre)

• Literal string searches (string)

• Glob patterns without regular expression support (glob)

3.1.1 - The type() options of regular expressions

By default, AxoSyslog uses PCRE-style regular expressions, which are supported on every platform starting with AxoSyslog version 3.1. To use other expression types, add the type() option after the regular expression.

The AxoSyslog application supports the following type() options:

Perl Compatible Regular Expressions (pcre)

Description: Uses Perl Compatible Regular Expressions (PCRE). If the type() parameter is not specified, AxoSyslog uses PCRE regular expressions by default.

For more information about the flags() options of PCRE regular expressions, see The flags() options of regular expressions.

Literal string searches (string)

Description: Matches the strings literally, without regular expression support. By default, only identical strings are matched. For partial matches, use the flags("prefix") or the flags("substring") flags.

For more information about the flags() options of literal string searches, see The flags() options of regular expressions.

Glob patterns without regular expression support (glob)

Description: Matches the strings against a pattern containing * and ? wildcards, without regular expression and character range support. The advantage of glob patterns to regular expressions is that globs can be processed much faster.

• *: matches an arbitrary string, including an empty string

• ?: matches an arbitrary character

• The wildcards can match the / character.

• You cannot use the * and ? literally in the pattern.

3.1.2 - The flags() options of regular expressions

Similarly to the type() options, the flags() options are also optional within regular expressions.

The following list describes each type() option’s flags() options.

3.1.2.1 - Perl Compatible Regular Expressions (PCRE)

Starting with AxoSyslog version 3.1, PCRE expressions are supported on every platform. If the type() parameter is not specified, AxoSyslog uses PCRE regular expressions by default.

The following example shows the structure of PCRE-style regular expressions in use.

Example: Using PCRE regular expressions

   rewrite r_rewrite_subst {
        subst("a*", "?", value("MESSAGE") flags("utf8" "global"));  
    };

PCRE-style regular expressions have the following flags() options:

disable-jit

Switches off the just-in-time compilation function for PCRE regular expressions.

dupnames

Allows using duplicate names for named subpatterns.

Configuration example:

   filter { match("(?<DN>foo)|(?<DN>bar)" value(MSG) flags(store-matches, dupnames)); };
    ...
    destination { file(/dev/stdout template("$DN\n")); };

global

Usable only in rewrite rules, flags("global") matches for every occurrence of the expression, not only the first one.

ignore-case

Disables case-sensitivity.

newline

When configured, it changes the newline definition used in PCRE regular expressions to accept either of the following:

• a single carriage-return
• linefeed
• the sequence carriage-return and linefeed (\\r, \\n and \\r\\n, respectively)

This newline definition is used when the circumflex and dollar patterns (^ and $) are matched against an input. By default, PCRE interprets the linefeed character as indicating the end of a line. It does not affect the \\r, \\n or \\R characters used in patterns.

store-matches

Stores the matches of the regular expression into the $0, ... $255 variables. The $0 stores the entire match, $1 is the first group of the match (parentheses), and so on. Named matches (also called named subpatterns), for example, (?<name>...), are stored as well. Matches from the last filter expression can be referenced in regular expressions.

unicode

Use Unicode support for UTF-8 matches: UTF-8 character sequences are handled as single characters.

utf8

An alias for the unicode flag.

3.1.2.2 - Literal string searches

Literal string searches have the following flags() options:

global

Usable only in rewrite rules, flags("global") matches for every occurrence of the expression, not only the first one.

ignore-case

Disables case-sensitivity.

prefix

During the matching process, patterns (also called search expressions) are matched against the input string starting from the beginning of the input string, and the input string is matched only for the maximum character length of the pattern. The initial characters of the pattern and the input string must be identical in the exact same order, and the pattern’s length is definitive for the matching process (that is, if the pattern is longer than the input string, the match will fail).

Example: matching / non-matching patterns for the input string ’exam'

For the input string 'exam',

• the following patterns will match:

  • 'ex' (the pattern contains the initial characters of the input string in the exact same order)
  • 'exam' (the pattern is an exact match for the input string)

• the following patterns will not match:

  • 'example' (the pattern is longer than the input string)
  • 'hexameter' (the pattern’s initial characters do not match the input string’s characters in the exact same order, and the pattern is longer than the input string)

store-matches

Stores the matches of the regular expression into the $0, ... $255 variables. The $0 stores the entire match, $1 is the first group of the match (parentheses), and so on. Named matches (also called named subpatterns), for example, (?<name>...), are stored as well. Matches from the last filter expression can be referenced in regular expressions.

substring

The given literal string will match when the pattern is found within the input. Unlike flags("prefix"), the pattern does not have to be identical with the given literal string.

3.1.2.3 - Glob patterns without regular expression support

There are no supported flags() options for glob patterns without regular expression support.

3.2 - Optimizing regular expressions

The host(), match(), and program() filter functions and some other objects accept regular expressions as parameters. But evaluating general regular expressions puts a high load on the CPU, which can cause problems when the message traffic is very high. Often the regular expression can be replaced with simple filter functions and logical operators. Using simple filters and logical operators, the same effect can be achieved at a much lower CPU load.

Example: Optimizing regular expressions in filters

Suppose you need a filter that matches the following error message logged by the xntpd NTP daemon:

   xntpd[1567]: time error -1159.777379 is too large (set clock manually);

The following filter uses regular expressions and matches every instance and variant of this message.

   filter f_demo_regexp {
        program("demo_program") and
        match("time error .* is too large .* set clock manually");
    };

Segmenting the match() part of this filter into separate match() functions greatly improves the performance of the filter.

   filter f_demo_optimized_regexp {
        program("demo_program") and
        match("time error") and
        match("is too large") and
        match("set clock manually");
    };