Template functions of AxoSyslog

The following template functions are available in AxoSyslog.

base64-encode

Syntax:

$(base64-encode argument)

Description: You can use the base64-encode template function to base64-encode strings and macros. The template function can receive multiple parameters (maximum 64). In this case, AxoSyslog joins the parameters into a single string and encodes this string. For example, $(base64-encode string1 string2) is equivalent to $(base64-encode string1string2).

Available in AxoSyslog version 3.18 and later.

basename

Syntax:

$(basename argument)

Description: Returns the filename from an argument (for example, a macro: $(basename ${FILE_NAME})) that contains a filename with a path. For example, $(basename "/var/log/messages.log") returns messages.log. To extract the path, use the dirname template function.

Available in AxoSyslog version 3.10 and later.

ceil

Syntax:

$(ceil argument)

Description: Rounds a floating-point number upwards to the nearest integer. For example, $(ceil 1.5) is 2, $(ceil -1.5) is -1. See also the floor and round template functions.

context-lookup

Syntax:

$(context-lookup [option] condition value-to-select)

Description: The context-lookup template function can search a message context when correlating messages (for example, when you use a pattern database or the grouping-by parser). The context-lookup template function requires a condition (a filter or a string), and returns a specific macro or template of the matching messages (for example, the ${MESSAGE}) as a list. It works similarly to the $(grep) template function, but it escapes its output properly, so that the returned value is a list that can be processed with other template functions that work on lists, for example, $(list-slice).

Example: Using the context-lookup template function

The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the tags name-value pair.

   $(context-lookup ("${username}" == "root") ${tags})

To limit the number of matches that the template function returns, use the --max-count option, for example, $(context-lookup --max-count 5 ("${username}" == "root") ${tags}). If you do not want to limit the number of matches, use --max-count 0.

You can to specify multiple name-value pairs as parameters, separated with commas. If multiple messages match the condition of context-lookup, these will be returned also separated by commas. This can be used for example, to collect the email recipients from postfix messages.

Available in AxoSyslog version 3.10 and later.

context-values

Syntax:

$(context-values $name-value1 $name-value2 ...)

Description: The context-values template function returns a list of every occurrence of the specified name-value pairs from the entire context. For example, if the context contains multiple messages, the $(context-values ${HOST}) template function will return a comma-separated list of the ${HOST} values that appear in the context.

Available in AxoSyslog version 3.10 and later.

dirname

Syntax:

$(dirname argument)

Description: Returns the path (without the filename) from an argument (for example, a macro: $(basename ${FILE_NAME}) that contains a filename with a path. For example, $(dirname "/var/log/messages.log") returns /var/log path. To extract the filename, use the basename template function.

Available in AxoSyslog version 3.10 and later.

echo

Syntax:

$(echo argument)

Description: Returns the value of its argument. Using $(echo ${HOST}) is equivalent to ${HOST}.

env

Syntax:

$(env <environment-variable>)

Description: Returns the value of the specified environment variable. Available in AxoSyslog 3.5 and later.

explode

Syntax:

$(explode <separator> <string1> <string2> ...)

Description: Turns a string separated by a specific character into a list. You can also use the implode template function, which turns a list into a string combining the pieces together with a separator. Available in AxoSyslog 3.21 and later.

Example: Using the explode template function

The following configuration example turns strings into a list. If there are several strings, AxoSyslog looks for a separator within each individual string. For example, string 2 is separated as string, 2 in the example below:

Enclose the strings in double-quotes or apostrophes and string 2 is separated as shown below:

The following examples replace the separator ';' character with a ',' character:

filter

Syntax: $(filter <filter-expression> <list>)

Description: Runs the filter expression on each element of a given list, and returns only those list elements that meet the requirements of the filter expression. The current value is referred by $_, similarly to the map template function.

Available in AxoSyslog version 3.30 and later.

Example: using the filter template function in your configuration

When used in configuration as seen in the example, the filter template function filters even numbers from an input list of 0, 1, 2 and 3:

log {
    source { example-msg-generator(num(1) values(INPUT => "0,1,2,3")); };
    destination {
        file("/dev/stdout"
            template("$(filter ('$(% $_ 2)' eq '0') $INPUT)\n)")
        );
    };
};

The returned values are 0 and 2.

Parameters

• <filter-expression>: Mandatory parameter. The <filter-expression> parameter can be:

  • a comparison
  • a filter
  • a logical expression built from filters (using and, or, and not)
  Note

  • When using the <filter-expression> parameter, you can refer other template functions, or use macros.
  • To refer to the variable bound to the current element of the list, use $_ macro.

  The following examples illustrate several ways that you can use a single filter, or a logical expression built from several filters.

      ('1' == '1')
      ('$_' le '1')
      ('$(% $_ 2)' eq '0')
      ('$_' le '1') and ('$(% $_ 2)' eq '0')
  

• list: Mandatory parameter. A AxoSyslog list.

format-cef-extension

AxoSyslog version 3.8 includes a new template function (format-cef-extension) to format name-value pairs as ArcSight Common Event Format extensions. Note that the template function only formats the selected name-value pairs, it does not provide any mapping. There is no special support for creating the prefix part of a Common Event Format (CEF) message. Note that the order of the elements is random. For details on the CEF extension escaping rules format, see the ArcSight Common Event Format.

You can use the value-pairs that AxoSyslog stores about the log message as CEF fields. Using value-pairs, you can:

• select which value-pairs to use as CEF fields,

• add custom value-pairs as CEF fields,

• rename value-pairs, and so on.

For details, see Structuring macros, metadata, and other value-pairs. Note that the syntax of format-* template functions is different from the syntax of value-pairs(): these template functions use a syntax similar to command lines.

Using the format-cef-extension template function has the following prerequisites:

• Set the on-error global option to drop-property, otherwise if the name of a name-value pair includes an invalid character, AxoSyslog drops the entire message. (Key name in CEF extensions can contain only the A-Z, a-z and 0-9 characters.)

      options {
         on-error("drop-property");
      };
  

• The log messages must be encoded in UTF-8. Use the encoding() option or the validate-utf8 flag in the message source.

Example: Using the format-cef-extension template function

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).

   $(format-cef-extension --scope syslog,all_macros,selected_macros \
      --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
      --pair MSGHDR=\"$PROGRAM[$PID]: \")

The following example selects every value-pair that has a name beginning with .cef., but removes the .cef. prefix from the key names.

   template("$(format-cef-extension --subkeys .cef.)\n")

The following example shows how to use this template function to store log messages in CEF format:

   destination d_cef_extension {
        file("/var/log/messages.cef" template("${ISODATE} ${HOST} $(format-cef-extension --scope selected_macros --scope nv_pairs)\n"));
    };

format-cim

Syntax: $(format-cim)

Description: Formats the message into Splunk Common Information Model (CIM) format. Applications that can receive messages in CIM format include Kibana, logstash, and Splunk. Applications that can be configured to log into CIM format include nflog and the Suricata IDS engine.

   destination d_cim {
        network(
            "192.168.1.1"
            template("$(format-cim)\n")
        );
    };

You can find the exact source of this template function in the AxoSyslog GitHub repository.

format-date

Syntax: $(format-date [options] format-string [timestamp])

Available in AxoSyslog 4.1 and later.

Description: The $(format-date) template function takes a timestamp in the DATETIME representation and formats it according to an strftime() format string. The DATETIME representation is a UNIX timestamp formatted as a decimal number, with an optional fractional part, where the seconds and the fraction of seconds are separated by a dot.

If the timestamp argument is missing, the timestamp of the message is used.

Options: --time-zone <TZstring>: Override timezone of the original timestamp

For example: $(format-date --time-zone PST8PDT %Y-%m-%dT%H:%M:%S 1667500613) corresponds to the following format and sets the timezone to Pacific Standard Time, Daylight Saving: “2022-11-03T11:36:53”

format-ewmm

Syntax: $(format-ewmm)

Description: The format-ewmm template function converts the message into the Enterprise-wide message model (EWMM) format. Available in version 3.16 and later.

The following is a sample log message in EWMM format.

   <13>1 2018-05-13T13:27:50.993+00:00 my-host @syslog-ng - - -
    {"MESSAGE":"<34>Oct 11 22:14:15 mymachine su: 'su root' failed for username on
    /dev/pts/8","HOST_FROM":"my-host","HOST":"my-host","FILE_NAME":"/tmp/in","._TAGS":".source.s_file"}

@include "scl.conf"

format-flat-json

Syntax: $(format-flat-json parameters)

Description: The format-flat-json template function is identical to the format-json template function, but nested JSON objects are flattened in the output. If you have to forward your log messages in JSON format, but the receiving application cannot handle nested JSON objects, use the format-flat-json template function.

Example: Flattened JSON output

The following example shows the difference between nested and flattened JSON objects.

• The output of $(format-json a.b.c=1) is a nested JSON object (whitespace added for better readability):

      {
          "a": {
              "b": {
              "c": "1"
              }
          }
      }
  

• The output of $(format-flat-json a.b.c=1) is a flattened JSON object (whitespace added for better readability):

      {
          "a.b.c": "1"
      }
  

For details on formatting log messages into JSON format, see format-json.

format-gelf

Syntax: $(format-gelf)

Description: Available in AxoSyslog 3.13 and later.

You can use the Graylog Extended Log Format (GELF) template together with the graylog2() destination to send syslog messages to Graylog. GELF is the native data format of Graylog.

Example: Using the format-gelf template function

The following configuration example shows how you can use the format-gelf template:

   destination graylog2 {
        network(
            "127.0.0.1"
            port(12201)
            transport(tcp)
            template("$(format-gelf)")
        );
    };

format-json

Syntax: $(format-json parameters)

Description: The format-json template function receives value-pairs as parameters and converts them into JavaScript Object Notation (JSON) format. Including the template function in a message template allows you to store selected information about a log message (that is, its content, macros, or other metadata) in JSON format. Note that the input log message does not have to be in JSON format to use format-json, you can reformat any incoming message as JSON.

You can use the value-pairs that AxoSyslog stores about the log message as JSON fields. Using value-pairs, you can:

• select which value-pairs to use as JSON fields,
• add custom value-pairs as JSON fields,
• rename value-pairs, and so on.

If the value-pair includes type information format-json can propagate it to the next component of the log path. For details, see Structuring macros, metadata, and other value-pairs. Note that the syntax of format-json is different from the syntax of value-pairs(): format-json uses a syntax similar to command lines.

Example: Using the format-json template function

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).

   $(format-json --scope syslog,all_macros,selected_macros
      --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId
      --pair MSGHDR="$PROGRAM[$PID]: ")

The following example shows how to use this template function to store log messages in JSON format:

   destination d_json {
        file(
            "/var/log/messages.json"
            template("$(format-json --scope selected_macros --scope nv_pairs)\n")
        );
    };

$(format-json --leave-initial-dot .SDATA.meta.sequenceID)

If you have to forward your log messages in JSON format, but the receiving application cannot handle nested JSON objects, use the format-flat-json template function. For details, see format-flat-json.

format-welf

This template function converts value-pairs into the WebTrends Enhanced Log file Format (WELF). The WELF format is a comma-separated list of name=value elements. Note that the order of the elements is random. If the value contains whitespace, it is enclosed in double-quotes, for example, name="value". For details on the WELF format, see https://www3.trustwave.com/support/kb/article.aspx?id=10899.

To select which value-pairs to convert, use the command-line syntax of the value-pairs() option. For details on selecting value-pairs, see value-pairs().

Example: Using the format-welf() template function

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message (since you will use the template-function in a template, you must escape the double-quotes).

   $(format-welf --scope syslog,all_macros,selected_macros
      --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId
      --pair MSGHDR="$PROGRAM[$PID]: ")

The following example shows how to use this template function to store log messages in WELF format:

   destination d_welf {
        file(
            "/var/log/messages.welf"
            template("$(format-welf --scope selected_macros --scope nv_pairs)\n")
        );
    };

geoip (DEPRECATED)

This template function is deprecated. Use geoip2 instead.

Syntax: $(geoip <IPv4-address>)

Description: This template function returns the 2-letter country code of any IPv4 address or host. IPv6 addresses are not supported. Currently only the 2-letter codes are supported, and only from the default database. For example, $(geoip $HOST)

To retrieve additional GeoIP information, see Looking up GeoIP data from IP addresses (DEPRECATED).

geoip2

Syntax:

```bash
$(geoip2 --database <path-to-geoip2-database-file>
    [ --field "registered_country.names.ru" ] ${HOST})
```

Description: This template function extracts specific fields from the mmdb database using the --field parameter. If you omit this parameter, it returns the 2-letter country code of any IPv4/IPv6 address or host.

To retrieve additional GeoIP information, see Looking up GeoIP2 data from IP addresses.

Starting with version 3.24, AxoSyslog tries to automatically detect the location of the database. If that is successful, the database() option is not mandatory.

getent

Syntax: $(getent)

Description: Available in AxoSyslog 3.13 and later.

You can use the getent template function to look up entries from the Name Service Switch libraries, such as, passwd, services, or protocols.

The following databases are supported:

• passwd

  Use this database to query data related to a user. Specify the user by either username or user ID. You can query the following data: username, user ID, group ID, GECOS field, home directory, or user shell.

      $(getent passwd testuser name)
      $(getent passwd testuser uid)
      $(getent passwd testuser gid)
      $(getent passwd testuser gecos)
      $(getent passwd testuser dir)
      $(getent passwd testuser shell)
  

  or

      $(getent passwd 1000 name)
      $(getent passwd 1000 uid)
      $(getent passwd 1000 gid)
      $(getent passwd 1000 gecos)
      $(getent passwd 1000 dir)
      $(getent passwd 1000 shell)
  

  The queried data is optional. When you do not query any data, the default behavior applies, which is as follows: user ID is returned for username, or username is returned for user ID.

  • Username $(getent passwd testuser) returns user ID 1000.

  • User ID $(getent passwd 1000) returns username testuser.

• group

  Use this database to query group-related data. The group can be specified using either group ID or group name. You can query the following data: group name, group ID, and members.

      $(getent group adm name)
      $(getent group adm gid)
      $(getent group adm members)
  

  The queried data is optional. The default behavior is as follows: group ID is returned for group name, or group name is returned for user ID.

  • Group name $(getent group adm) returns group ID 4.

  • Group ID $(getent group 4) returns group name adm.

• protocols

  Use this database to translate protocol name to protocol ID, or protocol ID to protocol string.

      $(getent protocols tcp)
      $(getent protocols 6)
  

• services

  Use this database to translate service name to service ID, or service ID to service name.

      $(getent services http)
      $(getent services 80)
  

graphite-output

Syntax: $(graphite-output parameters)

Description: Available in AxoSyslog 3.6 and later. This template function converts value-pairs from the incoming message to the Graphite plain text protocol format.

For details on selecting value-pairs in AxoSyslog and for possibilities to specify which information to convert to Graphite plain text protocol format, see Structuring macros, metadata, and other value-pairs. Note that the syntax of graphite-output is different from the syntax of value-pairs(): graphite-output uses a the command-line syntax used in the format-json template function.

Example: Using the graphite-output template function

The following configuration example shows, how to send value-pairs with names starting with “vmstat.” to Graphite running on localhost, port 2003:

   destination d_graphite {
        network( host("localhost") port(2003) template("$(graphite-output --key vmstat.*)"));
    };

grep

Syntax: $(grep condition value-to-select)

Description: The grep template function can search a message context when correlating messages (for example, when you use a pattern database or the grouping-by parser). The context-lookup template function requires a condition (a filter or a string), and returns a specific macro or template of the matching message (for example, the ${MESSAGE} field of the message).

Example: Using the grep template function

The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the auth_method name-value pair.

   $(grep ("${username}" == "root") ${auth_method})

You can to specify multiple name-value pairs as parameters, separated with commas. If multiple messages match the condition of grep, these will be returned also separated by commas. This can be used for example, to collect the email recipients from postfix messages.

hash

Syntax: $(<method> [opts] $arg1 $arg2 $arg3...)

Options: --length N, -l N

Truncate the hash to the first N characters.

Description: Calculates a hash of the string or macro received as argument using the specified hashing method. If you specify multiple arguments, effectively you receive the hash of the first argument salted with the subsequent arguments.

<method> can be one of md5, md4, sha1, sha256, sha512 and “hash”, which is equivalent to md5. Macros are expected as arguments, and they are concatenated without the use of additional characters.

The md4 <method> is deprecated.

This template function can be used for anonymizing sensitive parts of the log message (for example, username) that were parsed out using PatternDB before storing or forwarding the message. This way, the ability of correlating messages along this value is retained.

Also, using this template, quasi-unique IDs can be generated for data, using the --length option. This way, IDs will be shorter than a regular hash, but there is a very small possibility of them not being as unique as a non-truncated hash.

Example: Using the $(hash) template function

The following example calculates the SHA1 hash of the hostname of the message:

   $(sha1 $HOST)

The following example calculates the SHA256 hash of the hostname, using the salted string to salt the result:

   $(sha1 $HOST salted)

To use shorter hashes, set the --length:

   $(sha1 --length 6 $HOST)

To replace the hostname with its hash, use a rewrite rule:

   rewrite r_rewrite_hostname{set("$(sha1 $HOST)", value("HOST"));};

Example: Anonymizing IP addresses

The following example replaces every IPv4 address in the MESSAGE part with its SHA-1 hash:

   rewrite pseudonymize_ip_addresses_in_message {subst ("((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])[.]){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))", "$(sha1 $0)", value("MESSAGE"));};

if

Syntax: $(if (<condition>) <true template> <false template>)

Description: Returns the value of the <true template> parameter if the <condition> is true. If the <condition> is false, the value of <false template> is returned.

Example: Using pattern databases and the if template function

The following example returns violation if the username name-value pair of a message is root, and system otherwise.

   $(if ("${username}" == "root") "violation" "system")

This can be used to set the class of a message in pattern database rules based on the condition.

   <value name="username">$(if ("${username}" == "root") "violation" "system")</value>

Since template functions can be embedded into each other, it is possible to use another template function as the template of the first one. For example, the following expression returns root if the username is root, admin if the username is joe, and normal user otherwise.

   <value name="username">
        $(if ("${username}" == "root")
            "root"
            $(if ("${username}" == "joe") "admin" "normal user"))</value>

implode

Syntax: $(implode <separator> <string1>, <string2>, ...)

Description: Turns a list into a string combining the pieces together with a separator. You can also use the explode template function, which turns a string separated by a specific character into a list. Available in AxoSyslog 3.21 and later.

Example: Using the implode template function

The following configuration example shows how you can use the implode template to turn a list into a string:

You can also use a $(list-*) template function to further manipulate the list. The following example returns the first three elements of the list:

indent-multi-line

Syntax: $(indent-multi-line parameter)

Description: This template function makes it possible to write multi-line log messages into a file. The first line is written like a regular message, subsequent lines are indented with a tab, in compliance with RFC822.

Example: Using the indent-multi-line template function

The following example writes multi-line messages into a text file.

   destination d_file {
        file (
            "/var/log/messages"
            template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n")
        );
    };

ipv4-to-int

Syntax: $(ipv4-to-int parameter)

Description: Converts the specified IPv4 address to its numeric representation. The numerical value of an IPv4 address is calculated by treating the IP address as a 4-byte hexadecimal value. For example, the 192.168.1.1 address equals to: 192=C0, 168=A8, 1=01, 1=01, or C0A80101, which is 3232235777 in decimal representation.

List manipulation

The list-* template functions allow you to manipulate comma-separated lists. Such lists represent a simple array type in AxoSyslog. Note the following about formatting lists:

• Values are separated by commas, for example, "item1","item2","item3". The single-element list is an element without a comma.

• You can use shell-like quotation to embed commas, for example, "item1","ite\\,m2","item3".

• Empty values are skipped (except if they are quoted)

These template functions return a well-formed list, properly encoding and quoting all elements. If a template function returns a single element, all quotation is decoded and the value contains the literal value.

Starting with AxoSyslog version 3.10, the following list-related template functions are available. Certain functions allow you to reference an element using its number: note that the list index starts with zero, so the index of the first element is 0, the second element is 1, and so on.

list-append

Syntax: $(list-append ${list} ${name-value-pair1} ${name-value-pair2} ... )

Description: Returns a list and appends the values of the specified name-value pairs to the end of the list. You can also append elements to an empty list, for example, $(list-append '' 'element-to-add')

list-concat

Syntax: $(list-concat ${name-value-pair1} ${name-value-pair2} ... )

The commas between the parameters are optional.

Description: This template function creates (concatenates) a list of the values it receives as parameter. The values can be single values (for example, ${HOST}) or lists.

For example, the value of the $(list-concat ${HOST}, ${PROGRAM}, ${PID}) is a comma-separated list.

You can concatenate existing lists into a single list using:

   $(list-concat ${list1} ${list2})

list-count

Syntax: $(list-count ${list} )

Description: Returns the number of elements in the list.

list-head

Syntax: $(list-head ${list} )

Description: Returns the first element of the list, unquoted.

list-nth

Syntax: $(list-nth <index-number> ${list} )

Description: Returns the nth element of the list, unquoted. Note that the list index starts with zero, so (list-nth 1 ${list} ) returns the second element, and so on.

list-search

Syntax: $(list-search [OPTIONS] <pattern> ${list})

Description: The list-search template function searches the elements of ${list} starting at the specified start_index, then returns the index of the first match of <pattern> within ${list}`.

Options:

• --mode MODE: Matching mode, with the following possible values: literal (default), prefix, substring, glob, pcre
• --start-index N: Skips N elements in the ${list}

list-slice

Syntax: $(list-slice <from>:<to> ${list} )

Description: Returns the specified subset of the list. Note that the list index starts with zero, for example, $(list-slice 1:2 ${list} ) returns the second and third element of the list, and so on.

You can omit the from or to index if you want to start the subset from the beginning or end of the list, for example: 3: returns the list starting with the 4th element, while :3 returns the first four elements.

Negative numbers select an element from the end of the list, for example, -3: returns the last three element of the list.

list-tail

Syntax: $(list-tail ${list} )

Description: Returns the list without the first element. For example, if the ${mylist} list contains the one, two, three elements, then $(list-tail ${mylist} ) returns two, three.

length

Syntax: $(length "<macro>")

Description: Returns the length of the macro in characters, for example, the length of the message. For example, the following filter selects messages that are shorter than 16 characters:

   f_short {
        match ('-', value ("$(if ($(length "${MESSAGE}") <= 16) "-" "+")"));
    };

lowercase

Syntax: $(lowercase "<macro>")

Description: Returns the lowercase version of the specified string or macro. For example, the following example uses the lowercase version of the hostname in a directory name:

   destination d_file {
        file ("/var/log/${MONTH}/${DAY}/$(lowercase "${HOST}")/messages");
    };

Available in AxoSyslog 3.5 and later.

map

Syntax: $(map template list)

Description: Returns with a list that contains the results of applying a template function for each elements of a list.

Available in AxoSyslog version 3.28 and later.

Parameters:

• template: Mandatory. This template function is applied for each elements of the list. Use $_ macro to refer to the current list element.

• list: Mandatory. A list, or template.

Example:

When used in configuration as seen in the example, the map template function adds one to each element of a list:

   log {
      source { example-msg-generator(num(1) values(LST => "0,1,2")); };
      destination {
        file("/dev/stdout"template('$(map "$(+ 1 $_)" $LST)')
        );
      };
    };

The returned values are 1, 2, and 3.

Numerical operations

Syntax: $(<operation> "<value1>" "<value2>")

Description: These template functions allow you to manipulate numbers, that is, to perform addition (+), substraction (-), multiplication (*), division (/), and modulus (%). All of them require two numeric arguments. The result is NaN (Not-a-Number) if the parameters are not numbers, cannot be parsed, or if a division by zero would occur. For example, to add the value of two macros, use the following template function:

   $(+ "${<MACRO1>}" "${<MACRO2>}");

Starting with AxoSyslog version 3.22 and later, the numerical operators support floating-point values. They behave like the operators in the C programming language:

• If both operands are integers, they return an integer.

• If any of the operands is a floating-point number, they return a floating-point result.

For example:

   $(/ 3 2) # Both operands are integers, so the result is 1
    # One of the operands is a floating point number, so the result is also floating-point
    $(/ 3.0 2) # = 1.500000
    $(/ 3 2.0) # = 1.500000
    $(/ 3.0 2.0) # = 1.500000

To round floating-point numbers, you can use the ceil, floor, and round template functions.

When you are correlating messages and a name-value pair contains numerical values in the messages, you can calculate the lowest (min), highest (max), total (sum), and mean (average) values. These calculations process every message of the correlation context. For details on message correlation, see Correlating log messages. For example, if the messages of the context have a .myfields.load name-value pair, you can find the highest load value using the following template function.

   $(max ${.myfields.load})

or

Syntax: $(or <macro1> <macro2>)

Description: This template function returns the first non-empty argument.

padding

Syntax:

 $(padding <macro> <width> <prepended-character-or-string>)

Description: This template function returns the value of its first parameter (a string or macro), prepended with a string. This string is <width> long, and repeats the character or string set in the third parameter. If you use a single character, it is added >times. If you use a string, it is repeated until its length reaches <width>>The default padding character is ’ ’ (space). For example:

Example: Using the padding template function

If the value of the ${MESSAGE} macro is mymessage, then the output of the padding() template function is the following:

   $(padding ${MESSAGE} 10 X)

Output: XXXXXXXXXXmymessage

   $(padding ${MESSAGE} 10 foo)

Output: foofoofoofmymessage

python

Syntax:

 $(python <name-of-the-python-method-to-use> <arguments-of-the-method>)

Description: This template function enables you to write a custom template function in Python. You can define a Python block in your AxoSyslog configuration file, define one or more Python functions in it, and use the methods as template functions. If you use a Python block, AxoSyslog embeds a Python interpreter to process the messages.

The following points apply to using Python blocks in AxoSyslog in general:

• Python parsers and template functions are available in AxoSyslog version 3.10 and later.

  Python destinations and sources are available in AxoSyslog version 3.18 and later.

• Supported Python versions: 2.7 and 3.4+ (if you are using pre-built binaries, check the dependencies of the package to find out which Python version it was compiled with).

• The Python block must be a top-level block in the AxoSyslog configuration file.

• If you store the Python code in a separate Python file and only include it in the AxoSyslog configuration file, make sure that the PYTHONPATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start AxoSyslog manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc.

  In production, when AxoSyslog starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting AxoSyslog. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc".

• The Python object is initiated every time when AxoSyslog is started or reloaded.

  Warning If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.

• The Python block can contain multiple Python functions.

• Using Python code in AxoSyslog can significantly decrease the performance of AxoSyslog, especially if the Python code is slow. In general, the features of AxoSyslog are implemented in C, and are faster than implementations of the same or similar features in Python.

• Validate and lint the Python code before using it. The AxoSyslog application does not do any of this.

• Python error messages are available in the internal() source of AxoSyslog.

• You can access the name-value pairs of AxoSyslog directly through a message object or a dictionary.

• To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of AxoSyslog. For details, see Logging from your Python code.

The following points apply to Python parsers.

• The first argument in the definition of the Python function is the actual log message. This is implicitly passed to the function, you do not have to use it in the template function.

• The value of the template function is return value of the Python function.

• To reference a name-value pair or a macro in the Python function, use the dot-notation. For example, if the first argument in the definition of the function is called log-message, the value of the HOST macro is log-message.HOST, and so on.

• You can define new name-value pairs in the Python function. For example, if the first argument in the definition of the function is called log-message, you can create a new name-value pair like this: log_message["new-macro-name"]="value". This is useful when you parse a part of the message from Python, or lookup a value based on data extracted from the log message.

Declaration:

   python {
    def <name_of_the_python_function>(<log_message>, <optional_other_arguments>):
        # <your-python-code>
        return <value_of_the_template_function>
    };
    
    template <template-name> {
        template($(python <name_of_the_python_function>));
    };

Example: Writing template functions in Python

The following example creates a Python template function called return_message that returns the MESSAGE part of the log message.

   @version: 4.9.0
    
    python {
    def return_message(log_message):
        return log_message.MESSAGE
    };
    
    destination d_local {
        file("/tmp/logs.txt" template("[$(python return_message)]\n"));
    };

The following example creates a Python template function called resolve_host that receives an IP address as an argument, and attempts to resolve it into a hostname.

   @version: 4.9.0
    
    python {
    import socket
    
    def resolve_host(log_message, hostname):
        try:
            return socket.gethostbyaddr(hostname)[0]
        except (socket.herror, socket.error):
            return 'unknown'
    };
    
    destination d_local {
        file(
            "/tmp/logs.txt"
            template("${ISODATE} $(python resolve_host ${SOURCE_IP}) ${MESSAGE}\n")
        );
    };

replace-delimiter

Syntax: $(replace-delimiter "<old-delimiters>" "<new-delimiter>" "<macro>")

Description: Replaces the delimiter character with a new one. For example, the following example replaces the tabulators (\\t) in the message with semicolons (;):

   $(replace-delimiter "\t" ";" "${MESSAGE}")

Available in AxoSyslog 3.5 and later.

round

Syntax: $(round argument)

Description: Rounds a floating-point number to the nearest integer. For example, $(round 1.5) is 2. See also the ceil and floor template functions.

This template function has an optional second argument that sets the precision of rounding. The default is 0 (output a natural number), but values up to 20 are accepted. For example, $(round 2.123456 4) is 2.1235.

sanitize

Syntax: $(sanitize <options> "<macro1>" "<macro2> ...")

Description: This file replaces the special characters in macro values, for example, it can replace the slash (/) characters in a filename with the underscore (_) character. If you specify multiple arguments, they will be concatenated using the / character, so they can be used as separate directory levels when used in filenames.

The function has the following options:

• --ctrl-chars or -c

• Filter control characters (characters that have an ASCII code of 32 or lower). This option is used by default.

• --invalid-chars <characterlist> or -i <characterlist>

• The list of characters to be replaced with underscores (_). The default list contains the / character. The following example replaces the \ and @ characters, so for example, fo\o@bar becomes foobar:

      $(sanitize -i \@ $PROGRAM)
  

• --no-ctrl-chars or -C

• Do not filter the control characters (characters that have an ASCII code of 32 or lower).

• --replacement <replacement-character> or -r <replacement-character>

• The character used to replace invalid characters. By default, this is the underscore (_). The following example replaces invalid characters with colons instead of underscores, so for example, foo/bar becomes foo;bar:

      $(sanitize -r ; $PROGRAM)
  

Example: Using the sanitize template function

The following example uses the sanitize function on two macros, and the results are used as directory names in a file destination.

   file("/var/log/$(sanitize $HOST $PROGRAM)/messages");

This is equivalent to file("/var/log/$HOST/$PROGRAM/messages");, but any slashes in the values of the $HOST and $PROGRAM macros are replaced with underscores.

stardate

Syntax: $(stardate [option] "<date-in-unixtime>")

Description: Converts a date in UNIXTIME (for example, ${UNIXTIME}) into stardate, displaying the year and the progress of the year in a number of digits (YYYY.NNN). You can set the number of digits using the --digits option, for example:

   $(stardate --digits 2 "${R_UNIXTIME}")

strip

Syntax: $(strip "<macro>")

Description: Deletes whitespaces from the beginning and the end of a macro. You can specify multiple macros separated with whitespace in a single template function, for example:

   $(strip "${MESSAGE}" "${PROGRAM}")

substr

Syntax: $(substr "<argument>" "<offset>" "<length>")

Description: This function extracts a substring of a string.

• argument The string to extract the substring from, for example, "${MESSAGE}"

• offset Specifies where the substring begins (in characters). 0 means to start from the beginning of the string, 5 means to skip the first 5 characters of the string, and so on. Use negative numbers to specify where to start from the end of the string, for example, -1 means the last character, -5 means to start five characters before the end of the string.

• length Optional parameter: The number of characters to extract. If not specified, the substring will be extracted from the offset to the end of the string. Use negative numbers to stop the substring before the end of the string, for example, -5 means the substring ends five characters before the end of the string.

Example: Using the substr template function

Skip the first 15 characters of the message, and select the rest:

   $(substr "${MESSAGE}" "15");

Select characters 16-30 of the message (15 characters with offset 15):

   $(substr "${MESSAGE}" "15" "15");

Select the last 15 characters of the message:

   $(substr "${MESSAGE}" "-15");

A template that converts the message to RFC3164 (BSD-syslog) format and truncates the messages to 1023 characters:

   template t_truncate_messages {
        template("$(substr \"<$PRI>$DATE $HOST $MSGHDR$MESSAGE\" \"0\" \"1023\")\n");
        template-escape(no);
    };

tag

Available in AxoSyslog version 4.6 and later.

Syntax:

$(tag <name-of-the-tag> <value-if-set> <value-if-unset>)

Description: Adds bit-like tags to the messages.

• If you do not set the value-if-set and value-if-unset arguments, the $(tag) template function acts as a boolean and expands to 0 or 1, depending on whether the message has the specified tag set.
• If the value-if-set and value-if-unset arguments are set, $(tag) returns a string: the second argument (<value-if-set>) if the message has <tag>, and the third argument (<value-if-unset>) if the message doesn’t have <tag>.

tags-head

Available in AxoSyslog version 4.7 and later.

Syntax:

$(tags-head <name-of-first-tag> <name-of-second-tag> ... )

Description: This template function accepts multiple tag names, and returns the first one that is set.

Example config:

# resolves to "bar" if "bar" tag is set, but "foo" is not
template("$(tags-head foo bar baz)")

template

Syntax:

$(template <template-name>)
$(template $<dynamic-template-name>)
$(template $<dynamic-template-name> '<optional-fallback-template>')

Description: This template function looks up the in the configuration and uses that to format its result. The referenced template can be static or dynamic. For static templates, AxoSyslog resolves the template when it starts, or when the configuration is reloaded. For dynamic templates, the results are resolved runtime (for dynamic templates, the template name contains at least one ‘$’ character). For example, the name of the template to be invoked can be extracted from the message, or from a name-value pair set using the add-contextual-data() feature.

For dynamic templates, you can set an optional second template. This second template will be the results of the template function if resolving the dynamic template fails for some reason. For example:

   $(template ${my-dynamic-template} '$DATE $HOST $MSGHDR$MSG\n')

Available in AxoSyslog 3.22 and later.

uppercase

Syntax: $(uppercase "<macro>")

Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:

   destination d_file {
        file ("/var/log/${MONTH}/${DAY}/$(uppercase ${HOST})/messages");
    };

Available in AxoSyslog 3.5 and later.

url-decode

Syntax: $(url-decode <string-pr-macro-1> <string-pr-macro-2> ... )

Description: You can use the url-decode template function to decode url-encoded strings and macros. For example, $(url-decode %3C%3E) yields <>. The url-decode` can receive multiple parameters (maximum 64). In this case, each parameter is decoded separately, and simply concatenated.

Available in AxoSyslog version 3.18 and later.

url-encode

Syntax: $(url-encode ${MESSAGE})

Description: The url-encode template function escapes strings for use as URI path or query parameter segments or form encoded HTTP POST data. All input characters that are not a-z, A-Z, 0-9, ‘-’, ‘.’, ‘_’ or ‘~’ are converted to their “URL escaped” version.

Available in AxoSyslog version 3.18 and later. (In version 3.16-3.17, this template function was called urlencode.)

uuid

Syntax: $(uuid)

Description: Generates a Universally Unique IDentifier (UUID) that complies with RFC4122. That way, an UUID can be added to the message soon after it is received, so messages stored in multiple destinations can be identified. For example, when storing messages in a database and also in files, the UUID can be used to find a particular message both in the database and the files.

To generate a UUID, you can use a rewrite rule to create a new value-pair for the message.

Example: Using Universally Unique Identifiers

The following example adds a value-pair called MESSAGE_UUID to the message using a rewrite rule and a template.

   rewrite r_add_uuid {
        set("$(uuid)" value("MESSAGE_UUID"));
    };
    
    destination d_file {
        file (
            "/var/log/messages"
            template("$MESSAGE_UUID $ISODATE $HOST $MSG\n")
            template-escape(no)
        );
    };
    
    log {
        source(s_network);
        rewrite(r_add_uuid);
        destination(d_file);
    };