Structuring macros, metadata, and other value-pairs

• 1: Specifying data types in value-pairs
• 2: value-pairs()

1 - Specifying data types in value-pairs

Prior to version 4.0, AxoSyslog handled every data as strings, and allowed you to convert the strings into other types of data that only certain destinations data formats supported. For example, SQL, MongoDB, JSON, or AMQP support data types like numbers or dates. The AxoSyslog application allows you to specify the data type in templates (this is also called type-hinting or type-casting). If the destination driver supports data types, AxoSyslog converts the incoming data to the specified data type. For example, this allows you to store integer numbers as numbers in MongoDB, instead of strings.

Starting with AxoSyslog 4.0, each name-value pair is a (name, type, value) triplet, and several components of AxoSyslog have typing support, for example, json-parser() and the $(format-json) template function. For details, see the list of supported data types.

Using explicit type-hinting

You can explicitly type-cast a AxoSyslog template to a specific type. To use type-hinting, enclose the macro or template containing the data with the type: <datatype>("<macro>"), for example: int("$PID"). See the Type-hinting examples and the list of supported data types for details.

Not every destination or other component supports data types. For details, see the list of AxoSyslog components that support data types.

Type-hinting examples

The following example stores the MESSAGE, PID, DATE, and PROGRAM fields of a log message in a MongoDB database. The DATE and PID parts are stored as numbers instead of strings.

   mongodb(
        value-pairs(pair("date", datetime("$UNIXTIME"))
            pair("pid", int64("$PID"))
            pair("program", "$PROGRAM"))
            pair("message", "$MESSAGE"))
        )
    );

The following example formats the same fields into JSON.

   $(format-json date=datetime($UNIXTIME) pid=int64($PID) program=$PROGRAM message=$MESSAGE)

The following example formats the MESSAGE field as a JSON list.

   $(format-json message=list($MESSAGE))"

Data types in AxoSyslog

The AxoSyslog application currently supports the following data-types.

• boolean: Converts the data to a boolean value. Anything that begins with a t or 1 is converted to true, anything that begins with an f or 0 is converted to false.
• datetime: Use it only with UNIX timestamps, anything else will likely result in an error. This means that currently you can use only the $UNIXTIME macro for this purpose.
• double: A floating-point number.
• json: A JSON snippet. (Available in AxoSyslog 4.0 and later.)
• list: The data as a list. For details, see the list manipulation template functions in Template functions of AxoSyslog.
• null: An unset value.
• integer: A 32-bit or 64-bit integer, determined by the destination. For example, mongodb uses int32 if the number is less than MAXINT32 and int64 otherwise.
• string: The data as a string.

Components that support data types

In AxoSyslog 4.0 and later, the following AxoSyslog components that support data types. Other components treat every data as strings.

• Comparisons in filter expressions: the previously numeric operators are type-aware. The exact comparison depends on the types associated with the values you compare. For details, see Comparing macro values in filters.

• json-parser() and the $(format-json) template function:

  When using the json-parser(), AxoSyslog converts all elements of the JSON object to name-value pairs. Any type information originally present in the incoming JSON object is retained, and automatically propagated to other AxoSyslog components (for example, a destination) if they support types.

  • Elements without a type are treated as strings.
  • JSON lists (arrays) are converted to AxoSyslog lists, so you can manipulate them using the $(list-*) template functions.

• set(), groupset() rewrite rules:

  You can set the type of the field. Where you can use of templates in set() and groupset(), you can use type-casting, and the type information is properly promoted. For details, see Specifying data types in value-pairs.

• db-parser(): The db-parser() rules can associate types with values using the "type" attribute, for example:

  <value name="foobar" type="integer">$PID</value>
  

  The integer is a type-cast that associates $foobar with an integer type. db-parser()’s internal parsers (for example, @NUMBER@) automatically associate type information to the parsed name-value pair.

• add-contextual-data(): Name-value pairs that are populated using add-contextual-data() propagate type information, similarly to db-parser().

• map-value-pairs(): map-value-pairs() propagates type information.

• SQL type support: The sql() driver supports types, so that columns with specific types are stored as those types.

• Template type support: You can cast templates explicitly to a specific type. Templates also propagate type information from macros, template functions, and values in the template string.

• python() typing: All Python components (sources, destinations, parsers, and template functions) support all data types, except json().

• On-disk serialized formats (that is, disk buffer): Version 4.0 and newer are compatible with messages serialized with an earlier version, and the format is compatible for downgrades as well. This means that even if a newer version of AxoSyslog serialized a message, older versions and associated tools are able to read it (but drop the type information of course).

2 - value-pairs()

Description: The value-pairs() option allows you to select specific information about a message easily using predefined macro groups. The selected information is represented as name-value pairs and can be used formatted to JSON format, or directly used in a mongodb() destination.

Example: Using the value-pairs() option

The following example selects every available information about the log message, except for the date-related macros (R_* and S_*), selects the .SDATA.meta.sequenceId macro, and defines a new value-pair called MSGHDR that contains the program name and PID of the application that sent the log message.

   value-pairs(
        scope(nv_pairs core syslog all_macros selected_macros everything)
        exclude("R_*")
        exclude("S_*")
        key(".SDATA.meta.sequenceId")
        pair("MSGHDR" "$PROGRAM[$PID]: ")
    )

The following example selects the same information as the previous example, but converts it into JSON format.

   $(format-json --scope nv_pairs,core,syslog,all_macros,selected_macros,everything \
        --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \
        --pair MSGHDR="$PROGRAM[$PID]: ")

value-pairs() parameters

The value-pairs() option has the following parameters. The parameters are evaluated in the following order:

1. scope()
2. exclude()
3. key()
4. pair()

exclude()

Description: This option removes the specified macros from the selection. Use it to remove unneeded macros selected using the scope() parameter. For example, the following example removes the SDATA macros from the selection.

   value-pairs(
        scope(rfc5424 selected_macros)
        exclude(".SDATA*")
    )

The name of the macro to remove can include wildcards (*, ?). Regular expressions are not supported.

key()

Description: This option selects the specified macros. The selected macros will be included as MACRONAME = MACROVALUE, that is using key("HOST") will result in HOST = $HOST. You can use wildcards (*, ?) to select multiple macros. For example:

   value-pairs(
        scope(rfc3164)
        key("HOST")
    )

   value-pairs(
        scope(rfc3164)
        key("HOST", "PROGRAM")
    )

omit-empty-values()

Description: If this option is specified, AxoSyslog does not include value-pairs with empty values in the output. For example: $(format-json --scope none --omit-empty-values) or

   value-pairs(
        scope(rfc3164 selected-macros)
        omit-empty-values()
    )

Available in AxoSyslog version 3.21 and later.

pair()

Description: This option defines a new name-value pair to be included in the message. The value part can include macros, templates, and template functions as well. For example:

   value-pairs(
        scope(rfc3164)
        pair("TIME" "$HOUR:$MIN")
        pair("MSGHDR" "$PROGRAM[$PID]: ")
    )

rekey()

Description: This option allows you to manipulate and modify the name of the value-pairs. You can define transformations, which are are applied to the selected name-value pairs. The first parameter of the rekey() option is a glob pattern that selects the name-value pairs to modify. If you omit the pattern, the transformations are applied to every key of the scope. For details on globs, see glob.

If you want to modify the names of several message fields, see also map-value-pairs: Rename value-pairs to normalize logs.

• If rekey() is used within a key() option, the name-value pairs specified in the glob of the key() option are transformed.
• If rekey() is used outside the key() option, every name-value pair of the scope() is transformed.

The following transformations are available:

• add-prefix("<my-prefix>")

  Adds the specified prefix to every name. For example, rekey( add-prefix("my-prefix."))

• replace-prefix("<prefix-to-replace>", "<new-prefix>")

  Replaces a substring at the beginning of the key with another string. Only prefixes can be replaced. For example, replace-prefix(".class", ".patterndb") changes the beginning tag .class to .patterndb

  This option was called replace() in AxoSyslog version 3.4.

• lower

  Convert all keys to lowercase. Only supports US ASCII.

• shift("<number>")

  Cuts the specified number of characters from the beginning of the name.

• shift-levels("<number>")

  Similar to –shift, but instead of cutting characters, it cuts dot-delimited “levels” in the name (including the initial dot). For example, --shift-levels 2 deletes the prefix up to the second dot in the name of the key: .iptables.SRC becomes SRC

• upper

  Convert all keys to uppercase. Only supports US ASCII.

Example: Using the rekey() option

The following sample selects every value-pair that begins with .cee., deletes this prefix by cutting 4 characters from the names, and adds a new prefix (events.).

   value-pairs(
        key(".cee.*"
            rekey(
                shift(4)
                add-prefix("events.")
            )
        )
    )

The rekey() option can be used with the format-json template-function as well, using the following syntax:

   $(format-json --rekey .cee.* --add-prefix events.)

scope()

Description: This option selects predefined groups of macros. The following groups are available:

• nv-pairs: Every soft macro (name-value pair) associated with the message, except the ones that start with a dot (.) character. Macros starting with a dot character are generated within AxoSyslog and are not originally part of the message, therefore are not included in this group.

• dot-nv-pairs: Every soft macro (name-value pair) associated with the message which starts with a dot (.) character. For example, .classifier.rule_id and .sdata.*. Macros starting with a dot character are generated within AxoSyslog and are not originally part of the message.

• all-nv-pairs: Include every soft macro (name-value pair). Equivalent to using both nv-pairs and dot-nv-pairs.

• rfc3164: The macros that correspond to the RFC3164 (legacy or BSD-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, and $DATE.

• rfc5424: The macros that correspond to the RFC5424 (IETF-syslog) message format: $FACILITY, $PRIORITY, $HOST, $PROGRAM, $PID, $MESSAGE, $MSGID, $R_DATE, and the metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA..

  The rfc5424 group also has the following alias: syslog-proto. Note that the value of $R_DATE will be listed under the DATE key.

  The rfc5424 group does not contain any metadata about the message, only information that was present in the original message. To include the most commonly used metadata (for example, the $SOURCEIP macro), use the selected-macros group instead.

• all-macros: Include every hard macro. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

• selected-macros: Include the macros of the rfc3164 groups, and the most commonly used metadata about the log message: the $TAGS, $SOURCEIP, and $SEQNUM macros.

• sdata: The metadata from the structured-data (SDATA) part of RFC5424-formatted messages, that is, every macro that starts with .SDATA.

• everything: Include every hard and soft macros. This group is mainly useful for debugging, as it contains redundant information (for example, the date-related macros include the date-related information several times in various formats).

• none: Reset previously added scopes, for example, to delete automatically-added name-value pairs. The following example deletes every value-pair from the scope, and adds only the ones starting with iptables: $(format-welf --scope none .iptables.*)

For example:

   value-pairs(
        scope(rfc3164 selected-macros)
    )