This is the multi-page printable view of this section.
Click here to print.
Return to the regular view of this page.
parser: Parse and segment structured messages
The filters and default macros of AxoSyslog work well on the headers and metainformation of the log messages, but are rather limited when processing the content of the messages. Parsers can segment the content of the messages into name-value pairs, and these names can be used as user-defined macros. Subsequent filtering or other type of processing of the message can use these custom macros to refer to parts of the message. Parsers are global objects most often used together with filters and rewrite rules.
The AxoSyslog application provides the following possibilities to parse the messages, or parts of the messages, as shown on the following list. There are several built-in parsers for application-specific logs.
Note that by default, AxoSyslog parses every message as a syslog message. To disable parsing the message as a syslog message, use the flags(no-parse) option of the source. To explicitly parse a message as a syslog message, use the syslog parser. For details, see Parsing syslog messages.
1 - Apache access log parser
The Apache access log parser can parse the access log messages of the Apache HTTP Server. The AxoSyslog application can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The apache-accesslog-parser() supports both the Common Log Format and the Combined Log Format of Apache (for details, see the Apache HTTP Server documentation). The following is a sample log message:
127.0.0.1 - frank [10/Oct/2000:13:55:36 -0700] "GET /apache_pb.gif HTTP/1.0" 200 2326
Starting with version 3.21, virtualhost and the port of the virtualhost (vhost) is also supported, for example:
foo.com:443 1.2.3.4 - - [15/Apr/2019:14:30:16 -0400] "GET /bar.html HTTP/2.0" 500 - "https://foo.com/referer.html" "Mozilla/5.0 ..."
The AxoSyslog application extracts every field into name-value pairs, and adds the .apache. prefix to the name of the field.
Declaration:
parser parser_name {
apache-accesslog-parser(
prefix()
);
};
The parser extracts the following fields from the messages: vhost, port, clientip, ident, auth, timestamp, rawrequest, response, bytes, referrer, and agent. The rawrequest field is further segmented into the verb, request, and httpversion fields. The AxoSyslog apache-accesslog-parser() parser uses the same naming convention as Logstash.
Example: Using the apache-accesslog-parser parser
In the following example, the source is a log file created by an Apache web server. The parser automatically inserts the .apache. prefix before all extracted name-value pairs. The destination is a file that uses the format-json template function. Every name-value pair that begins with a dot (.) character will be written to the file (dot-nv-pairs). The log statement connects the source, the destination, and the parser.
source s_apache {
file(/var/log/access_log);
};
destination d_json {
file(
"/tmp/test.json"
template("$(format-json .apache.*)\n")
);
};
log {
source(s_apache);
parser { apache-accesslog-parser();};
destination(d_json);
};
To use this parser, the scl.conf file must be included in your AxoSyslog configuration:
The apache-accesslog-parser() is actually a reusable configuration snippet configured parse Apache access log messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
1.1 - Options of apache-accesslog-parser() parsers
The apache-accesslog-parser() has the following options.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, apache-accesslog-parser() uses the .apache. prefix. To modify it, use the following format:
parser {
apache-accesslog-parser(prefix("apache."));
};
template()
|
|
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
2 - Check Point Log Exporter parser
The Check Point Log Exporter parser can parse Check Point log messages. These messages do not completely comply with the syslog RFCs, making them difficult to parse. The checkpoint-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The parser can parse messages in the following formats:
<PRI><VERSION> <YYYY-MM-DD> <HH-MM-SS> <PROGRAM> <PID> <MSGID> - [key1:value1; key2:value2; ... ]
For example:
<134>1 2018-03-21 17:25:25 MDS-72 CheckPoint 13752 - [action:"Update"; flags:"150784"; ifdir:"inbound"; logid:"160571424"; loguid:"{0x5ab27965,0x0,0x5b20a8c0,0x7d5707b6}";]
Splunk format:
time=1557767758|hostname=r80test|product=Firewall|layer_name=Network|layer_uuid=c0264a80-1832-4fce-8a90-d0849dc4ba33|match_id=1|parent_rule=0|rule_action=Accept|rule_uid=4420bdc0-19f3-4a3e-8954-03b742cd3aee|action=Accept|ifdir=inbound|ifname=eth0|logid=0|loguid={0x5cd9a64e,0x0,0x5060a8c0,0xc0000001}|origin=192.168.96.80|originsicname=cn\=cp_mgmt,o\=r80test..ymydp2|sequencenum=1|time=1557767758|version=5|dst=192.168.96.80|inzone=Internal|outzone=Local|proto=6|s_port=63945|service=443|service_id=https|src=192.168.96.27|
If you find a message that the checkpoint-parser() cannot properly parse, contact us, so we can improve the parser.
By default, the Check Point-specific fields are extracted into name-value pairs prefixed with .checkpoint. For example, the action in the previous message becomes ${.checkpoint.action}. You can change the prefix using the prefix option of the parser.
Declaration:
@version: 4.9.0
@include "scl.conf"
log {
source { network(flags(no-parse)); };
parser { checkpoint-parser(); };
destination { ... };
};
Note that the parser expects that the entire incorrectly formatted syslog message (starting with its <PRI> value) is in $MSG, which you can achieve by using flags(no-parse) on the input driver.
The checkpoint-parser() is actually a reusable configuration snippet configured to parse Check Point messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, checkpoint-parser() uses the .checkpoint. prefix. To modify it, use the following format:
parser {
checkpoint-parser(prefix("myprefix."));
};
3 - Cisco parser
The Cisco parser can parse the log messages of various Cisco devices. The messages of these devices often do not completely comply with the syslog RFCs, making them difficult to parse. The cisco-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs, extracting also the Cisco-specific values, for example, the mnemonic. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The parser can parse variations of the following message format:
<pri>(sequence: )?(origin-id: )?(timestamp? timezone?: )?%msg
For example:
<189>29: foo: *Apr 29 13:58:40.411: %SYS-5-CONFIG_I: Configured from console by console
<190>30: foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 stopped - CLI initiated
<190>31: foo: *Apr 29 13:58:46.411: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 192.168.1.239 started - CLI initiated
<189>32: 0.0.0.0: *Apr 29 13:59:12.491: %SYS-5-CONFIG_I: Configured from console by console
<189>32: foo: *Apr 29 13:58:46.411: %SYSMGR-STANDBY-3-SHUTDOWN_START: The System Manager has started the shutdown procedure.
Note
Not every Cisco log message conforms to this format. If you find a message that the
cisco-parser() cannot properly parse,
contact us, so we can improve the parser.
The AxoSyslog application normalizes the parsed log messages into the following format:
${MESSAGE}=%FAC-SEV-MNEMONIC: message
${HOST}=origin-id
By default, the Cisco-specific fields are extracted into the following name-value pairs:${.cisco.facility}, ${.cisco.severity}, ${.cisco.mnemonic}. You can change the prefix using the prefix option.
Declaration:
@version: 4.9.0
@include "scl.conf"
log {
source { network(
transport("udp")
flags(no-parse)
); };
parser { cisco-parser(); };
destination { ... };
};
Note that you have to disable message parsing in the source using the flags(no-parse) option for the parser to work.
The cisco-parser() is actually a reusable configuration snippet configured to parse Cisco messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, cisco-parser() uses the .cisco. prefix. To modify it, use the following format:
parser {
cisco-parser(prefix("myprefix."));
};
4 - Parsing messages with comma-separated and similar values
The AxoSyslog application can separate parts of log messages (that is, the contents of the ${MESSAGE} macro) at delimiter characters or strings to named fields (columns) using the csv (comma-separated-values) parser The parsed fields act as user-defined macros that can be referenced in message templates, file- and tablenames, and so on.
Parsers are similar to filters: they must be defined in the AxoSyslog configuration file and used in the log statement. You can also define the parser inline in the log path.
Note
The order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.
To create a csv-parser(), you have to define the columns of the message, the separator characters or strings (also called delimiters, for example, semicolon or tabulator), and optionally the characters that are used to escape the delimiter characters (quote-pairs()).
Declaration:
parser <parser_name> {
csv-parser(
columns(column1, column2, ...)
delimiters(chars("<delimiter_characters>"), strings("<delimiter_strings>"))
);
};
Column names work like macros.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
Starting with AxoSyslog version 4.5, you can omit the columns() option, and extract the values into matches ($1, $2, $3, and so on), which are available as the anonymous list $*. For example:
@version: current
log {
source { tcp(port(2000) flags(no-parse)); };
parser { csv-parser(delimiters(',') dialect(escape-backslash)); };
destination { stdout(template("$ISODATE $*\n")); };
};
Example: Segmenting hostnames separated with a dash
The following example separates hostnames like example-1 and example-2 into two parts.
parser p_hostname_segmentation {
csv-parser(columns("HOSTNAME.NAME", "HOSTNAME.ID")
delimiters("-")
flags(escape-none)
template("${HOST}"));
};
destination d_file {
file("/var/log/messages-${HOSTNAME.NAME:-examplehost}");
};
log {
source(s_local);
parser(p_hostname_segmentation);
destination(d_file);
};
Example: Parsing Apache log files
The following parser processes the log of Apache web servers and separates them into different fields. Apache log messages can be formatted like:
"%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T %v"
Here is a sample message:
192.168.1.1 - - [31/Dec/2007:00:17:10 +0100] "GET /cgi-bin/example.cgi HTTP/1.1" 200 2708 "-" "curl/7.15.5 (i4 86-pc-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8c zlib/1.2.3 libidn/0.6.5" 2 example.mycompany
To parse such logs, the delimiter character is set to a single whitespace (delimiters(" ")). Whitespaces between quotes and brackets are ignored (quote-pairs('""[]')).
parser p_apache {
csv-parser(
columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME",
"APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS",
"APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT",
"APACHE.PROCESS_TIME", "APACHE.SERVER_NAME")
flags(escape-double-char,strip-whitespace)
delimiters(" ")
quote-pairs('""[]')
);
};
The results can be used for example, to separate log messages into different files based on the APACHE.USER_NAME field. If the field is empty, the nouser name is assigned.
log {
source(s_local);
parser(p_apache);
destination(d_file);
};
destination d_file {
file("/var/log/messages-${APACHE.USER_NAME:-nouser}");
};
Example: Segmenting a part of a message
Multiple parsers can be used to split a part of an already parsed message into further segments. The following example splits the timestamp of a parsed Apache log message into separate fields.
parser p_apache_timestamp {
csv-parser(
columns("APACHE.TIMESTAMP.DAY", "APACHE.TIMESTAMP.MONTH", "APACHE.TIMESTAMP.YEAR", "APACHE.TIMESTAMP.HOUR", "APACHE.TIMESTAMP.MIN", "APACHE.TIMESTAMP.SEC", "APACHE.TIMESTAMP.ZONE")
delimiters("/: ")
flags(escape-none)
template("${APACHE.TIMESTAMP}")
);
};
log {
source(s_local);
parser(p_apache);
parser(p_apache_timestamp);
destination(d_file);
};
Further examples:
4.1 - Options of CSV parsers
The AxoSyslog application can separate parts of log messages (that is, the contents of the ${MESSAGE} macro) at delimiter characters or strings to named fields (columns) using the csv (comma-separated-values) parser The parsed fields act as user-defined macros that can be referenced in message templates, file- and tablenames, and so on.
Parsers are similar to filters: they must be defined in the AxoSyslog configuration file and used in the log statement. You can also define the parser inline in the log path.
Note
The order of filters, rewriting rules, and parsers in the log statement is important, as they are processed sequentially.
To create a csv-parser(), you have to define the columns of the message, the separator characters or strings (also called delimiters, for example, semicolon or tabulator), and optionally the characters that are used to escape the delimiter characters (quote-pairs()).
Declaration:
parser <parser_name> {
csv-parser(
columns(column1, column2, ...)
delimiters(chars("<delimiter_characters>"), strings("<delimiter_strings>"))
);
};
Column names work like macros.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
columns()
|
|
| Synopsis: |
columns(“PARSER.COLUMN1”, “PARSER.COLUMN2”, …) |
Description: Specifies the name of the columns to separate messages to. These names will be automatically available as macros. The values of these macros do not include the delimiters.
Starting with AxoSyslog version 4.5, you can omit the columns() option, and extract the values into matches ($1, $2, $3, and so on), which are available as the anonymous list $*. For example:
@version: current
log {
source { tcp(port(2000) flags(no-parse)); };
parser { csv-parser(delimiters(',') dialect(escape-backslash)); };
destination { stdout(template("$ISODATE $*\n")); };
};
delimiters()
| Synopsis: |
delimiters(chars("<delimiter_characters>")) or delimiters("<delimiter_characters>")
delimiters(strings("<delimiter_string1>", "<delimiter_string2>", ...)")
delimiters(chars("<delimiter_characters>"), strings("<delimiter_string1>")) |
Description: The delimiter is the character or string that separates the columns in the message. If you specify multiple characters using the delimiters(chars("<delimiter_characters>)) option, every character will be treated as a delimiter. To separate the columns at the tabulator (tab character), specify \\t. For example, to separate the text at every hyphen (-) and colon (:) character, use delimiters(chars("-:")), Note that the delimiters will not be included in the column values.
String delimiters
If you have to use a string as a delimiter, list your string delimiters in the delimiters(strings("<delimiter_string1>", "<delimiter_string2>" ...)) format.
By default, AxoSyslog uses space as a delimiter. If you want to use only the strings as delimiters, you have to disable the space delimiter, for example: delimiters(chars(""), strings("<delimiter_string>"))
Otherwise, AxoSyslog will use the string delimiters in addition to the default character delimiter, so delimiters(strings("==")) actually equals delimiters(chars(" "), strings("==")), and not delimiters(chars(""), strings("=="))
Multiple delimiters
If you use more than one delimiter, note the following points:
- AxoSyslog will split the message at the nearest possible delimiter. The order of the delimiters in the configuration file does not matter.
- You can use both string delimiters and character delimiters in a parser.
- The string delimiters may include characters that are also used as character delimiters.
- If a string delimiter and a character delimiter both match at the same position of the input, AxoSyslog uses the string delimiter.
dialect()
|
|
| Synopsis: |
escape-none, escape-backslash, escape-double-char, or escape-backslash-with-sequences |
Description: Specifies how to handle escaping in the parsed message. Default value: escape-none
parser p_demo_parser {
csv-parser(
prefix(".csv.")
delimiters(" ")
dialect(escape-backslash)
flags(strip-whitespace, greedy)
columns("column1", "column2", "column3")
);
};
The following values are available.
escape-backslash: The parsed message uses the backslash (\\) character to escape quote characters.escape-backslash-with-sequences: The parsed message uses "" as an escape character but also supports C-style escape
sequences, like \n or \r. Available in AxoSyslog version 4.0 and later.escape-double-char: The parsed message repeats the quote character when the quote character is used literally. For example, to escape a comma (,), the message contains two commas (,,).escape-none: The parsed message does not use any escaping for using the quote character literally.
flags()
|
|
| Synopsis: |
drop-invalid, escape-none, escape-backslash, escape-double-char, greedy, strip-whitespace |
Description: Specifies various options for parsing the message. The following flags are available:
-
drop-invalid: When the drop-invalid option is set, the parser does not process messages that do not match the parser. For example, a message does not match the parser if it has less columns than specified in the parser, or it has more columns but the greedy flag is not enabled. Using the drop-invalid option practically turns the parser into a special filter, that matches messages that have the predefined number of columns (using the specified delimiters).
Note
Messages dropped as invalid can be processed by a
fallback log path. For details on the
fallback option, see
Log path flags.
-
greedy: The greedy option assigns the remainder of the message to the last column, regardless of the delimiter characters set. You can use this option to process messages where the number of columns varies.
-
strip-whitespace: The strip-whitespace flag removes leading and trailing whitespaces from all columns.
Example: Adding the end of the message to the last column
If the greedy option is enabled, AxoSyslog adds the not-yet-parsed part of the message to the last column, ignoring any delimiter characters that may appear in this part of the message.
For example, you receive the following comma-separated message: example 1, example2, example3, and you segment it with the following parser:
csv-parser(columns("COLUMN1", "COLUMN2", "COLUMN3") delimiters(","));
The COLUMN1, COLUMN2, and COLUMN3 variables will contain the strings example1, example2, and example3, respectively. If the message looks like example 1, example2, example3, some more information, then any text appearing after the third comma (that is, some more information) is not parsed, and possibly lost if you use only the variables to reconstruct the message (for example, to send it to different columns of an SQL table).
Using the greedy flag will assign the remainder of the message to the last column, so that the COLUMN1, COLUMN2, and COLUMN3 variables will contain the strings example1, example2, and example3, some more information.
csv-parser(columns("COLUMN1", "COLUMN2", "COLUMN3") delimiters(",") flags(greedy));
null()
Description: If the value of a column is the value of the null() parameter, AxoSyslog changes the value of the column to an empty string. For example, if the columns of the message contain the “N/A” string to represent empty values, you can use the null("N/A") option to change these values to empty stings.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
This parser does not have a default prefix. To configure a custom prefix, use the following format:
parser {
csv-parser(prefix("myprefix."));
};
on-type-error()
Description: Specifies what to do when casting a parsed value to a specific data type fails. Note that the flags(drop-invalid) option and the on-error() global option also affects the behavior.
Accepts the same values as the on-error() global option.
quote-pairs()
|
|
| Synopsis: |
quote-pairs(<quote_pairs>) |
Description: List quote-pairs between single quotes. Delimiter characters or strings enclosed between quote characters are ignored. Note that the beginning and ending quote character does not have to be identical, for example, [} can also be a quote-pair. For an example of using quote-pairs() to parse Apache log files, see Example: Parsing Apache log files.
template()
|
|
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
For examples, see Example: Segmenting hostnames separated with a dash and Example: Segmenting a part of a message.
5 - Parsing dates and timestamps
The date parser can extract dates from non-syslog messages. It operates by default on the ${MESSAGE} part of the log message, but can operate on any template or field provided. The parsed date will be available as the sender date (that is, the ${S_DATE}, ${S_ISODATE}, ${S_MONTH}, and so on, and related macros). (To store the parsed date as the received date, use the time-stamp(recvd) option.)
Note
Note that parsing will fail if the format string does not match the entire template or field. Since by default AxoSyslog uses the ${MESSAGE} part of the log message, parsing will fail, unless the log message contains only a date, but that is unlikely, so practically you will have to segment the message (for example, using a
csv-parser()) before using the
date-parser(). You can also use
date-parser() to parse dates received in a JSON or key-value-formatted log message.
Declaration
parser parser_name {
date-parser(
format("<format-string-for-the-date>")
template("<field-to-parse>")
);
};
Example: Using the date-parser()
In the following example, AxoSyslog parses dates like 01/Jan/2016:13:05:05 PST from a field called MY_DATE using the following format string: format("%d/%b/%Y:%H:%M:%S %Z") (how you create this field from the incoming message is not shown in the example). In the destination template every message will begin with the timestamp in ISODATE format. Since the syslog parser is disabled, AxoSyslog will include the entire original message (including the original timestamp) in the ${MESSAGE} macro.
source s_file {
file("/tmp/input" flags(no-parse));
};
destination d_file {
file(
"/tmp/output"
template("${S_ISODATE} ${MESSAGE}\n")
);
};
log {
source(s_file);
parser { date-parser(format("%d/%b/%Y:%H:%M:%S %Z") template("${MY_DATE}")); };
destination(d_file);
};
In the template option, you can use template functions to specify which part of the message to parse with the format string. The following example selects the first 24 characters of the ${MESSAGE} macro.
date-parser(format("%d/%b/%Y:%H:%M:%S %Z") template("$(substr ${MESSAGE} 0 24)") );
In AxoSyslog version 3.23 and later, you can specify a comma-separated list of formats to parse multiple date formats with a single parser. For example:
date-parser(format(
"%FT%T.%f",
"%F %T,%f",
"%F %T"
));
If you need to modify or correct the timezone of the message after parsing, see Rewrite the timezone of a message.
5.1 - Options of date-parser() parsers
The date-parser() parser has the following options.
|
|
| Synopsis: |
format(string) |
| Default: |
|
Description: Specifies the format how AxoSyslog should parse the date. You can use the following format elements:
%% PERCENT
%a day of the week, abbreviated
%A day of the week
%b month abbr
%B month
%c MM/DD/YY HH:MM:SS
%C ctime format: Sat Nov 19 21:05:57 1994
%d numeric day of the month, with leading zeros (eg 01..31)
%e like %d, but a leading zero is replaced by a space (eg 1..31)
%f microseconds, leading 0's, extra digits are silently discarded
%D MM/DD/YY
%G GPS week number (weeks since January 6, 1980)
%h month, abbreviated
%H hour, 24 hour clock, leading 0's)
%I hour, 12 hour clock, leading 0's)
%j day of the year
%k hour
%l hour, 12 hour clock
%L month number, starting with 1
%m month number, starting with 01
%M minute, leading 0's
%n NEWLINE
%o ornate day of month -- "1st", "2nd", "25th", etc.
%p AM or PM
%P am or pm (Yes %p and %P are backwards :)
%q Quarter number, starting with 1
%r time format: 09:05:57 PM
%R time format: 21:05
%s seconds since the Epoch, UCT
%S seconds, leading 0's
%t TAB
%T time format: 21:05:57
%U week number, Sunday as first day of week
%w day of the week, numerically, Sunday == 0
%W week number, Monday as first day of week
%x date format: 11/19/94
%X time format: 21:05:57
%y year (2 digits)
%Y year (4 digits)
%Z timezone in ascii format (for example, PST), or in format -/+0000
%z timezone in ascii format (for example, PST), or in format -/+0000 (Required element)
Warning
When using the %z and %Z format codes, consider that while %z strictly expects a specified timezone, and triggers a warning if the timezone is missing, %Z does not trigger a warning if the timezone is not specified.
For further information about the %z and %Z format codes, see the ‘DESCRIPTION’ section on the srtptime(3) - NetBSD Manual Pages.
For example, for the date 01/Jan/2016:13:05:05 PST use the following format string: "%d/%b/%Y:%H:%M:%S %Z"
In AxoSyslog version 3.23 and later, you can specify a comma-separated list of formats to parse multiple date formats with a single parser. For example:
date-parser(format(
"%FT%T.%f",
"%F %T,%f",
"%F %T"
));
template()
|
|
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
flags()
|
|
| Type: |
guess-timezone |
| Default: |
empty string |
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp. For example:
date-parser(flags(guess-timezone));
time-stamp()
|
|
| Synopsis: |
stamp or recvd |
| Default: |
stamp |
Description: Determines if the parsed date values are treated as sent or received date. If you use time-stamp(stamp), AxoSyslog adds the parsed date to the S_ macros (corresponding to the sent date). If you use time-stamp(recvd), AxoSyslog adds the parsed date to the R_ macros (corresponding to the received date).
time-zone()
|
|
| Synopsis: |
time-zone(string) |
| Default: |
|
Description: If this option is set, AxoSyslog assumes that the parsed timestamp refers to the specified timezone. The timezone set in the time-zone() option overrides any timezone information parsed from the timestamp.
The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.
value()
|
|
| Synopsis: |
string |
| Default: |
|
Available in AxoSyslog 4.1 and later.
Description: Instruct date-parser() to store the resulting timestamp in a name-value pair specified in value(), instead of changing the timestamp value of the LogMessage.
6 - db-parser: Process message content with a pattern database (patterndb)
6.1 - Classifying log messages
The AxoSyslog application can compare the contents of the received log messages to predefined message patterns. By comparing the messages to the known patterns, AxoSyslog is able to identify the exact type of the messages, and sort them into message classes. The message classes can be used to classify the type of the event described in the log message. The message classes can be customized, and for example, can label the messages as user login, application crash, file transfer, and so on events.
To find the pattern that matches a particular message, AxoSyslog uses a method called longest prefix match radix tree. This means that AxoSyslog creates a tree structure of the available patterns, where the different characters available in the patterns for a given position are the branches of the tree.
To classify a message, AxoSyslog selects the first character of the message (the text of message, not the header), and selects the patterns starting with this character, other patterns are ignored for the rest of the process. After that, the second character of the message is compared to the second character of the selected patterns. Again, matching patterns are selected, and the others discarded. This process is repeated until a single pattern completely matches the message, or no match is found. In the latter case, the message is classified as unknown, otherwise the class of the matching pattern is assigned to the message.
To make the message classification more flexible and robust, the patterns can contain pattern parsers: elements that match on a set of characters. For example, the NUMBER parser matches on any integer or hexadecimal number (for example, 1, 123, 894054, 0xFFFF, and so on). Other pattern parsers match on various strings and IP addresses. For the details of available pattern parsers, see Using pattern parsers.
The functionality of the pattern database is similar to that of the logcheck project, but it is much easier to write and maintain the patterns used by syslog-ng, than the regular expressions used by logcheck. Also, it is much easier to understand AxoSyslog pattens than regular expressions.
Pattern matching based on regular expressions is computationally very intensive, especially when the number of patterns increases. The solution used by AxoSyslog can be performed real-time, and is independent from the number of patterns, so it scales much better. The following patterns describe the same message: Accepted password for bazsi from 10.50.0.247 port 42156 ssh2
A regular expression matching this message from the logcheck project: Accepted (gssapi(-with-mic|-keyex)?|rsa|dsa|password|publickey|keyboard-interactive/pam) for [^[:space:]]+ from [^[:space:]]+ port [0-9]+( (ssh|ssh2))?
An AxoSyslog database pattern for this message: Accepted @QSTRING:auth_method: @ for@QSTRING:username: @from @QSTRING:client_addr: @port @NUMBER:port:@ ssh2
For details on using pattern databases to classify log messages, see Using pattern databases.
6.1.1 - The structure of the pattern database
The pattern database is organized as follows:
-
The pattern database consists of rulesets. A ruleset consists of a Program Pattern and a set of rules: the rules of a ruleset are applied to log messages if the name of the application that sent the message matches the Program Pattern of the ruleset. The name of the application (the content of the ${PROGRAM} macro) is compared to the Program Patterns of the available rulesets, and then the rules of the matching rulesets are applied to the message. (If the content of the ${PROGRAM} macro is not the proper name of the application, you can use the program-template() option to specify it.)
-
The Program Pattern can be a string that specifies the name of the application or the beginning of its name (for example, to match for sendmail, the program pattern can be sendmail, or just send), and the Program Pattern can contain pattern parsers. Note that pattern parsers are completely independent from the AxoSyslog parsers used to segment messages. Additionally, every rule has a unique identifier: if a message matches a rule, the identifier of the rule is stored together with the message.
-
Rules consist of a message pattern and a class. The Message Pattern is similar to the Program Pattern, but is applied to the message part of the log message (the content of the ${MESSAGE} macro). If a message pattern matches the message, the class of the rule is assigned to the message (for example, Security, Violation, and so on).
-
Rules can also contain additional information about the matching messages, such as the description of the rule, an URL, name-value pairs, or free-form tags.
-
Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers.
Note
If the ${PROGRAM} part of a message is empty, rules with an empty Program Pattern are used to classify the message.
If the same Program Pattern is used in multiple rulesets, the rules of these rulesets are merged, and every rule is used to classify the message. Note that message patterns must be unique within the merged rulesets, but the currently only one ruleset is checked for uniqueness.
If the content of the ${PROGRAM} macro is not the proper name of the application, you can use the program-template() option to specify it.
6.1.2 - How pattern matching works
The followings describe how patterns work. This information applies to program patterns and message patterns alike, even though message patterns are used to illustrate the procedure.
Patterns can consist of literals (keywords, or rather, keycharacters) and pattern parsers. Pattern parsers attempt to parse a sequence of characters according to certain rules.
Note
Wildcards and regular expressions cannot be used in patterns. The @ character must be escaped, that is, to match for this character, you have to write @@ in your pattern. This is required because pattern parsers are enclosed between @ characters.
When a new message arrives, AxoSyslog attempts to classify it using the pattern database. The available patterns are organized alphabetically into a tree, and AxoSyslog inspects the message character-by-character, starting from the beginning. This approach ensures that only a small subset of the rules must be evaluated at any given step, resulting in high processing speed. Note that the speed of classifying messages is practically independent from the total number of rules.
For example, if the message begins with the Apple string, only patterns beginning with the character A are considered. In the next step, AxoSyslog selects the patterns that start with Ap, and so on, until there is no more specific pattern left. The AxoSyslog application has a strong preference for rules that match the input string completely.
Note that literal matches take precedence over pattern parser matches: if at a step there is a pattern that matches the next character with a literal, and another pattern that would match it with a parser, the pattern with the literal match is selected. Using the previous example, if at the third step there is the literal pattern Apport and a pattern parser Ap@STRING@, the Apport pattern is matched. If the literal does not match the incoming string (for example, Apple), AxoSyslog attempts to match the pattern with the parser. However, if there are two or more parsers on the same level, only the first one will be applied, even if it does not perfectly match the message.
If there are two parsers at the same level (for example, Ap@STRING@ and Ap@QSTRING@), it is random which pattern is applied (technically, the one that is loaded first). However, if the selected parser cannot parse at least one character of the message, the other parser is used. But having two different parsers at the same level is extremely rare, so the impact of this limitation is much less than it appears.
6.1.3 - Artificial ignorance
Artificial ignorance is a method used to detect anomalies. When applied to log analysis, it means that you ignore the regular, common log messages — these are the result of the regular behavior of your system, and therefore are not too concerning. However, new messages that have not appeared in the logs before can signal important events, and should be therefore investigated. “By definition, something we have never seen before is anomalous” (Marcus J. Ranum).Â
The AxoSyslog application can classify messages using a pattern database: messages that do not match any pattern are classified as unknown. This provides a way to use artificial ignorance to review your log messages. You can periodically review the unknown messages — AxoSyslog can send them to a separate destination, and add patterns for them to the pattern database. By reviewing and manually classifying the unknown messages, you can iteratively classify more and more messages, until only the really anomalous messages show up as unknown.
Obviously, for this to work, a large number of message patterns are required. The radix-tree matching method used for message classification is very effective, can be performed very fast, and scales very well. Basically the time required to perform a pattern matching is independent from the number of patterns in the database.
6.2 - Using pattern databases
To classify messages using a pattern database, include a db-parser() statement in your syslog-ng.conf configuration file using the following syntax:
Declaration:
parser <identifier> {
db-parser(file("<database_filename>"));
};
Note that using the parser in a log statement only performs the classification, but does not automatically do anything with the results of the classification.
Example: Defining pattern databases
The following statement uses the database located at /opt/syslog-ng/var/db/patterndb.xml.
parser pattern_db {
db-parser(
file("/opt/syslog-ng/var/db/patterndb.xml")
);
};
To apply the patterns on the incoming messages, include the parser in a log statement:
log {
source(s_all);
parser(pattern_db);
destination( di_messages_class);
};
By default, AxoSyslog tries to apply the patterns to the body of the incoming messages, that is, to the value of the $MESSAGE macro. If you want to apply patterns to a specific field, or to an expression created from the log message (for example, using template functions or other parsers), use the message-template() option. For example:
parser pattern_db {
db-parser(
file("/opt/syslog-ng/var/db/patterndb.xml")
message-template("${MY-CUSTOM-FIELD-TO-PROCESS}")
);
};
By default, AxoSyslog uses the name of the application (content of the ${PROGRAM} macro) to select which rules to apply to the message. If the content of the ${PROGRAM} macro is not the proper name of the application, you can use the program-template() option to specify it. For example:
parser pattern_db {
db-parser(
file("/opt/syslog-ng/var/db/patterndb.xml")
program-template("${MY-CUSTOM-FIELD-TO-SELECT-RULES}")
);
};
Note that the program-template() option is available in AxoSyslog version 3.21 and later.
Note
The default location of the pattern database file is /opt/syslog-ng/var/run/patterndb.xml. The file option of the db-parser() statement can be used to specify a different file, thus different db-parser statements can use different pattern databases.
Example: Using classification results
The following destination separates the log messages into different files based on the class assigned to the pattern that matches the message (for example, Violation and Security type messages are stored in a separate file), and also adds the ID of the matching rule to the message:
destination di_messages_class {
file(
"/var/log/messages-${.classifier.class}"
template("${.classifier.rule_id};${S_UNIXTIME};${SOURCEIP};${HOST};${PROGRAM};${PID};${MESSAGE}\n")
template-escape(no)
);
};
Note that if you chain pattern databases, that is, use multiple databases in the same log path, the class assigned to the message (the value of ${.classifier.class}) will be the one assigned by the last pattern database. As a result, a message might be classified as unknown even if a previous parser successfully classified it. For example, consider the following configuration:
log {
...
parser(db_parser1);
parser(db_parser2);
...
};
Even if db_parser1 matches the message, db_parser2 might set ${.classifier.class} to unknown. To avoid this problem, you can use an ‘if’ statement to apply the second parser only if the first parser could not classify the message:
log {
...
parser{ db-parser(file("db_parser1.xml")); };
if (match("^unknown$" value(".classifier.class"))) {
parser { db-parser(file("db_parser2.xml")); };
};
...
};
For details on how to create your own pattern databases see The pattern database format.
Drop unmatched messages
If you want to automatically drop unmatched messages (that is, discard every message that does not match a pattern in the pattern database), use the drop-unmatched() option in the definition of the pattern database:
parser pattern_db {
db-parser(
file("/opt/syslog-ng/var/db/patterndb.xml")
drop-unmatched(yes)
);
};
Note that the drop-unmatched() option is available in AxoSyslog version 3.11 and later.
6.2.1 - Using parser results in filters and templates
The results of message classification and parsing can be used in custom filters and templates, for example, in file and database templates. The following built-in macros allow you to use the results of the classification:
-
The .classifier.class macro contains the class assigned to the message (for example, violation, security, or unknown).
-
The .classifier.rule_id macro contains the identifier of the message pattern that matched the message.
-
The .classifier.context_id macro contains the identifier of the context for messages that were correlated. For details on correlating messages, see Correlating log messages using pattern databases.
Example: Using classification results for filtering messages
To filter on a specific message class, create a filter that checks the .classifier_class macro, and use this filter in a log statement.
filter fi_class_violation {
match(
"violation"
value(".classifier.class")
type("string")
);
};
log {
source(s_all);
parser(pattern_db);
filter(fi_class_violation);
destination(di_class_violation);
};
Filtering on the unknown class selects messages that did not match any rule of the pattern database. Routing these messages into a separate file allows you to periodically review new or unknown messages.
To filter on messages matching a specific classification rule, create a filter that checks the .classifier.rule_id macro. The unique identifier of the rule (for example, e1e9c0d8-13bb-11de-8293-000c2922ed0a) is the id attribute of the rule in the XML database.
filter fi_class_rule {
match(
"e1e9c0d8-13bb-11de-8293-000c2922ed0a"
value(".classifier.rule_id")
type("string")
);
};
Pattern database rules can assign tags to messages. These tags can be used to select tagged messages using the tags() filter function.
Note
The AxoSyslog application automatically adds the class of the message as a tag using the .classifier.<message-class> format. For example, messages classified as “system” receive the .classifier.system tag. Use the tags() filter function to select messages of a specific class.
filter f_tag_filter {tags(".classifier.system");};
The message-segments parsed by the pattern parsers can also be used as macros as well. To accomplish this, you have to add a name to the parser, and then you can use this name as a macro that refers to the parsed value of the message.
Example: Using pattern parsers as macros
For example, you want to parse messages of an application that look like "Transaction: <type>", where <type> is a string that has different values (for example, refused, accepted, incomplete, and so on). To parse these messages, you can use the following pattern:
'Transaction: @ESTRING::.@'
Here the @ESTRING@ parser parses the message until the next full stop character. To use the results in a filter or a filename template, include a name in the parser of the pattern, for example:
'Transaction: @ESTRING:TRANSACTIONTYPE:.@'
After that, add a custom template to the log path that uses this template. For example, to select every accepted transaction, use the following custom filter in the log path:
match("accepted" value("TRANSACTIONTYPE"));
Note
The above macros can be used in database columns and filename templates as well, if you create custom templates for the destination or logspace.
Use a consistent naming scheme for your macros, for example, APPLICATIONNAME_MACRONAME.
6.2.2 - Correlating log messages using pattern databases
The AxoSyslog application can correlate log messages identified using pattern databases. Alternatively, you can also correlate log messages using the grouping-by() parser. For details, see Correlating messages using the grouping-by() parser.
Log messages are supposed to describe events, but applications often separate information about a single event into different log messages. For example, the Postfix email server logs the sender and recipient addresses into separate log messages, or in case of an unsuccessful login attempt, the OpenSSH server sends a log message about the authentication failure, and the reason of the failure in the next message. Of course, messages that are not so directly related can be correlated as well, for example, login-logout messages, and so on.
To correlate log messages with AxoSyslog, you can add messages into message-groups called contexts. A context consists of a series of log messages that are related to each other in some way, for example, the log messages of an SSH session can belong to the same context. As new messages come in, they may be added to a context. Also, when an incoming message is identified it can trigger actions to be performed, for example, generate a new message that contains all the important information that was stored previously in the context.
(For details on triggering actions and generating messages, see Triggering actions for identified messages.)
There are two attributes for pattern database rules that determine if a message matching the rule is added to a context: context-scope and context-id. The context-scope attribute acts as an early filter, selecting messages sent by the same process (${HOST}${PROGRAM}${PID} is identical), application (${HOST}${PROGRAM} is identical), or host, while the context-id actually adds the message to the context specified in the id. The context-id can be a simple string, or can contain macros or values extracted from the log messages for further filtering. Starting with AxoSyslog version 3.5, if a message is added to a context, AxoSyslog automatically adds the identifier of the context to the .classifier.context_id macro of the message.
Note
Message contexts are persistent and are not lost when AxoSyslog is reloaded (SIGHUP), but are lost when AxoSyslog is restarted.
Another parameter of a rule is the context-timeout attribute, which determines how long a context is stored, that is, how long AxoSyslog waits for related messages to arrive.
Note the following points about timeout values:
-
When a new message is added to a context, AxoSyslog will restart the timeout using the context-timeout set for the new message.
-
When calculating if the timeout has already expired or not, AxoSyslog uses the timestamps of the incoming messages, not system time elapsed between receiving the two messages (unless the messages do not include a timestamp, or the keep-timestamp(no) option is set). That way AxoSyslog can be used to process and correlate already existing log messages offline. However, the timestamps of the messages must be in chronological order (that is, a new message cannot be older than the one already processed), and if a message is newer than the current system time (that is, it seems to be coming from the future), AxoSyslog will replace its timestamp with the current system time.
Example: How AxoSyslog calculates context-timeout
Consider the following two messages:
<38>1990-01-01T14:45:25 customhostname program6[1234]: program6 testmessage
<38>1990-01-01T14:46:25 customhostname program6[1234]: program6 testmessage
If the context-timeout is 10 seconds and AxoSyslog receives the messages within 1 second, the timeout event will occour immediately, because the difference of the two timestamp (60 seconds) is larger than the timeout value (10 seconds).
-
Avoid using unnecessarily long timeout values on high-traffic systems, as storing the contexts for many messages can require considerable memory. For example, if two related messages usually arrive within seconds, it is not needed to set the timeout to several hours.
Example: Using message correlation
<rule xml:id="..." context-id="ssh-session" context-timeout="86400" context-scope="process">
<patterns>
<pattern>Accepted @ESTRING:usracct.authmethod: @for @ESTRING:usracct.username: @from @ESTRING:usracct.device: @port @ESTRING:: @@ANYSTRING:usracct.service@</pattern>
</patterns>
...
</rule>
For details on configuring message correlation, see the context-id, context-timeout, and context-scope attributes of pattern database rules.
6.2.2.1 - Referencing earlier messages of the context
When using the <value> element in pattern database rules together with message correlation, you can also refer to fields and values of earlier messages of the context by adding the @<distance-of-referenced-message-from-the-current> suffix to the macro. For example, if there are three log messages in a context, and you are creating a generated message for the third log message, the ${HOST}@1 expression refers to the host field of the current (third) message in the context, the ${HOST}@2 expression refers to the host field of the previous (second) message in the context, ${PID}@3 to the PID of the first message, and so on. For example, the following message can be created from SSH login/logout messages (for details on generating new messages, see Triggering actions for identified messages): An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE}.
Warning
When referencing an earlier message of the context, always enclose the field name between braces, for example, ${PID}@3. The reference will not work if you omit the braces.
Note
To use a literal @ character in a template, use @@.
Example: Referencing values from an earlier message
The following action can be used to log the length of an SSH session (the time difference between a login and a logout message in the context):
<actions>
<action>
<message>
<values>
<value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE} </value>
</values>
</message>
</action>
</actions>
If you do not know in which message of the context contains the information you need, you can use the grep, the context-lookup, or the context-values template functions.
Example: Using the grep template function
The following example selects the message of the context that has a username name-value pair with the root value, and returns the value of the auth_method name-value pair.
$(grep ("${username}" == "root") ${auth_method})
To perform calculations on fields that have numerical values, see Numerical operations.
6.3 - Triggering actions for identified messages
The AxoSyslog application can generate (trigger) messages automatically if certain events occur, for example, a specific log message is received, or the correlation timeout of a message expires. Basically, you can define messages for every pattern database rule that are emitted when a message matching the rule is received. Triggering messages is often used together with message correlation, but can also be used separately. When used together with message correlation, you can also create a new correlation context when a new message is received.
The generated message is injected into the same place where the db-parser() statement is referenced in the log path. To post the generated message into the internal() source instead, use the inject-mode() option in the definition of the parser.
Example: Sending triggered messages to the internal() source
To send the generated messages to the internal source, use the inject-mode(internal) option:
parser p_db {
db-parser(
file("mypatterndbfile.xml")
inject-mode(internal)
);
};
To inject the generated messages where the pattern database is referenced, use the inject-mode(pass-through) option:
parser p_db {
db-parser(
file("mypatterndbfile.xml")
inject-mode(pass-through)
);
};
The generated message must be configured in the pattern database rule. It is possible to create an entire message, use macros and values extracted from the original message with pattern database, and so on.
Example: Generating messages for pattern database matches
When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.
<actions>
<action>
<message>
<values>
<value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
</values>
</message>
</action>
</actions>
To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.
Example: Generating messages with inherited values
The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name
<actions>
<action>
<message inherit-properties='TRUE'>
<values>
<value name="PROGRAM">overriding-original-program-name</value>
</values>
</message>
</action>
</actions>
Example: Creating a new context from an action
In AxoSyslog version 3.8 and newer, you can create a new context as an action. For details, see Element: create-context.
The following example creates a new context whenever the rule matches. The new context receives 1000 as ID, and program as scope, and the content set in the <message> element of the >element.
<rule provider='test' id='12' class='violation'>
<patterns>
<pattern>simple-message-with-action-to-create-context</pattern>
</patterns>
<actions>
<action trigger='match'>
<create-context context-id='1000' context-timeout='60' context-scope='program'>
<message inherit-properties='context'>
<values>
<value name='MESSAGE'>context message</value>
</values>
</message>
</create-context>
</action>
</actions>
</rule>
For details on configuring actions, see the description of the pattern database format.
6.3.1 - Conditional actions
To limit when a message is triggered, use the condition attribute and specify a filter expression: the action will be executed only if the condition is met. For example, the following action is executed only if the message was sent by the host called myhost.
<action condition="'${HOST}' == 'myhost'">
You can use the same operators in the condition that can be used in filters. For details, see Comparing macro values in filters.
The following action can be used to log the length of an SSH session (the time difference between a login and a logout message in the context):
<actions>
<action>
<message>
<values>
<value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 ${DATE} </value>
</values>
</message>
</action>
</actions>
Example: Actions based on the number of messages
The following example triggers different actions based on the number of messages in the context. This way you can check if the context contains enough messages for the event to be complete, and execute a different action if it does not.
<actions>
<action condition='"$(context-length)" >= "4"'>
<message>
<values>
<value name="PROGRAM">event</value>
<value name="MESSAGE">Event complete</value>
</values>
</message>
</action>
<action condition='"$(context-length)" < "4"'>
<message>
<values>
<value name="PROGRAM">error</value>
<value name="MESSAGE">Error detected</value>
</values>
</message>
</action>
</actions>
6.3.2 - External actions
To perform an external action when a message is triggered, for example, to send the message in an email, you have to route the generated messages to an external application using the program() destination.
Example: Sending triggered messages to external applications
The following sample configuration selects the triggered messages and sends them to an external script.
-
Set a field in the triggered message that is easy to identify and filter. For example:
<values>
<value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
<value name="TRIGGER">yes</value>
</values>
-
Create a destination that will process the triggered messages.
destination d_triggers {
program("/bin/myscript"; );
};
-
Create a filter that selects the triggered messages from the internal source.
filter f_triggers {
match("yes" value ("TRIGGER") type(string));
};
-
Create a logpath that selects the triggered messages from the internal source and sends them to the script:
log { source(s_local); filter(f_triggers); destination(d_triggers); };
-
Create a script that will actually process the generated messages, for example:
#!/usr/bin/perl
while (<>) {
# body of the script to send emails, snmp traps, and so on
}
6.3.3 - Actions and message correlation
Certain features of generating messages can be used only if message correlation is used as well. For details on correlating messages, see Correlating log messages using pattern databases.
-
The AxoSyslog application automatically fills the fields for the generated message based on the scope of the context, for example, the HOST and PROGRAM fields if the context-scope is program.
-
When used together with message correlation, you can also refer to fields and values of earlier messages of the context by adding the @<distance-of-referenced-message-from-the-current> suffix to the macro. For details, see Referencing earlier messages of the context.
Example: Referencing values from an earlier message
The following action can be used to log the length of an SSH session (the time difference between a login and a logout message in the context):
<actions>
<action>
<message>
<values>
<value name="MESSAGE">An SSH session for ${SSH_USERNAME}@1 from ${SSH_CLIENT_ADDRESS}@2 closed. Session lasted from ${DATE}@2 to ${DATE} </value>
</values>
</message>
</action>
</actions>
-
You can use the name-value pairs of other messages of the context. If you set the inherit-properties attribute of the generated message to context, AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. This means that you can refer to a name-value pair without having to know which message of the context included it. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. To refer to an earlier value, use the @<distance-of-referenced-message-from-the-current> suffix format.
<action>
<message inherit-properties='context'>
Example: Using the inherit-properties option
For example, if inherit-properties is set to context, and you have a rule that collects SSH login and logout messages to the same context, you can use the following value to generate a message collecting the most important information form both messages, including the beginning and end date.
<value name="MESSAGE">An SSH session for ${SSH_USERNAME} from ${SSH_CLIENT_ADDRESS} closed. Session lasted from ${DATE}@2 to $DATE pid: $PID.</value>
The following is a detailed rule for this purpose.
<patterndb version='4' pub_date='2015-04-13'>
<ruleset name='sshd' id='12345678'>
<pattern>sshd</pattern>
<rules>
<!-- The pattern database rule for the first log message -->
<rule provider='me' id='12347598' class='system'
context-id="ssh-login-logout" context-timeout="86400"
context-scope="process">
<!-- Note the context-id that groups together the
relevant messages, and the context-timeout value that
determines how long a new message can be added to the
context -->
<patterns>
<pattern>Accepted @ESTRING:SSH.AUTH_METHOD: @for @ESTRING:SSH_USERNAME: @from @ESTRING:SSH_CLIENT_ADDRESS: @port @ESTRING:: @@ANYSTRING:SSH_SERVICE@</pattern>
<!-- This is the actual pattern used to identify
the log message. The segments between the @
characters are parsers that recognize the variable
parts of the message - they can also be used as
macros. -->
</patterns>
</rule>
<!-- The pattern database rule for the fourth log message -->
<rule provider='me' id='12347599' class='system' context-id="ssh-login-logout" context-scope="process">
<patterns>
<pattern>pam_unix(sshd:session): session closed for user @ANYSTRING:SSH_USERNAME@</pattern>
</patterns>
<actions>
<action>
<message inherit-properties='context'>
<values>
<value name="MESSAGE">An SSH session for ${SSH_USERNAME} from ${SSH_CLIENT_ADDRESS} closed. Session lasted from ${DATE}@2 to $DATE pid: $PID.</value>
<value name="TRIGGER">yes</value>
<!-- This is the new log message
that is generated when the logout
message is received. The macros ending
with @2 reference values of the
previous message from the context. -->
</values>
</message>
</action>
</actions>
</rule>
</rules>
</ruleset>
</patterndb>
-
It is possible to generate a message when the context-timeout of the original message expires and no new message is added to the context during this time. To accomplish this, include the trigger="timeout" attribute in the action element:
<action trigger="timeout">
Example: Sending alert when a client disappears
The following example shows how to combine various features of AxoSyslog to send an email alert if a client stops sending messages.
-
Configure your clients to send MARK messages periodically. It is enough to configure MARK messages for the destination that forwards your log messages to your AxoSyslog server (mark-mode(periodical)).
-
On your AxoSyslog server, create a pattern database rule that matches on the incoming MARK messages. In the rule, set the context-scope attribute to host, and the context-timeout attribute to a value that is higher than the mark-freq value set on your clients (by default, mark-freq is 1200 seconds, so set context-timeout at least to 1500 seconds, but you might want to use a higher value, depending on your environment).
-
Add an action to this rule that sends you an email alert if the context-timeout expires, and the server does not receive a new MARK message (<action trigger="timeout">).
-
On your AxoSyslog server, use the pattern database in the log path that handles incoming log messages.
6.4 - Creating pattern databases
6.4.1 - Using pattern parsers
Pattern parsers attempt to parse a part of the message using rules specific to the type of the parser. Parsers are enclosed between @ characters. The syntax of parsers is the following:
-
a beginning @ character,
-
the type of the parser written in capitals,
-
optionally a name,
-
parameters of the parser, if any, and
-
a closing @ character.
Example: Pattern parser syntax
A simple parser:
A named parser:
A named parser with a parameter:
A parser with a parameter, but without a name:
Patterns and literals can be mixed together. For example, to parse a message that begins with the Host: string followed by an IP address (for example, Host: 192.168.1.1), the following pattern can be used: Host:@IPv4@.
Note
Note that using parsers is a CPU-intensive operation. Use the ESTRING and QSTRING parsers whenever possible, as these can be processed much faster than the other parsers.
Example: Using the STRING and ESTRING parsers
For example, look at the following message: user=joe96 group=somegroup.
-
@STRING:mytext:@ parses only to the first non-alphanumeric character (=), parsing only user, so the value of the ${mytext} macro will be user
-
@STRING:mytext:=@ parses the equation mark as well, and proceeds to the next non-alphanumeric character (the whitespace), resulting in user=joe96
-
@STRING:mytext:= @ will parse the whitespace as well, and proceed to the next non-alphanumeric non-equation mark non-whitespace character, resulting in user=joe96 group=somegroup
Of course, usually it is better to parse the different values separately, like this: "user=@STRING:user@ group=@STRING:group@".
If the username or the group may contain non-alphanumeric characters, you can either include these in the second parameter of the parser (as shown at the beginning of this example), or use an ESTRING parser to parse the message till the next whitespace: "user=@ESTRING:user: @group=@ESTRING:group: @".
6.4.1.1 - Pattern parsers
The following parsers are available in AxoSyslog. The internal parsers (for example, @NUMBER@) automatically associate type information to the parsed name-value pair. For details on data types, see Specifying data types in value-pairs.
@ANYSTRING@
Parses everything to the end of the message, you can use it to collect everything that is not parsed specifically to a single macro. In that sense its behavior is similar to the greedy() option of the CSV parser.
@DOUBLE@
An obsolete alias of the @FLOAT@ parser.
@EMAIL@
This parser matches an email address. The parameter is a set of characters to strip from the beginning and the end of the email address. That way email addresses enclosed between other characters can be matched easily (for example, <user@example.com> or "user@example.com". Characters that are valid for a hostname are not stripped from the end of the hostname. This includes a trailing period if present.
For example, the @EMAIL:email:"[<]> parser will match any of the following email addresses: <user@example.com>, [user@example.com], "user@example.com", and set the value of the email macro to user@example.com.
@ESTRING@
This parser has a required parameter that acts as the stopcharacter: the parser parses everything until it finds the stopcharacter. For example, to stop by the next " (double quote) character, use @ESTRING::"@. You can use the colon (:) as stopcharacter as well, for example: @ESTRING:::@. You can also specify a stopstring instead of a single character, for example, @ESTRING::stop_here.@. The @ character cannot be a stopcharacter, nor can line-breaks or tabs.
@FLOAT@
A floating-point number that may contain a dot (.) character. (Up to AxoSyslog 3.1, the name of this parser was @DOUBLE@.)
@HOSTNAME@
Parses a generic hostname. The hostname may contain only alphanumeric characters (A-Z,a-z,0-9), hypen (-), or dot (.).
@IPv4@
Parses an IPv4 IP address (numbers separated with a maximum of 3 dots).
@IPv6@
Parses any valid IPv6 IP address.
@IPvANY@
Parses any IP address.
@LLADDR@
Parses a Link Layer Address in the xx:xx:xx:... form, where each xx is a 2 digit HEX number (an octet). The parameter specifies the maximum number of octets to match and defaults to 20. The MACADDR parser is a special wrapper using the LLADDR parser. For example, the following parser parses maximally 10 octets, and stores the results in the link-level-address macro:
@LLADDR:link-level-address:10@
@MACADDR@
Parses the standard format of a MAC-48 address, consisting of is six groups of two hexadecimal digits, separated by colons. For example, 00:50:fc:e3:cd:37.
@NLSTRING@
This parser parses everything until the next new-line character (more precisely, until the next Unix-style LF or Windows-style CRLF character). For single-line messages, NLSTRING is equivalent with ANYSTRING. For multi-line messages, NLSTRING parses to the end of the current line, while ANYSTRING parses to the end of the message. Using NLSTRING is useful when parsing multi-line messages, for example, Windows logs. For example, the following pattern parses information from Windows security auditing logs.
<pattern>Example-PC\Example: Security Microsoft Windows security auditing.: [Success Audit] A new process has been created.
Subject:
Security ID: @LNSTRING:.winaudit.SubjectUserSid@
Account Name: @LNSTRING:.winaudit.SubjectUserName@
Account Domain: @LNSTRING:.winaudit.SubjectDomainName@
Logon ID: @LNSTRING:.winaudit.SubjectLogonId@
Process Information:
New Process ID: @LNSTRING:.winaudit.NewProcessId@
New Process Name: @LNSTRING:.winaudit.NewProcessName@
Token Elevation Type: @LNSTRING:.winaudit.TokenElevationType@
Creator Process ID: @LNSTRING:.winaudit.ProcessId@
Process Command Line: @LNSTRING:.winaudit.CommandLine@
Token Elevation Type indicates the type of token that was assigned to the new process in accordance with User Account Control policy.</pattern>
@NUMBER@
A sequence of decimal (0-9) numbers (for example, 1, 0687, and so on). Note that if the number starts with the 0x characters, it is parsed as a hexadecimal number, but only if at least one valid character follows 0x. A leading hyphen (-) is accepted for non-hexadecimal numbers, but other separator characters (for example, dot or comma) are not. To parse floating-point numbers, use the @FLOAT@ parser.
@OPTIONALSET@
Parse any combination of the specified characters. For example, specifying a whitespace character parses any number of whitespaces, and can be used to process paddings (for example, log messages of the Squid application have whitespace padding after the username).
For example, the @OPTIONALSET:: "@ parser will parse any combination of whitespaces and double-quotes.
Available in 3.31 and later.
Note
The
@OPTIONALSET@ parser works almost exactly like the
@SET@ parser, but the
@OPTIONALSET@ parser continues parsing even if none of the specified characters are found.
@PCRE@
Use Perl-Compatible Regular Expressions (as implemented by the PCRE library), after the identification of the potential patterns has happened by the radix implementation.
Syntax: @PCRE:name:regexp@
@QSTRING@
Parse a string between the quote characters specified as parameter. Note that the quote character can be different at the beginning and the end of the quote, for example: @QSTRING::"@ parses everything between two quotation marks ("), while @QSTRING:\<\>@ parses from an opening bracket to the closing bracket. The @ character cannot be a quote character, nor can line-breaks or tabs.
@SET@
Parse any combination of the specified characters until another character is found. For example, specifying a whitespace character parses any number of whitespaces, and can be used to process paddings (for example, log messages of the Squid application have whitespace padding after the username).
For example, the @SET:: "@ parser will parse any combination of whitespaces and double-quotes.
Available in AxoSyslog 3.4 and later.
@STRING@
A sequence of alphanumeric characters (0-9, A-z), not including any whitespace. Optionally, other accepted characters can be listed as parameters (for example, to parse a complete sentence, add the whitespace as parameter, like: @STRING:: @). Note that the @ character cannot be a parameter, nor can line-breaks or tabs.
6.4.2 - What's new in the pattern database format V5
The V5 database format has the following differences compared to the V4 format:
-
The <ruleset> element can now store multiple reference URLs using the new <rule_urls> element. For details, see Element: ruleset.
-
In an <action>, you can now initialize a new context. As a result, the <message> element is not required. For details, see Element: create-context.
-
The inherit-properties attribute is deprecated, use the inherit-mode attribute instead. For details, see Element: action.
6.4.3 - The pattern database format
Pattern databases are XML files that contain rules describing the message patterns.
The following scheme describes the V5 format of the pattern database. This format is backwards-compatible with the earlier formats.
For a sample database containing only a single pattern, see Example: A pattern database containing a single rule.
Note
Use the pdbtool utility that is bundled with AxoSyslog to test message patterns and convert existing databases to the latest format. For details, see The pdbtool manual page.
To automatically create an initial pattern database from an existing log file, use the pdbtool patternize command. For details, see The pdbtool manual page.
Example: A pattern database containing a single rule
The following pattern database contains a single rule that matches a log message of the ssh application. A sample log message looks like:
Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2
The following is a simple pattern database containing a matching rule.
<patterndb version='5' pub_date='2010-10-17'>
<ruleset name='ssh' id='123456678'>
<pattern>ssh</pattern>
<rules>
<rule provider='me' id='182437592347598' class='system'>
<patterns>
<pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
</patterns>
</rule>
</rules>
</ruleset>
</patterndb>
Note that the rule uses macros that refer to parts of the message, for example, you can use the ${SSH_USERNAME} macro refer to the username used in the connection.
The following is the same example, but with a test message and test values for the parsers.
<patterndb version='4' pub_date='2010-10-17'>
<ruleset name='ssh' id='123456678'>
<pattern>ssh</pattern>
<rules>
<rule provider='me' id='182437592347598' class='system'>
<patterns>
<pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
</patterns>
<examples>
<example>
<test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>
<test_values>
<test_value name="SSH.AUTH_METHOD">password</test_value>
<test_value name="SSH_USERNAME">sampleuser</test_value>
<test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>
<test_value name="SSH_PORT_NUMBER">42156</test_value>
</test_values>
</example>
</examples>
</rule>
</rules>
</ruleset>
</patterndb>
6.4.3.1 - Element: patterndb
Location
/patterndb
Description
The container element of the pattern database.
Attributes
Children
Example
<patterndb version='4' pub_date='2010-10-25'>
6.4.3.2 - Element: ruleset
Location
/patterndb/ruleset
Description
A container element to group log patterns for an application or program. A <patterndb> element may contain any number of <ruleset> elements.
Attributes
-
name: The name of the application. Note that the function of this attribute is to make the database more readable, syslog-ng uses the <pattern> element to identify the applications sending log messages.
-
id: A unique ID of the application, for example, the md5 sum of the name attribute.
Children
-
patterns
-
rules
-
actions
-
tags
-
description: OPTIONAL — A description of the ruleset or the application.
-
url: OPTIONAL — An URL referring to further information about the ruleset or the application.
-
rule_urls: OPTIONAL — To list multiple URLs referring to further information about the ruleset or the application, enclose the <url> elements into an <urls> element.
Example
<ruleset name='su' id='480de478-d4a6-4a7f-bea4-0c0245d361e1'>
6.4.3.3 - Element: patterns
Location
/patterndb/ruleset/patterns
Description
A container element. A <patterns> element may contain any number of `elements.
Attributes
N/A
Children
-
pattern: The name of the application — syslog-ng matches this value to the ${PROGRAM} header of the syslog message to find the rulesets applicable to the syslog message.
Specifying multiple patterns is useful if two or more applications have different names (that is, different ${PROGRAM} fields), but otherwise send identical log messages.
It is not necessary to use multiple patterns if only the end of the ${PROGRAM} fields is different, use only the beginning of the ${PROGRAM} field as the pattern. For example, the Postfix email server sends messages using different process names, but all of them begin with the postfix string.
You can also use parsers in the program pattern if needed, and use the parsed results later. For example: `stfix\@ESTRING:.postfix.component:[@»
Note
If the `
element of a ruleset is not specified, AxoSyslog will use this ruleset as a fallback ruleset: it will apply the ruleset to messages that have an empty PROGRAM header, or if none of the program patterns matched the PROGRAM header of the incoming message.
Example
<patterns>
<pattern>firstapplication</pattern>
<pattern>otherapplication</pattern>
</patterns>
Using parsers in the program pattern:
<pattern>postfix\@ESTRING:.postfix.component:[@</pattern>
6.4.3.4 - Element: rules
Location
/patterndb/ruleset/rules
Description
A container element for the rules of the ruleset.
Attributes
N/A
Children
Example
<rules>
<rule provider='me' id='182437592347598' class='system'>
<patterns>
<pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
</patterns>
</rule>
</rules>
6.4.3.5 - Element: rule
Location
/patterndb/ruleset/rules/rule
Description
An element containing message patterns and how a message that matches these patterns is classified.
Note
If the following characters appear in the message, they must be escaped in the rule as follows:
The element may contain any number of elements.
Attributes
-
provider: The provider of the rule. This is used to distinguish between who supplied the rule, that is, if it has been created by Axoflow, or added to the XML by a local user.
-
id: The globally unique ID of the rule.
-
class: The class of the rule — this class is assigned to the messages matching a pattern of this rule.
Children
Example
<rule provider='example' id='f57196aa-75fd-11dd-9bba-001e6806451b' class='violation'>
The following example specifies attributes for correlating messages as well. For details on correlating messages, see Correlating log messages using pattern databases.
<rule provider='example' id='f57196aa-75fd-11dd-9bba-001e6806451b' class='violation' context-id='same-session' context-scope='process' context-timeout='360'>
6.4.3.6 - Element: patterns
Location
/patterndb/ruleset/rules/rule/patterns
Description
An element containing the patterns of the rule. If a element contains multiple *lements, the class of the *»>ssigned to every syslog message matching any of the patterns.
Attributes
N/A
Children
-
pattern: A pattern describing a log message. This element is also called message pattern. For example:
<pattern>+ ??? root-</pattern>
Note
Support for XML entities is limited, you can use only the following entities: \& \< \> \" \'. User-defined entities are not supported.
-
description: OPTIONAL — A description of the pattern or the log message matching the pattern.
-
urls
-
values
-
examples
Example
<patterns>
<pattern>Accepted @QSTRING:SSH.AUTH_METHOD: @ for@QSTRING:SSH_USERNAME: @from\ @QSTRING:SSH_CLIENT_ADDRESS: @port @NUMBER:SSH_PORT_NUMBER:@ ssh2</pattern>
</patterns>
6.4.3.7 - Element: urls
Location
/patterndb/ruleset/rules/rule/patterns/urls
Description
OPTIONAL — An element containing one or more URLs referring to further information about the patterns or the matching log messages.
Attributes
N/A
Children
- url: OPTIONAL — An URL referring to further information about the patterns or the matching log messages.
Example
N/A
6.4.3.8 - Element: values
Location
/patterndb/ruleset/rules/rule/patterns/values
Description
OPTIONAL — Name-value pairs that are assigned to messages matching the patterns, for example, the representation of the event in the message according to the Common Event Format (CEF) or Common Event Exchange (CEE). The names can be used as macros to reference the assigned values.
Attributes
N/A
Children
-
value: OPTIONAL — Contains the value of the name-value pair that is assigned to the message.
The <value> element of name-value pairs can include template functions. For details, see Using template functions, for examples, see if.
You can associate types with values using the "type" attribute, for example, integer is a type-cast that associates $foobar with an integer type. For details on data types, see Specifying data types in value-pairs.
<value name="foobar" type="integer">$PID</value>
db-parser()’s internal parsers (for example, @NUMBER@) automatically associate type information to the parsed name-value pair.
When used together with message correlation, the <value> element of name-value pairs can include references to the values of earlier messages from the same context. For details, see Correlating log messages using pattern databases.
-
name: The name of the name-value pair. It can also be used as a macro to reference the assigned value.
Example
<values>
<value name=".classifier.outcome">/Success</value>
</values>
6.4.3.9 - Element: actions
Location
/patterndb/ruleset/actions
Description
OPTIONAL — A container element for actions that are performed if a message is recognized by the pattern. For details on actions, see Triggering actions for identified messages.
Attributes
N/A
Children
Example
Example: Generating messages for pattern database matches
When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.
<actions>
<action>
<message>
<values>
<value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
</values>
</message>
</action>
</actions>
To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.
Example: Generating messages with inherited values
The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name
<actions>
<action>
<message inherit-properties='TRUE'>
<values>
<value name="PROGRAM">overriding-original-program-name</value>
</values>
</message>
</action>
</actions>
6.4.3.10 - Element: action
Location
/patterndb/ruleset/actions/action
Description
OPTIONAL — A container element describing an action that is performed when a message matching the rule is received.
Attributes
-
condition: An AxoSyslog filter expression. The action is performed only if the message matches the filter. The filter can include macros and name-value pairs extracted from the message. When using actions together with message-correlation, you can also use the $(context-length) macro, which returns the number of messages in the current context. For example, this can be used to determine if the expected number of messages has arrived to the context: condition='"$(context-length)" > "5"'
-
rate: Specifies maximum how many messages should be generated in the specified time period in the following format: <number-of-messages>/<period-in-seconds>. For example: 1/60 allows 1 message per minute. Rates apply within the scope of the context, that is, if context-scope="host" and rate="1/60", then maximum one message is generated per minute for every host that sends a log message matching the rule. Excess messages are dropped. Note that when applying the rate to the generated messages, AxoSyslog uses the timestamps of the log messages, similarly to calculating the context-timeout. That way rate is applied correctly even if the log messages are processed offline.
-
trigger: Specifies when the action is executed. The trigger attribute has the following possible values:
-
match: Execute the action immediately when a message matching the rule is received.
-
timeout: Execute the action when the correlation timer (context-timeout) of the pattern database rule expires. This is available only if actions are used together with correlating messages.
Children
-
create-context
-
message: A container element storing the message to be sent when the action is executed. Currently AxoSyslog sends these messages to the internal() destination.
-
For details on the message context, see Correlating log messages using pattern databases and Actions and message correlation. For details on triggering messages, see Triggering actions for identified messages
inherit-mode: This attribute controls which name-value pairs and tags are propagated to the newly generated message.
-
context: AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.
-
last-message: Only the name-value pairs appearing in the last message are copied. If the context contains only a single message, then it is the message that triggered the action.
-
none: An empty message is created, without inheriting any tags or name-value pairs.
This option is available in AxoSyslog 3.8 and later.
-
inherit-properties: This attribute is deprecated. Use the inherit-mode attribute instead.
If set to TRUE, the original message that triggered the action is cloned, including its name-value pairs and tags.
If set to context, AxoSyslog collects every name-value pair from each message stored in the context, and includes them in the generated message. If a name-value pair appears in multiple messages of the context, the value in the latest message will be used. Note that tags are not merged, the generated message will inherit the tags assigned to the last message of the context.
For details on the message context, see Correlating log messages using pattern databases and Actions and message correlation. For details on triggering messages, see Triggering actions for identified messages
This option is available in AxoSyslog 5.3.2 and later.
-
values: A container element for values and fields that are used to create the message generated by the action.
-
value: Sets the value of the message field specified in the name attribute of the element. For example, to specify the body of the generated message, use the following syntax:
<value name="MESSAGE">A log message matched rule number $.classifier.rule_id</value>
Note that currently it is not possible to add DATE, FACILITY, or SEVERITY fields to the message.
When the action is used together with message correlation, the AxoSyslog application automatically adds fields to the message based on the context-scope parameter. For example, using context-scope="process" automatically fills the HOST, PROGRAM, and PID fields of the generated message.
-
name: Name of the message field set by the value element.
Example: Generating messages for pattern database matches
When inserted in a pattern database rule, the following example generates a message when a message matching the rule is received.
<actions>
<action>
<message>
<values>
<value name="MESSAGE">A log message from ${HOST} matched rule number $.classifier.rule_id</value>
</values>
</message>
</action>
</actions>
To inherit the properties and values of the triggering message, set the inherit-properties attribute of the <message> element to TRUE. That way the triggering log message is cloned, including name-value pairs and tags. If you set any values for the message in the <action> element, they will override the values of the original message.
Example: Generating messages with inherited values
The following action generates a message that is identical to the original message, but its $PROGRAM field is set to overriding-original-program-name
<actions>
<action>
<message inherit-properties='TRUE'>
<values>
<value name="PROGRAM">overriding-original-program-name</value>
</values>
</message>
</action>
</actions>
6.4.3.11 - Element: create-context
Location
/patterndb/ruleset/actions/action/create-context
Description
OPTIONAL — Creates a new correlation context from the current message and its associated context. This can be used to “split” a context.
Available in AxoSyslog version 3.8 and later.
Attributes
Children
Example
The following example creates a new context whenever the rule matches. The new context receives 1000 as ID, and program as scope, and the content set in the <message> element of the >element.
<rule provider='test' id='12' class='violation'>
<patterns>
<pattern>simple-message-with-action-to-create-context</pattern>
</patterns>
<actions>
<action trigger='match'>
<create-context context-id='1000' context-timeout='60' context-scope='program'>
<message inherit-properties='context'>
<values>
<value name='MESSAGE'>context message</value>
</values>
</message>
</create-context>
</action>
</actions>
</rule>
6.4.3.12 - Element: tags
Location
/patterndb/ruleset/tags
Description
OPTIONAL — An element containing custom keywords (tags) about the messages matching the patterns. The tags can be used to label specific events (for example, user logons). It is also possible to filter on these tags later (for details, see Tagging messages). Starting with AxoSyslog 3.2, the list of tags assigned to a message can be referenced with the ${TAGS} macro.
Attributes
N/A
Children
- tag: OPTIONAL — A keyword or tags applied to messages matching the rule.
Example
<tags><tag>UserLogin</tag></tags>
6.4.3.13 - Element: example
Location
/patterndb/ruleset/rules/rule/patterns/examples/example
Description
OPTIONAL — A container element for a sample log message.
Attributes
N/A
Children
Example
<examples>
<example>
<test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>
<test_values>
<test_value name="SSH_AUTH_METHOD">password</test_value>
<test_value name="SSH_USERNAME">sampleuser</test_value>
<test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>
<test_value name="SSH_PORT_NUMBER" type="integer">42156</test_value>
</test_values>
</example>
</examples>
6.4.3.14 - Element: examples
Location
/patterndb/ruleset/rules/rule/patterns/examples
Description
OPTIONAL — A container element for sample log messages that should be recognized by the pattern. These messages can be used also to test the patterns and the parsers.
Attributes
N/A
Children
Example
<examples>
<example>
<test_message>Accepted password for sampleuser from 10.50.0.247 port 42156 ssh2</test_message>
<test_values>
<test_value name="SSH.AUTH_METHOD">password</test_value>
<test_value name="SSH_USERNAME">sampleuser</test_value>
<test_value name="SSH_CLIENT_ADDRESS">10.50.0.247</test_value>
<test_value name="SSH_PORT_NUMBER">42156</test_value>
</test_values>
</example>
</examples>
7 - Parsing enterprise-wide message model (EWMM) messages
The ewmm-parser() can be used to parse messages sent by another AxoSyslog host using the enterprise-wide message model (EWMM) format. Available in version 3.16 and later. Note that usually you do not have to use this parser directly, because the default-network-drivers() source automatically parses such messages.
Declaration:
parser parser_name {
ewmm-parser();
};
8 - Fortigate parser
The Fortigate parser can parse the log messages of FortiGate/FortiOS (Fortigate Next-Generation Firewall (NGFW)). These messages do not completely comply with the syslog RFCs, making them difficult to parse. The fortigate-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs. For details on using value-pairs, see Structuring macros, metadata, and other value-pairs. The parser can parse messages in the following format:
For example:
<189>date=2021-01-15 time=12:58:59 devname="FORTI_111" devid="FG100D3G12801312" logid="0001000014" type="traffic" subtype="local" level="notice" vd="root" eventtime=1610704739683510055 tz="+0300" srcip=91.234.154.139 srcname="91.234.154.139" srcport=45295 srcintf="wan1" srcintfrole="wan" dstip=213.59.243.9 dstname="213.59.243.9" dstport=46730 dstintf="unknown0" dstintfrole="undefined" sessionid=2364413215 proto=17 action="deny" policyid=0 policytype="local-in-policy" service="udp/46730" dstcountry="Russian Federation" srccountry="Russian Federation" trandisp="noop" app="udp/46730" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=5 craction=262144 crlevel="low"
If you find a message that the fortigate-parser() cannot properly parse, contact us, so we can improve the parser.
By default, the Fortigate-specific fields are extracted into name-value pairs prefixed with .fortigate. For example, the devname in the previous message becomes ${.fortigate.devname}. You can change the prefix using the prefix option of the parser.
Declaration:
@version: 4.9.0
@include "scl.conf"
log {
source { network(transport("udp") flags(no-parse)); };
parser { fortigate-parser(); };
destination { ... };
};
Note that you have to disable message parsing in the source using the flags(no-parse) option for the parser to work.
The fortigate-parser() is actually a reusable configuration snippet configured to parse Fortigate messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, websense-parser() uses the .websense. prefix. To modify it, use the following format:
parser {
websense-parser(prefix("myprefix."));
};
8.1 - Fortigate parser options
The fortigate-parser() has the following options:
prefix()
|
|
| Synopsis: |
prefix() |
| Default: |
“.panos.” |
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, fortigate-parser() uses the .fortigate. prefix. To modify it, use the following format:
parser {
fortigate-parser(prefix("myprefix."));
};
template()
|
|
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
9 - group-lines parser
Available in AxoSyslog version 4.2 and newer.
The group-lines() parser correlates multi-line messages received as separate, but subsequent lines into a single log message. AxoSyslog first collects the received messages into streams of related messages (based on the key() parameter), then grouped into correlation contexts up to timeout() seconds long. Multi-line messages are then identified within these contexts.
group-lines(
key("$FILE_NAME")
multi-line-mode("smart")
template("$MESSAGE")
timeout(10)
line-separator("\n")
);
The parser has the following options.
key()
Description: Specifies a template that determines which messages form a single stream. Messages where the template expansion results in the same key are considered part of the same stream. Using the key() option, you can extract multi-line messages even if different streams are interleaved in your input.
line-separator()
Description: In case a multi-line message is found, this string is inserted between the of the new multi-line message. Defaults to the newline character.
multi-line-garbage()
|
|
| Type: |
regular expression |
| Default: |
empty string |
Description: Use the multi-line-garbage() option when processing multi-line messages that contain unneeded parts between the messages. Specify a string or regular expression that matches the beginning of the unneeded message parts. If the multi-line-garbage() option is set, AxoSyslog ignores the lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix(). See also the multi-line-prefix() option.
When receiving multi-line messages from a source when the multi-line-garbage() option is set, but no matching line is received between two lines that match multi-line-prefix(), AxoSyslog will continue to process the incoming lines as a single message until a line matching multi-line-garbage() is received.
To use the multi-line-garbage() option, set the multi-line-mode() option to prefix-garbage.
Warning
If the multi-line-garbage() option is set, AxoSyslog discards lines between the line matching the multi-line-garbage() and the next line matching multi-line-prefix().
multi-line-mode()
|
|
| Type: |
indented, prefix-garbage, prefix-suffix, regexp, smart |
| Default: |
empty string |
Description: Use the multi-line-mode() option when processing multi-line messages. The AxoSyslog application provides the following methods to process multi-line messages:
-
indented: The indented mode can process messages where each line that belongs to the previous line is indented by whitespace, and the message continues until the first non-indented line. For example, the Linux kernel (starting with version 3.5) uses this format for /dev/log, as well as several applications, like Apache Tomcat.
source s_tomcat {
file("/var/log/tomcat/xxx.log" multi-line-mode(indented));
};
-
prefix-garbage: The prefix-garbage mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. For details on using multi-line-mode(prefix-garbage), see the multi-line-prefix() and multi-line-garbage() options.
-
prefix-suffix: The prefix-suffix mode uses a string or regular expression (set in multi-line-prefix()) that matches the beginning of the log messages, ignores newline characters from the source until a line matches the regular expression set in multi-line-suffix(), and treats the lines between multi-line-prefix() and multi-line-suffix() as a single message. Any other lines between the end of the message and the beginning of a new message (that is, a line that matches the multi-line-prefix() expression) are discarded. For details on using multi-line-mode(prefix-suffix), see the multi-line-prefix() and multi-line-suffix() options.
The prefix-suffix mode is similar to the prefix-garbage mode, but it appends the garbage part to the message instead of discarding it.
-
smart: The smart mode recognizes multi-line data backtraces even if they span multiple lines in the input. The backtraces are converted to a single log message for easier analysis. Backtraces for the following programming languages are recognized : Python, Java, JavaScript, PHP, Go, Ruby, and Dart.
smart mode is available in AxoSyslog version 4.2 and newer.
The regular expressions to recognize these programming languages are specified in an external file called /usr/share/syslog-ng/smart-multi-line.fsm (installation path depends on configure arguments), in a format that is described in that file.
Note
To format multi-line messages to your individual needs, consider the following:
-
To make multi-line messages more readable when written to a file, use a template in the destination and instead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). This expression inserts a tab after every newline character (except when a tab is already present), indenting every line of the message after the first. For example:
destination d_file {
file ("/var/log/messages"
template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n")
);
};
For details on using templates, see Templates and macros.
-
To actually convert the lines of multi-line messages to single line (by replacing the newline characters with whitespaces), use the flags(no-multi-line) option in the source.
multi-line-prefix()
|
|
| Type: |
regular expression starting with the ^ character |
| Default: |
empty string |
Description: Use the multi-line-prefix() option to process multi-line messages, that is, log messages that contain newline characters (for example, Tomcat logs). Specify a string or regular expression that matches the beginning of the log messages (always start with the ^ character). Use as simple regular expressions as possible, because complex regular expressions can severely reduce the rate of processing multi-line messages. If the multi-line-prefix() option is set, AxoSyslog ignores newline characters from the source until a line matches the regular expression again, and treats the lines between the matching lines as a single message. See also the multi-line-garbage() option.
Note
To format multi-line messages to your individual needs, consider the following:
-
To make multi-line messages more readable when written to a file, use a template in the destination and instead of the ${MESSAGE} macro, use the following: $(indent-multi-line ${MESSAGE}). This expression inserts a tab after every newline character (except when a tab is already present), indenting every line of the message after the first. For example:
destination d_file {
file ("/var/log/messages"
template("${ISODATE} ${HOST} $(indent-multi-line ${MESSAGE})\n")
);
};
For details on using templates, see Templates and macros.
-
To actually convert the lines of multi-line messages to single line (by replacing the newline characters with whitespaces), use the flags(no-multi-line) option in the source.
Example: Processing Tomcat logs
The log messages of the Apache Tomcat server are a typical example for multi-line log messages. The messages start with the date and time of the query in the YYYY.MM.DD HH:MM:SS format, as you can see in the following example.
2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
SEVERE: Catalina.start:
LifecycleException: service.getName(): "Catalina"; Protocol handler start failed: java.net.BindException: Address already in use null:8080
at org.apache.catalina.connector.Connector.start(Connector.java:1138)
at org.apache.catalina.core.StandardService.start(StandardService.java:531)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:710)
at org.apache.catalina.startup.Catalina.start(Catalina.java:583)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:288)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:177)
2010.06.09. 12:07:39 org.apache.catalina.startup.Catalina start
INFO: Server startup in 1206 ms
2010.06.09. 12:45:08 org.apache.coyote.http11.Http11Protocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8080
2010.06.09. 12:45:09 org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina
To process these messages, specify a regular expression matching the timestamp of the messages in the multi-line-prefix() option. Such an expression is the following:
source s_file{file("/var/log/tomcat6/catalina.2010-06-09.log" follow-freq(0) multi-line-mode(regexp) multi-line-prefix("[0-9]{4}\.[0-9]{2}\.[0-9]{2}\.") flags(no-parse));};
};
Note that flags(no-parse) is needed to prevent AxoSyslog trying to interpret the date in the message.
multi-line-suffix()
|
|
| Type: |
regular expression |
| Default: |
empty string |
Description: Use the multi-line-suffix() option when processing multi-line messages. Specify a string or regular expression that matches the end of the multi-line message.
To use the multi-line-suffix() option, set the multi-line-mode() option to prefix-suffix. See also the multi-line-prefix() option.
scope()
|
|
| Type: |
process, program, host, or global |
| Default: |
global |
Description: Specifies which messages belong to the same context. The following values are available:
process: Only messages that are generated by the same process of a client belong to the same context, that is, messages that have identical ${HOST}, ${PROGRAM} and ${PID} values.program: Messages that are generated by the same application of a client belong to the same context, that is, messages that have identical ${HOST} and ${PROGRAM} values.host: Every message generated by a client belongs to the same context, only the ${HOST} value of the messages must be identical.global: Every message belongs to the same context. This is the default value.
template()
Description: A template string that specifies what constitutes an line to group-lines(). In simple cases this is ${MSG} or ${RAWMSG}.
10 - iptables parser
The iptables parser can parse the log messages of the iptables command. Available in version 3.16 and later.
Declaration:
@version: 4.9
@include "scl.conf"
log {
source { system(); };
parser { iptables-parser(); };
destination { ... };
};
The iptables-parser() is actually a reusable configuration snippet configured to parse iptables messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, iptables-parser() uses the .iptables. prefix. To modify it, use the following format:
parser {
iptables-parser(prefix("myprefix."));
};
11 - JSON parser
JavaScript Object Notation (JSON) is a text-based open standard designed for human-readable data interchange. It is used primarily to transmit data between a server and web application, serving as an alternative to XML. It is described in RFC 4627. The AxoSyslog application can separate parts of incoming JSON-encoded log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs.
You can refer to the separated parts of the JSON message using the key of the JSON object as a macro. For example, if the JSON contains {"KEY1":"value1","KEY2":"value2"}, you can refer to the values as ${KEY1} and ${KEY2}. If the JSON content is structured, AxoSyslog converts it to dot-notation-format. For example, to access the value of the following structure {"KEY1": {"KEY2": "VALUE"}}, use the ${KEY1.KEY2} macro.
Warning
If the names of keys in the JSON content are the same as the names of AxoSyslog soft macros, the value from the JSON content will overwrite the value of the macro. For example, the {"PROGRAM":"value1","MESSAGE":"value2"} JSON content will overwrite the ${PROGRAM} and ${MESSAGE} macros. To avoid overwriting such macros, use the prefix() option.
Hard macros cannot be modified, so they will not be overwritten. For details on the macro types, see Hard versus soft macros.
Note
When using the json-parser(), AxoSyslog converts all elements of the JSON object to name-value pairs. Any type information originally present in the incoming JSON object is retained, and automatically propagated to other AxoSyslog components (for example, a destination) if they support types.
- Elements without a type are treated as strings.
- JSON lists (arrays) are converted to AxoSyslog lists, so you can manipulate them using the
$(list-*) template functions.
Note that prior to version 4.0, AxoSyslog handled every data as strings.
The JSON parser discards messages if it cannot parse them as JSON messages, so it acts as a JSON-filter as well.
To create a JSON parser, define a parser that has the json-parser() option. Defining the prefix and the marker are optional. By default, the parser will process the ${MESSAGE} part of the log message. To process other parts of a log message with the JSON parser, use the template() option. You can also define the parser inline in the log path.
Declaration:
parser parser_name {
json-parser(
marker()
prefix()
);
};
Example: Using a JSON parser
In the following example, the source is a JSON encoded log message. The syslog parser is disabled, so that AxoSyslog does not parse the message: flags(no-parse). The json-parser inserts “.json.” prefix before all extracted name-value pairs. The destination is a file that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination and the parser.
source s_json {
network(
port(21514
flags(no-parse)
);
};
destination d_json {
file(
"/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n")
);
};
parser p_json {
json-parser (prefix(".json."));
};
log {
source(s_json);
parser(p_json);
destination(d_json);
};
You can also define the parser inline in the log path.
source s_json {
network(
port(21514)
flags(no-parse)
);
};
destination d_json {
file(
"/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n")
);
};
log {
source(s_json);
parser {
json-parser (prefix(".json."));
};
destination(d_json);
};
11.1 - Options of JSON parsers
The JSON parser has the following options.
|
|
| Synopsis: |
extract-prefix() |
Description: Extract only the specified subtree from the JSON message. Use the dot-notation to specify the subtree. The rest of the message will be ignored. For example, assuming that the incoming object is named msg, the json-parser(extract-prefix("foo.bar[5]")); parser is equivalent to the msg.foo.bar[5] javascript code. Note that the resulting expression must be a JSON object in order to extract its members into name-value pairs.
This feature also works when the top-level object is an array, because you can use an array index at the first indirection level, for example: json-parser(extract-prefix("[5]")), which is equivalent to msg[5].
In addition to alphanumeric characters, the key of the JSON object can contain the following characters: \!"#$%&'()*+,-/:;<=>?@\\^_{|}~`
It cannot contain the following characters: .[]
The following parser converts messages in the logstash eventlog v0 format to the v1 format.
parser p_jsoneventv0 {
channel {
parser {
json-parser(extract-prefix("@fields"));
};
parser {
json-parser(prefix(".json."));
};
rewrite {
set("1" value("@version"));
set("${.json.@timestamp}" value("@timestamp"));
set("${.json.@message}" value("message"));
};
};
};
key-delimiter()
|
|
| Type: |
character |
| Default: |
. |
Description: The delimiter character to use when parsing flattened keys. Supports Only single characters.
marker
Description: Use a marker in case of mixed log messages, to identify JSON encoded messages for the parser.
Some logging implementations require a marker to be set before the JSON payload. The JSON parser is able to find these markers and parse the message only if it is present.
Example: Using the marker option in JSON parser
This json parser parses log messages which use the “@cee:” marker in front of the json payload. It inserts “.cee.” in front of the name of name-value pairs, so later on it is easier to find name-value pairs that were parsed using this parser. (For details on selecting name-value pairs, see value-pairs().)
parser {
json-parser(
marker("@cee:")
prefix(".cee.")
);
};
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
This parser does not have a default prefix. To configure a custom prefix, use the following format:
parser {
json-parser(prefix("myprefix."));
};
template()
|
|
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
12 - Parsing key=value pairs
The AxoSyslog application can separate a message consisting of whitespace or comma-separated key=value pairs (for example, Postfix log messages) into name-value pairs. You can also specify other separator character instead of the equal sign, for example, colon (:) to parse MySQL log messages. The AxoSyslog application automatically trims any leading or trailing whitespace characters from the keys and values, and also parses values that contain unquoted whitespace. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs.
You can refer to the separated parts of the message using the key of the value as a macro. For example, if the message contains KEY1=value1,KEY2=value2, you can refer to the values as ${KEY1} and ${KEY2}.
Note
If a log message contains the same key multiple times (for example, key1=value1, key2=value2, key1=value3, key3=value4, key1=value5), then AxoSyslog only stores the last (rightmost) value for the key. Using the previous example, AxoSyslog will store the following pairs: key1=value5, key2=value2, key3=value4.
Warning
If the names of keys in the message is the same as the names of AxoSyslog soft macros, the value from the parsed message will overwrite the value of the macro. For example, the PROGRAM=value1, MESSAGE=value2 content will overwrite the ${PROGRAM} and ${MESSAGE} macros. To avoid overwriting such macros, use the prefix() option.
Hard macros cannot be modified, so they will not be overwritten. For details on the macro types, see Hard versus soft macros.
The parser discards message sections that are not key=value pairs, even if they appear between key=value pairs that can be parsed.
The names of the keys can contain only the following characters: numbers (0-9), letters (a-z,A-Z), underscore (_), dot (.), hyphen (-). Other special characters are not permitted.
To parse key=value pairs, define a parser that has the kv-parser() option. Defining the prefix is optional. By default, the parser will process the ${MESSAGE} part of the log message. You can also define the parser inline in the log path.
Declaration:
parser parser_name {
kv-parser(
prefix()
);
};
Example: Using a key=value parser
In the following example, the source is a log message consisting of comma-separated key=value pairs, for example, a Postfix log message:
Jun 20 12:05:12 mail.example.com <info> postfix/qmgr[35789]: EC2AC1947DA: from=<me@example.com>, size=807, nrcpt=1 (queue active)
The kv-parser inserts the “.kv.” prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination and the parser.
source s_kv {
network(port(21514));
};
destination d_json {
file("/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n"));
};
parser p_kv {
kv-parser (prefix(".kv."));
};
log {
source(s_kv);
parser(p_kv);
destination(d_json);
};
You can also define the parser inline in the log path.
source s_kv {
network(port(21514));
};
destination d_json {
file("/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n"));
};
log {
source(s_kv);
parser {
kv-parser (prefix(".kv."));
};
destination(d_json);
};
You can set the separator character between the key and the value to parse for example, key:value pairs, like MySQL logs:
Mar 7 12:39:25 myhost MysqlClient[20824]: SYSTEM_USER:'oscar', MYSQL_USER:'my_oscar', CONNECTION_ID:23, DB_SERVER:'127.0.0.1', DB:'--', QUERY:'USE test;'
parser p_mysql {
kv-parser(value-separator(":") prefix(".mysql."));
};
12.1 - Options of key=value parsers
The kv-parser has the following options.
|
|
| Synopsis: |
extract-stray-words-into(") |
Description: Specifies the name-value pair where AxoSyslog stores any stray words that appear before or between the parsed key-value pairs (mainly when the pair-separator() option is also set). If multiple stray words appear in a message, then AxoSyslog stores them as a comma-separated list. Note that the prefix() option does not affect the name-value pair storing the stray words. Default value:N/A
For example, consider the following message:
VSYS=public; Slot=5/1; protocol=17; source-ip=10.116.214.221; source-port=50989; destination-ip=172.16.236.16; destination-port=162;time=2016/02/18 16:00:07; interzone-emtn_s1_vpn-enodeb_om; inbound; policy=370;
This is a list of key-value pairs, where the value separator is = and the pair separator is ;. However, before the last key-value pair (policy=370), there are two stray words: interzone-emtn_s1_vpn-enodeb_om inbound. If you want to store or process these, specify a name-value pair to store them in the extract-stray-words-into() option, for example, extract-stray-words-into("my-stray-words"). The value of ${my-stray-words} for this message will be interzone-emtn_s1_vpn-enodeb_om, inbound
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, kv-parser() uses the .kv. prefix. To modify it, use the following format:
parser {
kv-parser(prefix("myprefix."));
};
pair-separator()
|
|
| Synopsis: |
pair-separator(") |
Description: Specifies the character or string that separates the key-value pairs from each other. Default value: , .
For example, to parse key1=value1;key2=value2 pairs, use kv-parser(pair-separator(";")); .
template()
|
|
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
value-separator()
|
|
| Synopsis: |
value-separator("") |
Description: Specifies the character that separates the keys from the values. Default value: =.
For example, to parse key:value pairs, use kv-parser(value-separator(":"));.
13 - Linux audit parser
The Linux audit parser can parse the log messages of the Linux Audit subsystem (auditd). The AxoSyslog application can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The following is a sample log message of auditd:
type=SYSCALL msg=audit(1441988805.991:239): arch=c000003e syscall=59 success=yes exit=0 a0=7fe49a6d0e98 a1=7fe49a6d0e40 a2=7fe49a6d0e80 a3=2 items=2 ppid=3652 pid=3660 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=5 comm="dumpe2fs" exe="/sbin/dumpe2fs" key=(null)
type=EXECVE msg=audit(1441988805.991:239): argc=3 a0="dumpe2fs" a1="-h" a2="/dev/sda1"
type=CWD msg=audit(1441988805.991:239): cwd="/"
type=PATH msg=audit(1441988805.991:239): item=0 name="/sbin/dumpe2fs" inode=137078 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=PATH msg=audit(1441988805.991:239): item=1 name="/lib64/ld-linux-x86-64.so.2" inode=5243184 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
type=PROCTITLE msg=audit(1441988805.991:239): proctitle=64756D7065326673002D68002F6465762F73646131
Certain fields of the audit log can be encoded in hexadecimal format, for example, the arch field, or the a<number> fields in the previous example. The AxoSyslog application automatically decodes these fields (for example, the c000003e value becomes x86_64).
The AxoSyslog application extracts every field into name-value pairs. It automatically decodes the following fields:
-
name
-
proctitle
-
path
-
dir
-
comm
-
ocomm
-
data
-
old
-
new
To parse the log messages of the Linux Audit subsystem, define a parser that has the linux-audit-parser() option. By default, the parser will process the ${MESSAGE} part of the log message. To process other parts of a log message, use the template() option. You can also define the parser inline in the log path.
Declaration:
parser parser_name {
linux-audit-parser(
prefix()
template()
);
};
Example: Using the linux-audit-parser() parser
In the following example, the source is a log file created by auditd. Since the audit log format is not a syslog format, the syslog parser is disabled, so that AxoSyslog does not parse the message: flags(no-parse). The parser inserts “.auditd.” prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination, and the parser.
source s_auditd {
file(/var/log/audit/audit.log flags(no-parse));
};
destination d_json {
file(
"/tmp/test.json"
template("$(format-json .auditd.*)\n")
);
};
parser p_auditd {
linux-audit-parser (prefix(".auditd."));
};
log {
source(s_auditd);
parser(p_auditd);
destination(d_json);
};
You can also define the parser inline in the log path.
source s_auditd {
file(/var/log/audit/audit.log);
};
destination d_json {
file(
"/tmp/test.json"
template("$(format-json .auditd.*)\n")
);
};
log {
source(s_auditd);
parser {
linux-audit-parser (prefix(".auditd."));
};
destination(d_json);
};
13.1 - Options of linux-audit-parser() parsers
The linux-audit-parser() has the following options.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, linux-audit-parser() uses the .auditd. prefix. To modify it, use the following format:
parser {
linux-audit-parser(prefix("myprefix."));
};
template()
|
|
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
14 - MariaDB parser
The MariaDB parser can parse the log messages of the MariaDB Audit Plugin. The parser supports the syslog output typess’ format. Available in version 3.37 and later.
Declaration:
@version: 4.9
@include "scl.conf"
log {
source { system(); };
parser { mariadb-audit-parser(); };
destination { ... };
};
The mariadb-audit is a reusable configuration snippet configured to parse MariaDB Audit Plugin messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, mariadb-audit uses the .mariadb. prefix. To modify it, use the following format:
parser {
mariadb-audit-parser(prefix("myprefix."));
};
15 - metrics-probe
Available in AxoSyslog version 4.1.1 and newer.
metrics-probe() is a special parser that counts the messages that pass through the log path, and creates labeled stats counters based on the fields of the passing messages.
You can configure the name of the keys and the labels. Note that the keys are automatically prefixed with the syslogng_ string. You can use templates in the values of the labels.
The minimal configuration creates counters with the key syslogng_classified_events_total and labels app, host, program and source. For example:
parser p_metrics_probe {
metrics-probe();
# Same as:
#
# metrics-probe(
# key("classified_events_total")
# labels(
# "app" => "${APP}"
# "host" => "${HOST}"
# "program" => "${PROGRAM}"
# "source" => "${SOURCE}"
# )
# );
};
This configuration results in counters like:
syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
syslogng_classified_events_total{app="example-app", host="localhost", program="bar", source="s_local_1"} 1
syslogng_classified_events_total{app="example-app", host="localhost", program="foo", source="s_local_1"} 1
You can query the metrics by running the following command:
syslog-ng-ctl stats prometheus
For example, the following metrics-probe() parser creates a counter called syslogng_custom_key that counts messages that have their custom_label_name_1 field set to foobar, and for these messages it creates separate counters based on the value of the custom_label_name_2 field.
parser p_metrics_probe {
metrics-probe(
key("custom_key") # adds "syslogng_" prefix => "syslogng_custom_key"
labels(
"custom_label_name_1" => "foobar"
"custom_label_name_2" => "${.custom.field}"
)
);
};
This configuration results in counters like:
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="bar"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="foo"} 1
syslogng_custom_key{custom_label_name_1="foobar", custom_label_name_2="baz"} 3
Starting with AxoSyslog 4.4, you can create dynamic labels as well.
Options
increment()
|
|
| Type: |
integer or template |
| Default: |
1 |
Available in AxoSyslog version 4.2 and newer.
Sets a template, which resolves to a number that defines the increment of the counter. The following example defines a counter called syslogng_input_event_bytes_total, and increases its value with the size of the incoming message (in bytes).
metrics-probe(
key("input_event_bytes_total")
labels(
"cluster" => "`cluster-name`"
"driver" => "kubernetes"
"id" => "${SOURCE}"
"namespace" => "${`prefix`namespace_name}"
"pod" => "${`prefix`pod_name}"
)
increment("${RAWMSG_SIZE}")
);
key()
|
|
| Type: |
string |
| Default: |
classified_events_total |
The name of the counter to create. Note that the value of this option is always prefixed with syslogng_, so for example key("my-custom-key") becomes syslogng_my-custom-key.
labels()
|
|
| Type: |
|
| Default: |
See the description |
The labels used to create separate counters, based on the fields of the messages processed by metrics-probe(). Use the following format:
labels(
"name-of-the-label-in-the-output" => "field-of-the-message"
)
Default value:
labels(
"app" => "${APP}"
"host" => "${HOST}"
"program" => "${PROGRAM}"
"source" => "${SOURCE}"
)
This results in counters like:
syslogng_classified_events_total{app="example-app", host="localhost", program="baz", source="s_local_1"} 3
Dynamic labels
Available in AxoSyslog 4.4 and newer.
Dynamic labelling allows you to use every available value-pairs() options in the labels, for example, key(), rekey(), pair(), or scope().
For example:
metrics-probe(
key("foo")
labels(
"static-label" => "bar"
key(".my_prefix.*" rekey(shift-levels(1)))
)
);
syslogng_foo{static_label="bar",my_prefix_baz="anotherlabel",my_prefix_foo="bar",my_prefix_nested_axo="flow"} 4
level()
|
|
| Type: |
integer (0-3) |
| Default: |
0 |
Available in AxoSyslog version 4.2 and newer.
Sets the stats level of the generated metrics.
Note: Drivers configured with internal(yes) register their metrics on level 3. That way if you are creating an SCL, you can disable the built-in metrics of the driver, and create metrics manually using metrics-probe().
16 - Netskope parser
The Netskope parser can parse Netskope log messages. These messages do not completely comply with the syslog RFCs, making them difficult to parse. The netskope-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The parser can parse messages in the following format:
<PRI>{JSON-formatted-log-message}
For example:
<134>{"count": 1, "supporting_data": {"data_values": ["x.x.x.x", "user@domain.com"], "data_type": "user"}, "organization_unit": "domain/domain/Domain Users/Enterprise Users", "severity_level": 2, "category": null, "timestamp": 1547421943, "_insertion_epoch_timestamp": 1547421943, "ccl": "unknown", "user": "user@domain.com", "audit_log_event": "Login Successful", "ur_normalized": "user@domain.com", "_id": "936289", "type": "admin_audit_logs", "appcategory": null}
If you find a message that the netskope-parser() cannot properly parse, contact us, so we can improve the parser.
The AxoSyslog application sets the ${PROGRAM} field to Netskope.
By default, the Netskope-specific fields are extracted into name-value pairs prefixed with .netskope. For example, the organization_unit in the previous message becomes ${.netskope.organization_unit}. You can change the prefix using the prefix option of the parser.
Declaration:
@version: 4.9.0
@include "scl.conf"
log {
source { network(flags(no-parse)); };
parser { netskope-parser(); };
destination { ... };
};
Note that you have to disable message parsing in the source using the flags(no-parse) option for the parser to work.
The netskope-parser() is actually a reusable configuration snippet configured to parse Netskope messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, netskope-parser() uses the .netskope. prefix. To modify it, use the following format:
parser {
netskope-parser(prefix("myprefix."));
};
17 - Parse OpenTelemetry messages
By default, AxoSyslog doesn’t parse the fields of incoming OpenTelemetry messages into name-value pairs, but are only available for forwarding using the opentelemetry() destination. To parse the fields into name-value pairs, use the opentelemetry() parser.
The opentelemetry() parser parses the fields into name-value pairs starting with the .otel. prefix.
- The type of the message is stored in the
.otel.type field (possible values: log, metric, and span).
- Resource information is mapped into the
.otel.resource.<...> , for example, .otel.resource.dropped_attributes_count, or .otel.resource.schema_url.
- Scope information is mapped into
.otel.scope.<...>, for example, .otel.scope.name, .otel.scope.schema_url.
- The fields of log records are mapped into
.otel.log.<...>, for example, .otel.log.body, .otel.log.severity_text.
- The fields of metrics are mapped into
.otel.metric.<...>, for example, .otel.metric.name, .otel.metric.unit.
- The type of the metric is mapped into
.otel.metric.data.type. Possible values: gauge, sum, histogram, exponential_histogram, summary.
- The actual data is mapped into
.otel.metric.data.<type>.<...>, for example, .otel.metric.data.gauge.data_points.0.time_unix_nano.
- The fields of traces are mapped into
.otel.span.<...>, for example, .otel.span.name, .otel.span.trace_state. Repeated fields have an index, for example, .otel.span.events.5.time_unix_nano.
For details on the parsed fields, you can check the OpenTelemetry proto files.
Mapping data types
String, bool, int64, double, and bytes values are mapped to their respective AxoSyslog name-value type, for example, .otel.resource.attributes.string_key becomes a string value.
The mapping of AnyValue type fields is limited.
ArrayValue and KeyValueList types are stored serialized with protobuf type. Note that protobuf and bytes types are only available, unless explicitly type cast. For example, bytes(${.otel.log.span_id}). When using template functions, use --include-bytes, for example, $(format-json .otel.* --include-bytes. In the case of $(format-json), the content is base64-encoded into the bytes content.
Options
set-hostname()
|
|
| Synopsis: |
`yes |
| Default: |
yes |
Available in AxoSyslog 4.8 and later.
Description: If set to yes, the parser extracts the host.name resource attribute if available in the message. Otherwise, it leaves the HOST field as-is.
18 - panos-parser(): parsing PAN-OS log messages
The PAN-OS (a short version of Palo Alto Networks Operating System) parser can parse log messages originating from Palo Alto Networks devices. Even though these messages completely comply to the RFC standards, their MESSAGE part is not a plain text. Instead, the MESSAGE part contains a data structure that requires additional parsing.
The panos-parser() of AxoSyslog solves this problem, and can separate PAN-OS log messages to name-value pairs. For details on using value-pairs in AxoSyslog, see Structuring macros, metadata, and other value-pairs.
Prerequisites
- Version 3.29 of AxoSyslog or later.
- PAN-OS log messages from Palo Alto Networks devices.
Limitations
The panos-parser() only works on AxoSyslog version 3.29 or later.
Configuration
You can include the panos-parser() in your AxoSyslog configuration like this:
parser p_parser{
panos-parser();
};
To use this parser, the scl.conf file must be included in your AxoSyslog configuration:
The panos-parser() is a reusable configuration snippet configured to parse Palo Alto Networks PAN-OS log messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
18.1 - Message format parsed by panos-parser()
This section illustrates the most commonly used PAN-OS log format on the AxoSyslog side.
For information about customizing log format on the PAN-OS side, see the relevant section of the PAN-OS® Administrator’s Guide.
Using the panos-parser(), the parsed messages in AxoSyslog have the following general format:
<PRI><TIMESTAMP> <HOST> <PALO-ALTO-fields-in-CSV-format>
There are several “types” of log formats in Palo Alto Networks PAN-OS. For example, the most commonly used SYSTEM type has the following message format on the AxoSyslog side after parsing:
<12>Apr 14 16:48:54 paloalto.test.net 1,2020/04/14 16:48:54,unknown,SYSTEM,auth,0,2020/04/14 16:48:54,,auth-fail,,0,0,general,medium,failed authentication for user 'admin'. Reason: Invalid username/password. From: 10.0.10.55.,1718,0x0,0,0,0,0,,paloalto
18.2 - PAN-OS parser options
The panos-parser() has the following options:
prefix()
|
|
| Synopsis: |
prefix() |
| Default: |
“.panos.” |
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
19 - PostgreSQL csvlog
Available in AxoSyslog version 4.5.0 and later.
This parser processes messages in the PostgreSQL csvlog format.
The following sample message is a multi-line message with embedded NL characters. This is a single, multi-line log entry that starts with the timestamp.
2023-08-08 12:05:52.805 UTC,,,22113,,64d22fa0.5661,1,,2023-08-08 12:05:52 UTC,23/74060,0,LOG,00000,"automatic vacuum of table ""tablename"": index scans: 0
pages: 0 removed, 4 remain, 0 skipped due to pins, 0 skipped frozen
tuples: 114 removed, 268 remain, 0 are dead but not yet removable, oldest xmin: 149738000
buffer usage: 97 hits, 0 misses, 6 dirtied
avg read rate: 0.000 MB/s, avg write rate: 114.609 MB/s
system usage: CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s",,,,,,,,,""
The postgresql-csvlog-parser() extracts the information from this message into a set of name-value pairs. By default, the name-value pairs have the .pgsql prefix.
@version: current
log {
source { file("/var/log/pgsql.log" follow-freq(1) flags(no-parse)); };
parser { postgresql-csvlog-parser() };
destination { ... };
};
The postgresql-csvlog-parser() driver is actually a reusable configuration snippet configured to parse log messages using the csv-parser(). For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
on-type-error()
Description: Specifies what to do when casting a parsed value to a specific data type fails. Note that the flags(drop-invalid) option and the on-error() global option also affects the behavior.
Accepts the same values as the on-error() global option.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
20 - Python parser
The Python log parser (available in AxoSyslog version 3.10 and later) allows you to write your own parser in Python. Practically, that way you can process the log message (or parts of the log message) any way you need. For example, you can import external Python modules to process the messages, query databases to enrich the messages with additional data, and many other things.
The following points apply to using Python blocks in AxoSyslog in general:
-
Python parsers and template functions are available in AxoSyslog version 3.10 and later.
Python destinations and sources are available in AxoSyslog version 3.18 and later.
-
Supported Python versions: 2.7 and 3.4+ (if you are using pre-built binaries, check the dependencies of the package to find out which Python version it was compiled with).
-
The Python block must be a top-level block in the AxoSyslog configuration file.
-
If you store the Python code in a separate Python file and only include it in the AxoSyslog configuration file, make sure that the PYTHONPATH environment variable includes the path to the Python file, and export the PYTHON_PATH environment variable. For example, if you start AxoSyslog manually from a terminal and you store your Python files in the /opt/syslog-ng/etc directory, use the following command: export PYTHONPATH=/opt/syslog-ng/etc.
In production, when AxoSyslog starts on boot, you must configure your startup script to include the Python path. The exact method depends on your operating system. For recent Red Hat Enterprise Linux, Fedora, and CentOS distributions that use systemd, the systemctl command sources the /etc/sysconfig/syslog-ng file before starting AxoSyslog. (On openSUSE and SLES, /etc/sysconfig/syslog file.) Append the following line to the end of this file: PYTHONPATH="<path-to-your-python-file>", for example, PYTHONPATH="/opt/syslog-ng/etc".
-
The Python object is initiated every time when AxoSyslog is started or reloaded.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
-
The Python block can contain multiple Python functions.
-
Using Python code in AxoSyslog can significantly decrease the performance of AxoSyslog, especially if the Python code is slow. In general, the features of AxoSyslog are implemented in C, and are faster than implementations of the same or similar features in Python.
-
Validate and lint the Python code before using it. The AxoSyslog application does not do any of this.
-
Python error messages are available in the internal() source of AxoSyslog.
-
You can access the name-value pairs of AxoSyslog directly through a message object or a dictionary.
-
To help debugging and troubleshooting your Python code, you can send log messages to the internal() source of AxoSyslog. For details, see Logging from your Python code.
Declaration:
Python parsers consist of two parts. The first is a AxoSyslog parser object that you use in your AxoSyslog configuration, for example, in the log path. This parser references a Python class, which is the second part of the Python parsers. The Python class processes the log messages it receives, and can do virtually anything that you can code in Python.
parser <name_of_the_python_parser>{
python(
class("<name_of_the_python_class_executed_by_the_parser>")
);
};
python {
class MyParser(object):
def init(self, options):
'''Optional. This method is executed when syslog-ng is started or reloaded.'''
return True
def deinit(self):
'''Optional. This method is executed when syslog-ng is stopped or reloaded.'''
pass
def parse(self, msg):
'''Required. This method receives and processes the log message.'''
return True
};
Methods of the python() parser
The init (self, options) method (optional)
The AxoSyslog application initializes Python objects only when it is started or reloaded. That means it keeps the state of internal variables while AxoSyslog is running. The init method is executed as part of the initialization. You can perform any initialization steps that are necessary for your parser to work. For example, if you want to perform a lookup from a file or a database, you can open the file or connect to the database here, or you can initialize a counter that you will increase in the parse() method.
The return value of the init() method must be True. If it returns False, or raises an exception, AxoSyslog will not start.
options: This optional argument contains the contents of the options() parameter of the parser object as a Python dict.
parser my_python_parser{
python(
class("MyParser")
options("regex", "seq: (?P<seq>\\d+), thread: (?P<thread>\\d+), runid: (?P<runid>\\d+), stamp: (?P<stamp>[^ ]+) (?P<padding>.*$)")
);
};
class MyParser(object):
def init(self, options):
pattern = options["regex"]
self.regex = re.compile(pattern)
self.counter = 0
return True
The parse(self, log_message) method
The parse() method processes the log messages it receives, and can do virtually anything that you can code in Python. This method is required, otherwise AxoSyslog will not start.
The return value of the parse() method must be True. If it returns False, or raises an exception, AxoSyslog will drop the message.
-
To reference a name-value pair or a macro in the Python code, use the following format. For example, if the first argument in the definition of the function is called log-message, the value of the HOST macro is log-message['HOST'], and so on. (The log-message contains the entire log message (not just the text body) in a structure similar to a Python dict, but it is actually an object.)
-
You can define new name-value pairs in the Python function. For example, if the first argument in the definition of the function is called log-message, you can create a new name-value pair like this: log_message["new-macro-name"]="value". This is useful when you parse a part of the message from Python, or lookup a value based on data extracted from the log message.
Note that the names of the name-value pairs are case-sensitive. If you create a new name-value pair called new-macro-name in Python, and want to reference it in another part of the AxoSyslog configuration file (for example, in a template), use the ${new-macro-name} macro.
-
You cannot override hard macros (see Hard versus soft macros).
-
To list all available keys (names of name-value pairs), use the log_message.keys() function.
The deinit(self) method (optional)
This method is executed when AxoSyslog is stopped or reloaded.
Warning
If you reload AxoSyslog, existing Python objects are destroyed, therefore the context and state information of Python blocks is lost. Log rotation and updating the configuration of AxoSyslog typically involves a reload.
Example: Parse loggen logs
The following sample code parses the messages of the loggen tool (for details, see The loggen manual page). The following is a sample loggen message:
<38>2017-04-05T12:16:46 localhost prg00000[1234]: seq: 0000000000, thread: 0000, runid: 1491387406, stamp: 2017-04-05T12:16:46 PADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADDPADD
The AxoSyslog parser object references the LoggenParser class and passes a set of regular expressions to parse the loggen messages. The init() method of the LoggenParser class compiles these expressions into a pattern. The parse method uses these patterns to extract the fields of the message into name-value pairs. The destination template of the AxoSyslog log statement uses the extracted fields to format the output message.
@version: 4.9.0
@include "scl.conf"
parser my_python_parser{
python(
class("LoggenParser")
options("regex", "seq: (?P<seq>\\d+), thread: (?P<thread>\\d+), runid: (?P<runid>\\d+), stamp: (?P<stamp>[^ ]+) (?P<padding>.*$)")
);
};
log {
source { tcp(port(5555)); };
parser(my_python_parser);
destination {
file("/tmp/regexparser.log.txt" template("seq: $seq thread: $thread runid: $runid stamp: $stamp my_counter: $MY_COUNTER"));
};
};
python {
import re
class LoggenParser(object):
def init(self, options):
pattern = options["regex"]
self.regex = re.compile(pattern)
self.counter = 0
return True
def deinit(self):
pass
def parse(self, log_message):
match = self.regex.match(log_message['MESSAGE'])
if match:
for key, value in match.groupdict().items():
log_message[key] = value
log_message['MY_COUNTER'] = self.counter
self.counter += 1
return True
return False
};
Example: Parse Windows eventlogs in Python - performance
The following example uses regular expressions to process Windows log messages received in XML format. The parser extracts different fields from messages received from the Security and the Application eventlog containers. Using the following configuration file, AxoSyslog could process about 25000 real-life Windows log messages per second.
@version: 4.9.0
options {
keep-hostname(yes);
keep-timestamp(no);
stats-level(2);
use-dns(no);
};
source s_network_aa5fdf25c39d4017a8e504cdb641b477 {
network(
flags(no-parse)
ip(0.0.0.0)
log-fetch-limit(1000)
log-iw-size(100000)
max-connections(100)
port(514)
);
};
parser p_python_parser_79c31da44bb64de6b5de84be4ae15a15 {
python(options("regex_for_security", ".* Security ID: (?P<security_id>\\S+) Account Name: (?P<account_name>\\S+) Account Domain: (?P<account_domain>\\S+) Logon ID: (?P<logon_id>\\S+).*Process Name: (?P<process_name>\\S+).*EventID (?P<event_id>\\d+)", "regex_others", "(.*)EventID (?P<event_id>\\d+)")
class("EventlogParser"));
};
destination d_file_78363e1dd90c4ebcbb0ee1eff5a2e310 {
file(
"/var/testdb_working_dir/fcd713a2-d48e-4025-9192-ec4a9852cafa.$HOST"
flush-lines(1000)
log-fifo-size(200000)
);
};
log {
source(s_network_aa5fdf25c39d4017a8e504cdb641b477);
parser(p_python_parser_79c31da44bb64de6b5de84be4ae15a15);
destination(d_file_78363e1dd90c4ebcbb0ee1eff5a2e310);
flags(flow-control);
};
python {
import re
class EventlogParser(object):
def init(self, options):
self.regex_security = re.compile(options["regex_for_security"])
self.regex_others = re.compile(options["regex_others"])
return True
def deinit(self):
pass
def parse(self, log_message):
security_match = self.regex_security.match(log_message['MESSAGE'])
if security_match:
for key, value in security_match.groupdict().items():
log_message[key] = value
else:
others_match = self.regex_others.match(log_message['MESSAGE'])
if others_match:
for key, value in others_match.groupdict().items():
log_message[key] = value
return True
};
21 - Regular expression (regexp) parser
The AxoSyslog application can parse fields from a message with the help of regular expressions. This can be also achieved with the match() filter, by setting the store-matches flag, but the regexp-parser() offers more flexibility, like multiple patterns and setting the prefix of the created name-value pairs.
Note
The
regexp-parser() can create additional name-value pairs only if “named capture groups” are used in the regular expression, for example
(?<test_field>\w+). For more information, see “named capture groups” in
PCRE documentation.
For more information about regular expressions in AxoSyslog, see Regular expressions.
For example:
Declaration:
parser p_regexp {
regexp-parser(
patterns( ... )
);
};
Example: Using a regexp-parser()
In the following example, the incoming log message is the following:
Apr 20 11:09:46 test_field -> test_value
The regexp-parser inserts the .regexp. prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (.) character will be written to the file (dot-nv-pairs). The log line connects the source, the parser and the destination.
source s_network {
network(
port(21514)
flags(no-parse)
);
};
parser p_regexp {
regexp-parser(
patterns(".*test_field -> (?<test_field>.*)$")
prefix(".regexp.")
);
};
destination d_file {
file(
"/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n")
);
};
log {
source(s_network);
parser(p_regexp);
destination(d_file);
};
You can also define the parser inline in the log path.
source s_network {
network(
port(21514)
flags(no-parse)
);
};
destination d_file {
file(
"/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n")
);
};
log {
source(s_network);
parser{
regexp-parser(
patterns(".*test_field -> (?<test_field>.*)$")
prefix(".regexp.")
);
};
destination(d_file);
};
You can set multiple patterns:
parser p_regexp {
regexp-parser(
patterns(".*test_field -> (?<test_field>.*)$", ".*other_format: (?<foo>.*)$")
prefix(".regexp.")
);
};
21.1 - Options of Regular expression parsers
The Regular expression parser has the following options.
flags()
|
|
| Type: |
assume-utf8, dont-store-legacy-msghdr, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8 |
| Default: |
empty set |
Description: Specifies the log parsing options of the source.
-
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
-
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
-
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
-
exit-on-eof: If this flag is set on a source, AxoSyslog stops when an EOF (end of file) is received. Available in version 4.9 and later.
-
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
-
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
-
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
-
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
The following example illustrates using the no-header flag with the syslog-parser() parser:
parser p_syslog {
syslog-parser(
flags(no-header)
);
};
-
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
source s_dell {
network(
port(2000)
flags(no-hostname)
);
};
-
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
-
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
-
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
-
store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message, so this flag is active. To disable it, use the dont-store-legacy-msghdr flag.
-
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
-
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
-
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
patterns()
|
|
| Synopsis: |
patterns(“pattern1” “pattern2”) |
| Mandatory: |
yes |
Description: The regular expression patterns that you want to find a match. regexp-parser() supports multiple patterns, and stops the processing at the first successful match.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
This parser does not have a default prefix. To configure a custom prefix, use the following format:
parser p_regexp{
regexp-parser(
patterns( ... )
prefix("myprefix.")
);
};
template()
|
|
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
22 - Structured data (SDATA) parser
Available in AxoSyslog 4.1 and later.
The sdata-parser() allows you to parse an RFC5424-style structured data string. You can use it to parse this relatively complex format separately, for example, to process malformatted messages. You can use the optional prefix option to add a specific string before the names of the parsed name-value pairs.
Declaration
parser parser_name {
sdata-parser(
template("<string-or-template-to-parse>")
prefix("<prefix-for-parsed-name-value-pairs>")
);
};
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
template()
|
|
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
23 - Sudo parser
The sudo parser can parse the log messages of the sudo command. Available in version 3.16 and later.
Declaration:
@version: 4.9
@include "scl.conf"
log {
source { system(); };
parser { sudo-parser(); };
destination { ... };
};
The sudo-parser() is actually a reusable configuration snippet configured to parse sudo messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, sudo-parser() uses the .sudo. prefix. To modify it, use the following format:
parser {
sudo-parser(prefix("myprefix."));
};
24 - Parsing syslog messages
By default, AxoSyslog parses every message using the syslog-parser as a syslog message, and fills the macros with values of the message. The syslog-parser does not discard messages: the message cannot be parsed as a syslog message, the entire message (including its header) is stored in the ${MSG} macro. If you do not want to parse the message as a syslog message, use the flags(no-parse) option of the source.
You can also use the syslog-parser to explicitly parse a message, or a part of a message as a syslog message (for example, after rewriting the beginning of a message that does not comply with the syslog standards).
Example: Using junctions
For example, suppose that you have a single network source that receives log messages from different devices, and some devices send messages that are not RFC-compliant (some routers are notorious for that). To solve this problem in earlier versions of AxoSyslog, you had to create two different network sources using different IP addresses or ports: one that received the RFC-compliant messages, and one that received the improperly formatted messages (for example, using the flags(no-parse) option). Using junctions this becomes much more simple: you can use a single network source to receive every message, then use a junction and two channels. The first channel processes the RFC-compliant messages, the second everything else. At the end, every message is stored in a single file. The filters used in the example can be host() filters (if you have a list of the IP addresses of the devices sending non-compliant messages), but that depends on your environment.
log {
source {
syslog(
ip(10.1.2.3)
transport("tcp")
flags(no-parse)
);
};
junction {
channel {
filter(f_compliant_hosts);
parser {
syslog-parser();
};
};
channel {
filter(f_noncompliant_hosts);
};
};
destination {
file("/var/log/messages");
};
};
Since every channel receives every message that reaches the junction, use the flags(final) option in the channels to avoid the unnecessary processing the messages multiple times:
log {
source {
syslog(
ip(10.1.2.3)
transport("tcp")
flags(no-parse)
);
};
junction {
channel {
filter(f_compliant_hosts);
parser {
syslog-parser();
};
flags(final);
};
channel {
filter(f_noncompliant_hosts);
flags(final);
};
};
destination {
file("/var/log/messages");
};
};
Note that by default, the syslog-parser attempts to parse the message as an RFC3164-formatted (BSD-syslog) message. To parse the message as an RFC5424-formatted message, use the flags(syslog-protocol) option in the parser.
syslog-parser(flags(syslog-protocol));
Parsing errors
AxoSyslog 4.7 and newer automatically adds the following tags if it encounters errors when parsing syslog messages.
message.utf8_sanitized: The message is not valid UTF-8.syslog.missing_timestamp: The message has no timestamp.syslog.invalid_hostname: The hostname field doesn’t seem to be valid, for example, it contains invalid characters. For details on the valid characters, see the check-hostname() global option.syslog.missing_pri: The priority (PRI) field is missing from the message.syslog.unexpected_framing: An octet count was found in front of the message, suggested invalid framing.syslog.rfc3164_missing_header: The date and the host are missing from an RFC3164-formatted message - practically that’s the entire header of RFC3164-formatted messages.syslog.rfc5424_unquoted_sdata_value: An RFC5424 message contains an incorrectly quoted SDATA field.message.parse_error: Some other parsing error occurred.
24.1 - Options of syslog-parser() parsers
The syslog-parser() has the following options:
default-facility()
|
|
| Type: |
facility string |
| Default: |
kern |
Description: This parameter assigns a facility value to the messages received from the file source if the message does not specify one.
default-priority()
|
|
| Type: |
priority string |
| Default: |
|
Description: This parameter assigns an emergency level to the messages received from the file source if the message does not specify one. For example, default-priority(warning).
drop-invalid()
|
|
| Type: |
yes or no |
| Values: |
`yes |
| Default: |
no |
Description: This option determines how the syslog-parser() affects messages when parsing fails.
If you set drop-invalid() to yes, syslog-parser() will drop the message if the parsing fails.
If you set drop-invalid() to no, the parsing error triggers syslog-parser() to rewrite and extend the original log message with the following additional information:
- It prepends the following message to the contents of the
$MESSAGE field: Error processing log message.
- It sets the contents of the
$PROGRAM field to syslog-ng.
- It sets the contents of the
facility field to syslog.
- It sets the contents of the
severity field to error.
Note
With the drop-invalid(no) option syslog-parser() will work in the same way as the sources which receive syslog-protocol/BSD-format messages.
Example: enabling the drop-invalid() option
parser p_syslog { syslog-parser(drop-invalid(yes)); };
flags()
|
|
| Type: |
assume-utf8, dont-store-legacy-msghdr, empty-lines, expect-hostname, kernel, no-hostname, no-multi-line, no-parse, sanitize-utf8, store-legacy-msghdr, store-raw-message, syslog-protocol, validate-utf8 |
| Default: |
empty set |
Description: Specifies the log parsing options of the source.
-
assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
-
dont-store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (AxoSyslog automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
-
empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, AxoSyslog removes empty lines automatically.
-
exit-on-eof: If this flag is set on a source, AxoSyslog stops when an EOF (end of file) is received. Available in version 4.9 and later.
-
expect-hostname: If the expect-hostname flag is enabled, AxoSyslog will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
-
guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp.
-
kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
-
no-header: The no-header flag triggers AxoSyslog to parse only the PRI field of incoming messages, and put the rest of the message contents into $MSG.
Its functionality is similar to that of the no-parse flag, except the no-header flag does not skip the PRI field.
Note
Essentially, the no-header flag signals AxoSyslog that the syslog header is not present (or does not adhere to the conventions / RFCs), so the entire message (except from the PRI field) is put into $MSG.
The following example illustrates using the no-header flag with the syslog-parser() parser:
parser p_syslog {
syslog-parser(
flags(no-header)
);
};
-
no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way AxoSyslog assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
source s_dell {
network(
port(2000)
flags(no-hostname)
);
};
-
no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
-
no-parse: By default, AxoSyslog parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The AxoSyslog application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.
If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, AxoSyslog generates a new syslog header (timestamp, host, and so on) automatically. Note that even though flags(no-parse) disables message parsing, some flags can still be used, for example, the no-multi-line flag.
-
sanitize-utf8: When using the sanitize-utf8 flag, AxoSyslog converts non-UTF-8 input to an escaped form, which is valid UTF-8.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
-
store-legacy-msghdr: By default, AxoSyslog stores the original incoming header of the log message, so this flag is active. To disable it, use the dont-store-legacy-msghdr flag.
-
store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another AxoSyslog node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
-
syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
-
validate-utf8: The validate-utf8 flag enables encoding-verification for messages.
Prior to version 4.6, this flag worked only when parsing RFC3164 messages. Starting with version 4.6, it works also for RFC5424 and raw messages.
For RFC5424-formatted messages, if the BOM character is missing, but the message is otherwise UTF-8 compliant, AxoSyslog automatically adds the BOM character to the message.
The byte order mark (BOM) is a Unicode character used to signal the byte-order of the message text.
For the syslog-parser() you can also set the following flags:
-
check-hostname: Equivalent with the check-hostname() global option, but only applies to this parser.
-
no-piggyback-errors: Do not attribute the message to AxoSyslog in case of errors. Things already processed or extracted are retained, for example: ${MESSAGE} retains its value (potentially the raw message), other macros like ${HOST}, ${PROGRAM}, or ${PID} may or may not be extracted. The error is indicated by setting ${MSGFORMAT} set to “syslog:error”.
Available in AxoSyslog 4.8.1 and later.
sdata-prefix()
|
|
| Type: |
string |
| Default: |
.SDATA. |
Available in AxoSyslog 4.1 and later.
Description: Adds a specific string before the names of the parsed SDATA fields to store the name-value pairs created from the SDATA fields separately. Note that unless the value of sdata-prefix starts with .SDATA., using this option excludes the parsed fields from the sdata and rfc5424 scopes of the value pairs.
template()
|
|
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
25 - Parsing tags
The AxoSyslog application can tag log messages, and can include these tags in the log messages, as described in Tagging messages. The tags-parser() can parse these tags from the incoming messages and re-tag them. That way if you add tags to a log message on a AxoSyslog client, the message will have the same tags on the AxoSyslog server. Available in version 3.23 and later.
Specify the macro that contains the list of tags to parse in the template() option of the parser, for example, the SDATA field that you used to transfer the tags, or the name of the JSON field that contains the tags after using the json-parser().
tags-parser(template("${<macro-or-field-with-tags>}"));
26 - Websense parser
The Websense parser can parse the log messages of Websense Content Gateway (Raytheon|Websense, now Forcepoint). These messages do not completely comply with the syslog RFCs, making them difficult to parse. The websense-parser() of AxoSyslog solves this problem, and can separate these log messages to name-value pairs. For details on using value-pairs in AxoSyslog see Structuring macros, metadata, and other value-pairs. The parser can parse messages in the following format:
<PRI><DATE> <TIMEZONE> <IP-ADDRESS> <NAME=VALUE PAIRS>
For example:
<159>Dec 19 10:48:57 EST 192.168.1.1 vendor=Websense product=Security product_version=7.7.0 action=permitted severity=1 category=153 user=- src_host=192.168.2.1 src_port=62189 dst_host=example.com dst_ip=192.168.3.1 dst_port=443 bytes_out=197 bytes_in=76 http_response=200 http_method=CONNECT http_content_type=- http_user_agent=Mozilla/5.0_(Windows;_U;_Windows_NT_6.1;_enUS;_rv:1.9.2.23)_Gecko/20110920_Firefox/3.6.23 http_proxy_status_code=200 reason=- disposition=1034 policy=- role=8 duration=0 url=https://example.com
If you find a message that the websense-parser() cannot properly parse, contact us, so we can improve the parser.
The AxoSyslog application sets the ${PROGRAM} field to Websense.
By default, the websense-specific fields are extracted into name-value pairs prefixed with .websense. For example, the product_version in the previous message becomes ${.websense.product_version}. You can change the prefix using the prefix option of the parser.
Declaration:
@version: 4.9.0
@include "scl.conf"
log {
source { network(flags(no-parse)); };
parser { websense-parser(); };
destination { ... };
};
Note that you have to disable message parsing in the source using the flags(no-parse) option for the parser to work.
The websense-parser() is actually a reusable configuration snippet configured to parse websense messages. For details on using or writing such configuration snippets, see Reusing configuration blocks. You can find the source of this configuration snippet on GitHub.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
By default, websense-parser() uses the .websense. prefix. To modify it, use the following format:
parser {
websense-parser(prefix("myprefix."));
};
27 - Windows XML Event Log (EVTX) parser
Available in AxoSyslog version 4.5 and later.
The new windows-eventlog-xml-parser() can parse messages in the Windows XML Event Log (EVTX) format.
Example configuration:
parser p_win {
windows-eventlog-xml-parser(prefix(".winlog."));
};
The windows-eventlog-xml-parser() parser has the same parameters are the same as the xml() parser.
Don’t forget to include the parsers in a log statement to actually use it:
log {
source(s_local);
parser(windows-eventlog-xml-parser(prefix(".winlog.")));
destination(d_local);
};
28 - XML parser
Extensible Markup Language (XML) is a text-based open standard designed for both human-readable and machine-readable data interchange. Like JSON, it is used primarily to transmit data between a server and web application. It is described in W3C Recommendation: Extensible Markup Language (XML).
The XML parser processes input in XML format, and adds the parsed data to the message object.
To create an XML parser, define an xml_parser that has the xml() option. By default, the parser will process the ${MESSAGE} part of the log message. To process other parts of a log message using the XML parser, use the template() option. You can also define the parser inline in the log path.
Declaration:
parser xml_name {
xml(
template()
prefix()
drop-invalid()
exclude-tags()
strip-whitespaces()
);
};
Example: Using an XML parser
In the following example, the source is an XML-encoded log message. The destination is a file that uses the format-json template. The log line connects the source, the destination and the parser.
source s_local {
file("/tmp/aaa");
};
destination d_local {
file(
"/tmp/bbb"
template("$(format-json .xml.*)\n")
);
};
parser xml_parser {
xml();
};
log {
source(s_local);
parser(xml_parser);
destination(d_local);
};
You can also define the parser inline in the log path.
log {
source(s_file);
parser { xml(prefix(".SDATA")); };
destination(d_file);
};
The XML parser inserts an “.xml” prefix by default before the extracted name-value pairs. Since format-json replaces a dot with an underscore at the beginning of keys, the “.xml” prefix becomes “_xml”. Attributes get an _ prefix. For example, from the XML input:
<tags attr='attrval'>part1<tag1>Tag1 Leaf</tag1>part2<tag2>Tag2 Leaf</tag2>part3</tags>
The following output is generated:
{"_xml":{"tags":{"tag2":"Tag2 Leaf","tag1":"Tag1 Leaf","_attr":"attrval","tags":"part1part2part3"}}}
When the text is separated by tags on different levels or tags on the same level, the parser simply concatenates the different parts of text. For example, from this input XML:
<tag>
<tag1>text1</tag1>
<tag1>text2</tag1>
</tag>
The following output is generated:
.xml.tag.tag1 = text1text2
Whitespaces are kept as they are in the XML input. No collapsing happens on significant whitespaces. For example, from this input XML:
<133>Feb 25 14:09:07 webserver syslogd: <b>|Test\n\n Test2|</b>\n
The following output is generated:
[2017-09-04T13:20:27.417266] Setting value; msg='0x7f2fd8002df0', name='.xml.b', value='|Test\x0a\x0a Test2|'
However, note that users can choose to strip whitespaces using the strip-whitespaces() option.
Configuration hints
Define a source that correctly detects the end of the message, otherwise the XML parser will consider the input invalid, resulting in a parser error.
To ensure that the end of the XML document is accurately detected, use any of the following options:
In case you experience issues, start syslog-ng with debug logs enabled. There will be a debug log about the incoming log entry, which shows the complete message to be parsed. The entry should contain the entire XML document.
Note
If your log messages are entirely in .xml format, make sure to disable any message parsing on the source side by including the flags("no-parse") option in your source statement. This will put the entire log message in the $MESSAGE macro, which is the field that the XML parser parses by default.
28.1 - Limitations of the XML parsers
The XML parser comes with certain limitations.
Vector-like structures:
It is not possible to address each element of a vector-like structure individually. For example, take this input:
<vector>
<entry>value1</entry>
<entry>value2</entry>
...
<entry>valueN</entry>
</vector>
After parsing, the entries cannot be addressed individually. Instead, the text of the entries will be concatenated:
vector.entry = "value1value2...valueN"
Note that xmllint has the same behavior:
$ xmllint --xpath "/vector/entry/text()" test.xml
value1value2valueN%
CDATA:
The XML parser does not support CDATA. CDATA inside the XML input is ignored. This is true for the processing instructions as well.
Inherited limitations:
The XML parser is based on the glib XML subset parser, called “GMarkup” parser, which is not a full-scale XML parser. It is intended to parse a simple markup format that is a subset of XML. Some limitations are inherited:
-
Do not use the XML parser if you expect to interoperate with applications generating full-scale XML. Instead, use it for application data files, configuration files, log files, and so on, where you know your application will be the only one writing the file.
-
The XML parser is not guaranteed to display an error message in the case of invalid XML. It may accept invalid XML. However, it does not accept XML input that is not well-formed (a condition that is weaker than requiring XML to be valid).
No support for long keys:
If the key is longer than 255 characters, AxoSyslog drops the entry and an error log is emitted. There is no chunking or any other way of recovering data, not even partial data. The entry will be replaced by an empty string.
28.2 - Options of the XML parsers
The XML parser has the following options.
drop-invalid
|
|
| Synopsis: |
drop-invalid() |
| Format: |
`yes |
| Default: |
no |
| Mandatory: |
no |
Description: If set, messages with an invalid XML will be dropped entirely.
| Synopsis: |
exclude-tags() |
| Format: |
list of globs |
| Default: |
None
If not set, no filtering is done. |
| Mandatory: |
no |
Description: The XML parser matches tags against the listed globs. If there is a match, the given subtree of the XML will be omitted.
parser xml_parser {
xml(
template("$MSG")
exclude-tags("tag1", "tag2", "inner*")
);
};
From this XML input:
<tag1>Text1</tag1><tag2>Text2</tag2><tag3>Text3<innertag>TextInner</innertag></tag3>
The following output is generated:
{"_xml":{"tag3":"Text3"}}
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
-
To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
-
To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
-
If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by AxoSyslog. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard versus soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
The prefix() option is optional and its default value is ".xml".
strip-whitespaces
|
|
| Synopsis: |
strip-whitespaces() |
| Format: |
`yes |
| Default: |
no |
| Mandatory: |
no |
Description: Strip the whitespaces from the XML text nodes before adding them to the message.
Example: Using strip-whitespaces
parser xml_parser {
xml(
template("$MSG")
strip-whitespaces(yes)
);
};
From this XML input:
The following output is generated:
template()
|
|
| Synopsis: |
template("${<macroname>}") |
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).