Parsing syslog messages

By default, AxoSyslog parses every message using the syslog-parser as a syslog message, and fills the macros with values of the message. The syslog-parser does not discard messages: the message cannot be parsed as a syslog message, the entire message (including its header) is stored in the ${MSG} macro. If you do not want to parse the message as a syslog message, use the flags(no-parse) option of the source.

You can also use the syslog-parser to explicitly parse a message, or a part of a message as a syslog message (for example, after rewriting the beginning of a message that does not comply with the syslog standards).

Example: Using junctions

For example, suppose that you have a single network source that receives log messages from different devices, and some devices send messages that are not RFC-compliant (some routers are notorious for that). To solve this problem in earlier versions of AxoSyslog, you had to create two different network sources using different IP addresses or ports: one that received the RFC-compliant messages, and one that received the improperly formatted messages (for example, using the flags(no-parse) option). Using junctions this becomes much more simple: you can use a single network source to receive every message, then use a junction and two channels. The first channel processes the RFC-compliant messages, the second everything else. At the end, every message is stored in a single file. The filters used in the example can be host() filters (if you have a list of the IP addresses of the devices sending non-compliant messages), but that depends on your environment.

   log {
        source {
            syslog(
                ip(10.1.2.3)
                transport("tcp")
                flags(no-parse)
            );
        };
        junction {
            channel {
                filter(f_compliant_hosts);
                parser {
                    syslog-parser();
                };
            };
            channel {
                filter(f_noncompliant_hosts);
            };
        };
        destination {
            file("/var/log/messages");
        };
    };

Since every channel receives every message that reaches the junction, use the flags(final) option in the channels to avoid the unnecessary processing the messages multiple times:

   log {
        source {
            syslog(
                ip(10.1.2.3)
                transport("tcp")
                flags(no-parse)
            );
        };
        junction {
            channel {
                filter(f_compliant_hosts);
                parser {
                    syslog-parser();
                };
                flags(final);
            };
            channel {
                filter(f_noncompliant_hosts);
                flags(final);
            };
        };
        destination {
            file("/var/log/messages");
        };
    };

Note that by default, the syslog-parser attempts to parse the message as an RFC3164-formatted (BSD-syslog) message. To parse the message as an RFC5424-formatted message, use the flags(syslog-protocol) option in the parser.

   syslog-parser(flags(syslog-protocol));

Parsing errors

AxoSyslog 4.7 and newer automatically adds the following tags if it encounters errors when parsing syslog messages.

• message.utf8_sanitized: The message is not valid UTF-8.
• syslog.missing_timestamp: The message has no timestamp.
• syslog.invalid_hostname: The hostname field doesn’t seem to be valid, for example, it contains invalid characters. For details on the valid characters, see the check-hostname() global option.
• syslog.missing_pri: The priority (PRI) field is missing from the message.
• syslog.unexpected_framing: An octet count was found in front of the message, suggested invalid framing.
• syslog.rfc3164_missing_header: The date and the host are missing from an RFC3164-formatted message - practically that’s the entire header of RFC3164-formatted messages.
• syslog.rfc5424_unquoted_sdata_value: An RFC5424 message contains an incorrectly quoted SDATA field.
• message.parse_error: Some other parsing error occurred.