Parsing dates and timestamps

1: Options of date-parser() parsers

The date parser can extract dates from non-syslog messages. It operates by default on the ${MESSAGE} part of the log message, but can operate on any template or field provided. The parsed date will be available as the sender date (that is, the ${S_DATE}, ${S_ISODATE}, ${S_MONTH}, and so on, and related macros). (To store the parsed date as the received date, use the time-stamp(recvd) option.)

Note Note that parsing will fail if the format string does not match the entire template or field. Since by default AxoSyslog uses the ${MESSAGE} part of the log message, parsing will fail, unless the log message contains only a date, but that is unlikely, so practically you will have to segment the message (for example, using a csv-parser()) before using the date-parser(). You can also use date-parser() to parse dates received in a JSON or key-value-formatted log message.

Declaration

   parser parser_name {
        date-parser(
            format("<format-string-for-the-date>")
            template("<field-to-parse>")
        );
    };

Example: Using the date-parser()

In the following example, AxoSyslog parses dates like 01/Jan/2016:13:05:05 PST from a field called MY_DATE using the following format string: format("%d/%b/%Y:%H:%M:%S %Z") (how you create this field from the incoming message is not shown in the example). In the destination template every message will begin with the timestamp in ISODATE format. Since the syslog parser is disabled, AxoSyslog will include the entire original message (including the original timestamp) in the ${MESSAGE} macro.

   source s_file {
        file("/tmp/input" flags(no-parse));
    };
    
    destination d_file {
        file(
            "/tmp/output"
            template("${S_ISODATE} ${MESSAGE}\n")
        );
    };
    
    log {
        source(s_file);
        parser { date-parser(format("%d/%b/%Y:%H:%M:%S %Z") template("${MY_DATE}")); };
        destination(d_file);
    };

In the template option, you can use template functions to specify which part of the message to parse with the format string. The following example selects the first 24 characters of the ${MESSAGE} macro.

   date-parser(format("%d/%b/%Y:%H:%M:%S %Z") template("$(substr ${MESSAGE} 0 24)") );

In AxoSyslog version 3.23 and later, you can specify a comma-separated list of formats to parse multiple date formats with a single parser. For example:

   date-parser(format(
        "%FT%T.%f",
        "%F %T,%f",
        "%F %T"
    ));

If you need to modify or correct the timezone of the message after parsing, see Rewrite the timezone of a message.

1 - Options of date-parser() parsers

The date-parser() parser has the following options.

format()


Synopsis:	format(string)
Default:

Description: Specifies the format how AxoSyslog should parse the date. You can use the following format elements:

%a: The day of week, using the locale’s weekday names. Either the abbreviated or full name may be specified.
%A: Same as %a.
%b: The month, using the locale’s month names. Either the abbreviated or full name may be specified.
%B: The same as %b.
%c: The date and time, using the locale’s date and time format.
%C: The century number [0,99]. Leading zeros are permitted but not required. This conversion should be used in conjunction with the %y conversion.
%d: The day of month [1,31]. Leading zeros are permitted but not required.
%D: The date as %m/%d/%y.
%e: The same as %d.
%F: The date as %Y-%m-%d (the ISO 8601 date format).
%g: The year corresponding to the ISO week number, without the century. (A NetBSD extension.)
%G: The year corresponding to the ISO week number, with the century. (A NetBSD extension.)
%h: The same as %b.
%H: The hour (24-hour clock) [0,23]. Leading zeros are permitted but not required.
%I: The hour (12-hour clock) [1,12]. Leading zeros are permitted but not required.
%j: The day number of the year [1,366]. Leading zeros are permitted but not required.
%k: The same as %H.
%l: The same as %I.
%m: The month number [1,12]. Leading zeros are permitted but not required.
%M: The minute [0,59]. Leading zeros are permitted but not required.
%n: Any white-space, including none.
%p: The locale’s equivalent of a.m. or p.m.
%r: The time (12-hour clock) with %p, using the locale’s time format.
%R: The time as %H:%M.
%S: The seconds [0,60]. Leading zeros are permitted but not required.
%s: The number of seconds since the Epoch, UTC (see mktime(3)). (A NetBSD extension.)
%f: Fraction of the second (with or without a leading dot).
%t: Any white-space, including none.
%T: The time as %H:%M:%S.
%u: The day of the week as a decimal number, where Monday = 1. (A NetBSD extension.)
%U: The week number of the year (Sunday as the first day of the week) as a decimal number [0,53]. Leading zeros are permitted but not required. All days in a year preceding the first Sunday are considered to be in week 0.
%V: The ISO 8601:1988 week number as a decimal number. If the week (starting on Monday) that contains January 1 has more than three days in the new year, then it is considered the first week of the year. If it has fewer than four days in the new year, then it is considered the last week of the previous year. Weeks are numbered from 1 to 53. (A NetBSD extension.)
%w: The weekday as a decimal number [0,6], with 0 representing Sunday. Leading zeros are permitted but not required.
%W: The week number of the year (Monday as the first day of the week) as a decimal number [0,53]. Leading zeros are permitted but not required. All days in a year preceding the first Monday are considered to be in week 0.
%x: The date, using the locale’s date format.
%X: The time, using the locale’s time format.
%y: The year within the 20th century [69,99] or the 21st century [0,68]. Leading zeros are permitted but not required. If specified in conjunction with %C, specifies the year [0,99] within that century.
%Y: The year, including the century (i.e., 1996).
%Z: Timezone in ascii format (for example, PST), or in format -/+0000, accepts : in the middle of timezones (ISO 8601)
%z: Timezone in ascii format (for example, PST), or in format -/+0000, accepts : in the middle of timezones (ISO 8601) (Required element)
%%: matches a literal %. No argument is converted.

Warning

When using the %z and %Z format codes, consider that while %z strictly expects a specified timezone, and triggers a warning if the timezone is missing, %Z does not trigger a warning if the timezone is not specified.

For further information about the %z and %Z format codes, see the ‘DESCRIPTION’ section on the srtptime(3) - NetBSD Manual Pages.

For example, for the date 01/Jan/2016:13:05:05 PST use the following format string: "%d/%b/%Y:%H:%M:%S %Z"

In AxoSyslog version 3.23 and later, you can specify a comma-separated list of formats to parse multiple date formats with a single parser. For example:

   date-parser(format(
        "%FT%T.%f",
        "%F %T,%f",
        "%F %T"
    ));

template()


Synopsis:	`template("${<macroname>}")`

Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).

flags()


Type:	guess-timezone
Default:	empty string

guess-timezone: Attempt to guess the timezone of the message if this information is not available in the message. Works when the incoming message stream is close to real time, and the timezone information is missing from the timestamp. For example:

   date-parser(flags(guess-timezone));

time-stamp()


Synopsis:	`stamp` or `recvd`
Default:	stamp

Description: Determines if the parsed date values are treated as sent or received date. If you use time-stamp(stamp), AxoSyslog adds the parsed date to the S_ macros (corresponding to the sent date). If you use time-stamp(recvd), AxoSyslog adds the parsed date to the R_ macros (corresponding to the received date).

time-zone()


Synopsis:	time-zone(string)
Default:

Description: If this option is set, AxoSyslog assumes that the parsed timestamp refers to the specified timezone. The timezone set in the time-zone() option overrides any timezone information parsed from the timestamp.

The timezone can be specified by using the name, for example, time-zone("Europe/Budapest")), or as the timezone offset in +/-HH:MM format, for example, +01:00). On Linux and UNIX platforms, the valid timezone names are listed under the /usr/share/zoneinfo directory.

value()


Synopsis:	string
Default:

Available in AxoSyslog 4.1 and later.

Description: Instruct date-parser() to store the resulting timestamp in a name-value pair specified in value(), instead of changing the timestamp value of the LogMessage.