Log File Parser Definition

NAVIGATION  Monitor > Log Monitoring > Log Parser > Log File Parser Definition

The Log File Parser Definition page defines templates and parameters used to parse log files. Definitions are subsequently assigned to machine IDs using the Log Parser page. Log parsers are initially private, but can be shared with other users.

The Log File Parsing Cycle

The parsing of a log file is triggered whenever the log file is changed. In most cases this involves appending new text to the end of the file. To avoid scanning the entire log file from the beginning each time the file is updated, the agent parses log files as follows:

  • After each update the agent stores a "bookmark" of the last 512 bytes of a log file.
  • When the log file is updated again, the agent compares the bookmark from the old update with the same byte position in the new update.
  • Since log files may be archived before the log parser is run, parsing can include archives files if they exist.
  • You can specify sets of log files and sets of archive files by specifying full pathnames with asterisk (*) and question mark (?) wildcards. If a set of files is specified the parser begins with the latest file in the set.
  • If the bookmark text is the same in both the old update and the new update, the agent begins parsing text after the bookmark.
  • If the bookmark text is not the same and no Log Archive Path is specified, the agent parses the entire log file from the beginning. If a Log Archive Path is specified, the agent searches for the bookmark in the archive files. If the bookmark cannot be found, the agent bookmarks the end of the log file and starts parsing from there in the next cycle.
  • Once parsing is completed a new bookmark is defined based on the last 512 bytes of the newly updated log file and the process repeats itself.

Note: The parsing of a log file is not a procedure event itself. Only a new configuration, or reconfiguration, using Log Parser, Assign Parser Sets or Parser Summary generates a procedure you can see in the Procedure History or Pending Procedure tabs of the Machine Summary page.

Save

Select Save to save changes to a log file parser definition.

Save As...

Select Save As... to save a log file parser definition under a different name.

Delete

Select Delete to delete a log file parser definition.

Share...

You can share log file parser definitions you own with other VSA users, user roles, or make the procedure public to all users.

Parser Name

Enter the name of the parser.

Log File Path

Enter the full UNC pathname or mapped drive pathname on the target machine of the log file you want to parse. You can use asterisk (*) or question mark (?) wildcards to specify a set of log files. If a log file set is specified, the log parser starts with the latest log file first. Example: \\morpheus\logs\message.log or c:\logs\message.log. When specifying a UNC path to a share accessed by an agent machine—for example \\machinename\share—ensure the share's permissions allow read/write access using the agent credential specified for that agent machine in Agent > Manage Agents.

Log Archive Path

Enter the full UNC pathname or mapped drive pathname on the target machine of the archive files you want to parse. You can use asterisk (*) or question mark (?) wildcards to specify a set of archive files. If an archive set is specified, the log parser starts with the latest log file first. Example: If message.log is archived daily to a file in messageYYYYMMDD.log format, then you can specify c:\logs\message*.log. When specifying a UNC path to a share accessed by an agent machine—for example \\machinename\share—ensure the share's permissions allow read/write access using the agent credential specified for that agent machine in Agent > Manage Agents.

Description

Enter a description for the log parser.

Template

The template is used to compare with the log entry in the log file to extract out the required data into parameters. Parameters are enclosed with $ character in template.

Enter a pattern of text and log file parameters. This pattern is used to search from the beginning of each line in a log file. If a pattern finds a match in the log file, the log file parameters in the pattern are populated with the values extracted from the log file.

You can use a percent (%) wildcard to specify an alphanumeric string of any length. A log file parameter is bracketed with the dollar ($) symbol. Enter $$ to match a pattern of text containing a $ symbol. Enter %% to match a pattern of text containing a % symbol.

Note: Template text patterns are case sensitive.

Example

  • Log text: 126 Oct 19 2007 12:30:30 127.0.0.1 Device0[123]: return error code -1!
  • Template: $EventCode$ $Time$ $HostComputer$ $Dev$[$PID$]:%error code $ErrorCode$!
  • Parsed result:
    EventCode=126
    Time= 2007/10/19 12:30:30 Friday
    HostComputer=127.0.0.1
    Dev=Device0
    PID=123
    ErrorCode=-1

Guidelines

  • To enter a tab character in the template edit box:
    1. Copy and paste a tab character from log data.
    2. Use {tab} if it is enter manually.
  • To create a template it is easier to copy the original text into the template, then replace the characters that can be ignored with %. Then replace the characters that are saved to a parameter with a parameter name.
  • Make sure all parameters in the template are defined in Log File Parameters.
  • A date time parameter must have both date and time information from the source data, otherwise just use a string parameter.

Skipping Characters

To skip characters, use $[n]$, where n is the number of characters to skip. Use $var[n]$ to retrieve a fixed number of characters to be a variable value.

Example

  • Log text: 0123456789ABCDEFGHIJ
  • Template: $[10]$ABC$str[3]$
  • Result for parameter str is DEF.

Multi-line Template

If checked, multiple lines of text and log file parameters are used to parse the log file.

NOTE   The character string {tab} can be used as a tab character and {nl} can be used as a new line break. {nl} cannot be used in single line template. % can be used as wildcard character.

Output Template

Enter a pattern of text and log file parameters to store in Log Monitoring.

Example:

  • Output template:  Received device error from $Dev$ on $HostComputer$. Code = $ErrorCode$.
  • Result output: Received device error from Device0 on 127.0.0.1. Code = -1.

Apply

Click Apply to add or update a parameter entered in the Name field.

Clear All

Click Clear All to remove all parameters from the parameter list.

Log File Parameters

Name

Once the template is created, you need to define the list of parameters used by the template. All the parameters in the template have to be defined, otherwise the parser returns an error. Available parameters are integer, unsigned integer, long, unsigned long, float, double, datetime, string. The length of parameter name is limited to 32 characters.

Enter the name of a parameter used to store a value. Parameters are subsequently used in the Template and Output Template text boxes.

NOTE   Do not bracket the name of the parameter with $ symbols in the Name field. This is only required when the parameter is entered in the Template and Output Template text boxes.

Type

Enter the data type appropriate for the parameter. If data parsed from a log file cannot be stored using that data type, the parameter remains empty.

Date Format

If the Type selected is Date Time, enter a Date Format.

  • yy, yyyy, YY, YYYY - two or four digit year
  • M - single or two digit month
  • MM - two digit month
  • MMM - abbreviation of month name, ex. "Jan"
  • MMMM - full month name, ex. "January"
  • D, d - single or two digit day
  • DD, dd - two digit day
  • DDD, ddd - abbreviation name of day of week, Ex. "Mon"
  • DDDD, dddd - full name of day of week, ex. "Monday"
  • H, h - single or two digit hour
  • HH, hh - two digit hour
  • m - single or two digit minute
  • mm  - two digit minute
  • s - single or two digit second
  • ss - two digit second
  • f  - one or more digit of fraction of second
  • ff - fffffffff - two to nine digit
  • t - one character time mark, ex. "a"
  • tt - two-character time mark, ex. "am"

NOTE  Date and time filtering in views and reports are based on the log entry time. If you include a $Time$ parameter using the Date Time data type in your template, Log Monitoring uses the time stored in the $Time$ parameter as the log entry time. If a $Time$ parameter is not included in your template, then the time the entry was added to Log Monitoring serves as the log entry time. Each date time parameter must contain at least the month, day, hour, and second data.

Example:

  • Date time string: Oct 19 2007 12:30:30
  • DateTime template: MMM DD YYYY hh:mm:ss

UTC Date

Log Monitoring stores all date/time values as universal time, coordinated (UTC). This enables UTC date and times to be automatically converted to the user's local time when Log Monitoring data is displayed or when reports are generated.

If blank, the date and time values stored in the log file parameter are converted from the local time of the machine ID assigned the log parser to UTC. If checked, the date and time values stored in the log file parameter are UTC and no conversion is necessary.