XPLG Patterns Language
XPLG contains an easy to use, GUI based, advanced parsing wizard to help users apply Patterns on the processed data to normalize log records into tabular format - this helps to get optimized data view and efficient processing, querying, anomalies calculations, correlations and other insights.
Records in the log can usually be presented by a combination of types. It is recommended to present the logged information in as detailed way as possible. Doing so gives greater possibilities in manipulating and analyzing the data, such as filtering by a specific ip, priority, date, or specific text column. You should try to be as descriptive as possible.
Note that you can configure several patterns for one log using XpoLog multi pattern.
Each command will be treated as a column of data in the log view table.
Note: If a pattern is formulated incorrectly, the log records may display wrong data or no data at all.
IMPORTANT!!!
After defining a pattern, it is highly recommended to click the verify pattern link to see the results of your definition in the table at the bottom of the page.
If you do not see data at all or you see wrong data, check your pattern definition.
For any definition problem, send XpoLog support the log example by email to support@xplg.com, and we will help you define a pattern.
The following table describes the Patterns language:
Pattern Type | Description | Syntax | Attributes | Examples | ||
|---|---|---|---|---|---|---|
Text in Log | Pattern | Result | ||||
String | any string of characters, including multi line strings | {string} | No special attributes | Hello world. Hello world. Hello | {string} H{string} wor{string} {string} | Hello world. first column = ello; second column = ld. Hello world. |
Text | any single-line string of characters | {text} | No special attributes | Hello world. | {text} | Hello world. |
Date | a date string | {date, MM-dd-yyyy} | UI Foramt: after date object is parsed, its possible to modify its display by using ';' separator and specifying the desired view format
Also, see Optional Identifiers for Date Pattern section below. | Date in Log: 10/Jan/2013:11:32:38
18:05:23 {date,locale=it,dd MM yyyy HH:mm:ss}
| Pattern to parse date:
{date,timeDiff=-5000,HH:mm:ss} | |
Timestamp | a timestamp representing a date string | {timestamp} | product: XpoLog expects a timestamp in milliseconds, in case the timestamp is not in milliseconds use the product to add zeros at the end |
|
|
|
Number | a numeric string | {number} | No special attributes | Thread-1 Test 5 done calling to id 5667 | {string}-{number} {string} {number} {string} calling to id {number} | first column = Thread;second column = 1 first column = Test; second column = 5; third column = done 5667 |
Choice | a set of strings that can appear in a record | {choice,value1;value2...,valueN} Note: All optional choices should appear as a semicolon separated list inside the tag. | No special attributes | first record = red, | {choice,red;green;blue} | The matching choice of each record is displayed. |
IP Address | An IP address | {ip} | No special attributes | 127.0.0.1 | {ip} | 127.0.0.1 |
Geo IP Address | A Geo IP address | {geoip,type=country:region:city} Note: All optional combinations of country, region and city are valid, for example: country:region, region:city, country etc. | No special attributes | 66.249.71.162 193.35.249.128 | {geoip:Remote Host,type=country:region:city} {geoip:Remote Host,type=country} | 66.249.71.162 United States CA Mountain View 193.35.249.128 United Kingdom |
Priority | A set of priorities that can appear in a record | {priority,priority1;priority2...,priorityN} Note: All optional priorities should appear as a semi-colon separated list inside the tag. | No special attributes | first record = DEBUG, | {priority,DEBUG;INFO;ERROR} | The matching priority of each record is displayed. |
Expression | The expression that will be used according to the source columns given | {expression} | No special attributes | Assuming you have two columns in your log, and you want to create a new column in your log view, represents a combining of these two records (highlighted in the example): Mon Jul 10 04:33:51 2006 100 300 | {date:Date,EEE MMM dd HH:mm:ss yyyy} {num:Num1} {num:Num2}{expression,Num2-Num1} | Mon Jul 10 04:33:51 2006 100 300 200 |
Regular Expression | A regular expression, used to extract part of the data from another column | {regexp,refIndex=index / refName=column_name;multiLine=true/false,(regular_expression_to_extract)} OR for date/timestamp {regexp,refIndex=index OR refName=column_name;columnType=date/timestamp;dateFormat=DATE_FORMAT_IN_DATA;dateUIFormat=DISPLAY_DATE_FORMAT;multiLine=true/false,(regular_expression_to_extract)} | refIndex/refName (mandatory): the zero-based index of the source column / the name of the source column columnType (mandatory for date/timestamp only):
multiLine (optional): indicates whether the record spreads over more than one line | Assuming you have the next record in your log, and you need to extract the error code (highlighted in the example) that is embedded in the message:
["case_ID" : "1234", "created_at" : "2023-06-28 16:37:01"] | {date:Date,EEE MMM dd HH:mm:ss yyyy}{regexp:Error Code,refName=Message;multiLine=true,(ORA-\d+)}{string:Message}
{text:Message}{regexp:created_at,refName=Message;columnType=date;dateFormat=yyyy-MM-dd HH:mm:ss,"created_at" : "([^"]+)}{regexp:case_ID,refName=Message;,case_ID" : "([^"]+)} | Mon Jul 10 04:33:51 2006 ORA-336 ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K, |
Properties | A set of key-value properties that can appear in a record | {properties,keysSep=[Keys_Separator];propSep=[Properties_Separator];,key1;key2;....;keyN} | keysSep (mandatory): the character/s that separate the key from the value propSep (mandatory): the character/s that separate properties | Assuming you have the next record in your log: 2014-06-02 09:06:45,100 Type=INFO, Account=admin, ID=54, Team=all | {date:Date,yyyy-MM-dd HH:mm:ss,SSS} {properties: pairs,keysSep==;propSep=,;,Type;Account;ID;Team} | 2014-06-02 09:06:45,100 INFO admin 54 all Type=INFO Account=admin ID=54 Team=all |
Json | A set of key-value pairs | {json,key1;key2;key3;....;keyN} | No special attributes except date.If the date is inside the JSON object the following parameters may be specified: Note: If one of the keys of a json is a json object then the full path from the top json object should be specified using '->' to represent a level. Note: In case that you would like to grant ftype to your json column, use the following syntax: JSON_KEY#_#ftype=[FTYPE] Example: data->row->virtualMemory#_#ftype=virtualMemory Note: In case that you would like to grant a different name to your json column (mostly needed during parsing of json within json, use the following syntax: JSON_KEY#_#name=[Column_Name] Example: ;data->row->instanceCount#_#name=instanceCount Granting both ftype and column name example: data->row->arguments#_#ftype=args#_#name=Arguments | Assuming you have the next record in your log: 2014-11-24 12:02:32,083 {"red":"#f00", "green":"#0f0", "blue":"#00f", "yellow":"#ff0","black":"#000"} Date usage example: JSON in JSON example: | {date:Date,yyyy-MM-dd HH:mm:ss,SSS} {json,red;green;blue;yellow} | 2014-11-24 12:02:32,083 #f00 #0f0 #00f #ff0 {"black":"#000"} |
Term | a constant string that appears in a record and needs to be displayed in the log view | {term,TERM} TERM is the constant string | No special attributes | Assuming you have the next two records in your log: | Multi-pattern: | 03/07/2005 03:44:56 app1 192.168.11.44 success |
Free Text | any text you wish to display in the log view, usually used in multi-pattern logs to distinguish records | {freetext,FREETEXT} FREETEXT is the text that you wish to display in the log view. | No special attributes | Assuming you have the next records in your log: | Multi-pattern: {date,dd/MM/yyyy HH:mm:ss} {freetext,Dollar}$ {string} | 03/07/2005 03:44:56 Dollar success |
Value Mapping | maps an original value from log to a converted value | {map,val1=convertedVal1;val2=convertedVal2;val3=convertedVal3;...;valN=convertedValN} | No special attributes | Assuming you have in the log under one of the columns the following values: | Using the following will convert the values as follows: | 200 will be presented as 'OK' |
Value Mapping | maps an original value from log to a converted value | {map,refIndex=ORIG_COL_INDEX,file:FULL_PATH_TO_FILE} | refIndex: numeric value which represents the index of the column in which its values will be converted (first column index is 0) file: | Assuming you have in the log under one of the columns the following values: | Using the following will convert the values as follows:
| 200 will be presented as 'OK' |
Value Mapping | maps an original value from log to a converted value | {regexp,refIndex=index | refName=column_name;columnType=map;mapping=val1:convertedVal1^val2:convertedVal2^...^valN:convertedValN,(regular_expression_to_extract)} | refIndex/refName (mandatory): the zero-based index of the source column / the name of the source column columnType (mandatory) mapping (mandatory) | Assuming you have the next record in your log, and you need to map the Logon Type (highlighted in the example) that is embedded in the message: 1400504701000*;*Microsoft-Windows-Security-Auditing*;*A logon was attempted using explicit credentials. Security ID: S-1-0-0 Logon ID: 0xdc01 Logon Type: 3 | {timestamp:Date,MM/dd/yyyy HH:mm:ss}*;*{text:Source}*;*{regexp:Logon Type,refName=description;columnType=map;mapping=2:Local^3:Network,Logon Type: ([\d]+).*}{string:Description} | 05/19/2014 16:05:01 Microsoft-Windows-Security-Auditing Network A logon was attempted using explicit credentials. Security ID: S-1-0-0 Logon ID: 0xdc01 Logon Type: 3 |
Block | an optional string that does not appear in all records | {block,start,emptiness=true}XXX{block,end,emptiness=true} | start: indicates a block start | first record = 64.236.16.52:8080, | {string}{block,start,emptiness=true}:{string}{block,end,emptiness=true} | |
Horizontal Tab | a tab delimiter | {tab} | No special attributes | one 123 | {string}{tab}{number} | first column = one, |
End of Line | end of line, used in records that spread over more than one line | {eol} | No special attributes | name : logger | {string}{eol}level : {priority,db0;db1} | first column = logger, |
End of Entity | end of entity, used to mark the end of a record, improves the efficiency of the parsing process | {eoe} | No special attributes | name : logger | string}{eol}level : {priority,db0;db1}{eoe} | first row = logger,db0 |
Optional Identifiers for Date Pattern
The following table provides examples of optional identifiers that can be used in a Date pattern.
Identifier | Text in Log | Pattern |
|---|---|---|
MM - numeric month | 01-25-1986 | {date,MM-dd-yyyy} |
MMMMM - full textual month | 25/July/1986 | {date,dd/MMMMM/yyyy} |
MMM - textual month | 25/Jul/1986 | {date,dd/MMM/yyyy} |
dd - numeric day | 01:25:1986 | {date,MM:dd:yyyy} |
EEEEE - full textual day | Friday 01-25-00 | {date,EEEEE MM-dd-yy} |
EEE - textual day | Fri 01-25-00 | {date,EEE MM-dd-yy} |
yy - 2 digit year | 25/Jul/86 | {date,dd/MMM/yy} |
yyyy - 4 digit year | 25/Jul/1986 | {date,dd/MMM/yyyy} |
HH - 24 hour | 18:05:23 | {date,HH:mm:ss} |
hh - 12 hour | 6:05:23 PM | {date,hh:mm:ss} |
a - AM/PM marker | 6:05:23 PM | {date,hh:mm:ss a} |
mm - minute | 18-05-23 | {date,HH-mm-ss} |
ss - second | 18:05:23 | {date,HH:mm:ss} |
SSS - millisecond | 18:05:23 253 | {date,HH:mm:ss SSS} |
z - general time zone | 18:05:23 EST | {date,HH:mm:ss z} |
Z - RFC 822 time zone | 18:05:23 -0400 | {date,HH:mm:ss Z} |
X - ISO 8601 time zone | 18:05:23 -04:00 | {date,HH:mm:ss XXX} |
'TEXT' - a constant text that appears in the date string | 07-1986D25 | {date,MM-yyyy'D'dd} |
Attributes Supported by All Types
All pattern types support the attributes described in the following table.
Attribute | Remark | Examples |
|---|---|---|
Name | This attribute should always follow the tag name with a leading colon. | {string:Title} {date:Start Date,dd/mm/yyyy} {priority:Severity,DEBUG;INFO;ERROR} {number:Status Code} |
uiMessageLength | This attribute allows you to limit the length of data displayed in a specific column. If the data is longer than specified, it will be divided to several lines. | {string:Title,uiMessageLength=20} |
charsLength | This attribute allows you to force the existence of a fixed number of characters in a string, even if there are less characters in the record. | {string:Title,charsLength=10} |
stopPattern | This attribute allows you to set a regular expression that will serve as the column's delimiter. This is useful in case there is no natural delimiter (such as space or a non-word character) between two columns. | {text,stopPattern=\d+.\d+.\d+} |
masker | This attribute allows you to set a regular expression for masking the column's data. If a match is found for the column's value, then the matched part will be displayed as a string of asterisks ('*'). | {string:UserID,masker=(.+)} |
Special Cases
The right/left curly brackets characters ( { } ) are reserved in XpoLog syntax, therefore they can be used as literal only if as \u007B (left curly bracket) or \u007D (right curly bracket).
The quote character ( ' ) can be used as literal only if preceded by another quote ''.
Examples of Patterns Used on Logs
The following are examples of patterns that can be used to tune the parsing results of logs.
Log | Parsed Records | Data Pattern |
|---|---|---|
Log 1 | 2003-02-12 12:37:26 ContextConfig[/examples]: Missing application web.xml, using defaults only | {date,yyyy-MM-dd HH:mm:ss} {string}[/{string}]: {string} |
Log2 | 127.0.0.1 - - [26/Dec/2001:19:49:23 +0200] "GET / HTTP/1.1" 200 1494 | {string} - - [{date,dd/MMM/yyyy:HH:mm:ss Z} +0200] "{string}" {number} {number} |
Log3 | [Wed Dec 26 19:52:48 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/examples/ | [{string} {date,EEE MMM dd HH:mm:ss yyyy}] [{priority,debug;info;warn;error;fatal}] [{string} {string}] {string} |
Log4 | 28/02/03 20:23:16 ERR Critical error on section 34 on module 5 [Channel 9] 4.4.4.4 | {date,dd/MM/yy HH:mm:ss} {priority,DBG;FLW;WRN;ERR} {string} [{string}] {string} |
Log5 | 5 d MBGN Talk to port | {number}{tab}{string}{tab}{string}{tab}{string} |
Log6 | 5 d MBGN procId=123 Talk to port | {number:ID}TAB{text:Severity}TAB{text:Event ID}TAB{block,start,emptiness=true}procId={text:Proc ID}TAB{block,end,emptiness=true}{string:Message}{block,start,emptiness=true}TAB{block,end,emptiness=true} |