XPLG Patterns Language

Text in Log

String

any string of characters, including multi line strings

{string}

No special attributes

Hello world. 

Hello world.

Hello
world. 

{string}

H{string} wor{string}

{string}

Hello world.  

first column = ello; second column = ld.

Hello world.

Text

any single-line string of characters

{text}

No special attributes

Hello world.

 {text}

Hello world.

Date

a date string

{date, MM-dd-yyyy}

UI Foramt: after date object is parsed, its possible to modify its display by using ';' separator and specifying the desired view format

timeDiff: specify time offset in milliseconds
 locale: specify the locale in which the log was written
localeView: specify the locale in which the log should be displayed.

Also, see Optional Identifiers for Date Pattern section below.

Date in Log: 10/Jan/2013:11:32:38

18:05:23

18:05:23

{date,locale=it,dd MM yyyy HH:mm:ss}

{date,locale=it; localeView=en,dd MM yyyy}

Pattern to parse date:
{date:Date,dd/MMM/yyyy:HH:mm:ss}

Pattern with different display:
{date:Date,dd/MMM/yyyy:HH:mm:ss;yyyy-MM-dd HH:mm:ss}

{date,timeDiff=1000,HH:mm:ss}

{date,timeDiff=-5000,HH:mm:ss}

 
10/Jan/2013:11:32:38


2013-01-10 11:32:38

Timestamp 

 a timestamp representing a date string

 {timestamp}

product: XpoLog expects a timestamp in milliseconds, in case the timestamp is not in milliseconds use the product to add zeros at the end

• 1388534400000

• 1414973208

• {timestamp,yyyy/MM/dd HH:mm:ss.SSS}  

• {timestamp,product=1000,yyyy-MM-dd HH:mm:ss.SSS}

•  2014/01/01 02:00:00.000

• 2014/11/03 02:06:48.000

 Number

 a numeric string

 {number}

No special attributes

Thread-1

Test 5 done 

calling to id 5667

 {string}-{number}

{string} {number} {string}

calling to id {number}

first column = Thread;second column = 1

first column = Test; second column = 5; third column = done

5667

Choice 

  a set of strings that can appear in a record

 {choice,value1;value2...,valueN}

Note: All optional choices should appear as a semicolon separated list inside the tag.

No special attributes

first record = red,
second record = green,
third record = blue  

 {choice,red;green;blue}

The matching choice of each record is displayed.

IP Address

 An IP address

 {ip}

No special attributes

 127.0.0.1

  {ip}

  127.0.0.1

Geo IP Address

A Geo IP address

 {geoip,type=country:region:city}

Note: All optional combinations of country, region and city are valid, for example: country:region, region:city, country etc.

No special attributes

66.249.71.162

193.35.249.128

{geoip:Remote Host,type=country:region:city}

{geoip:Remote Host,type=country}

66.249.71.162 United States CA Mountain View

193.35.249.128 United Kingdom

Priority

A set of priorities that can appear in a record

 {priority,priority1;priority2...,priorityN}

Note: All optional priorities should appear as a semi-colon separated list inside the tag.

No special attributes

first record = DEBUG,
second record = INFO,
third record = ERROR

{priority,DEBUG;INFO;ERROR} 

The matching priority of each record is displayed.

Expression (internal)

The expression that will be used according to the source columns given

 {expression}

No special attributes

Assuming you have two columns in your log, and you want to create a new column in your log view, represents a combining of these two records (highlighted in the example):

Mon Jul 10 04:33:51 2024 “100” “300”

{date:Date,EEE MMM dd HH:mm:ss yyyy} “{text:Num1}” “{text:Num2}”{expression,Num2-Num1}

Mon Jul 10 04:33:51 2024 100 300 200

Expression (external -JavaScript)

The expression that will be used according to the source columns given

 {expression}

JavaScript file reference and one or more columns that the JavaScript expects

You can send a column or multiple columns to an external JavaScript file that will perform any operation on them and return the result to a new column in the log.
Note: its possible to send multiple log columns to a JavaScript function that will do something with them and return a result.

For example, let’s convert a number in a log column from hexadecimal value to decimal value using an external JavaScript function.

In XPLG configuration directory create a new directory - scripts. Inside place a file convertHexToDec.js with the following contents:

function execute(col1){
var result = hexToDecimal(col1);
return result;

}

function hexToDecimal(paramValue) {
// Use the parseInt function with a radix of 16
if (paramValue && paramValue.length > 0) {
return parseInt(paramValue, 16);
} else {
return paramValue; // or return an error or empty string based on your requirement
}
}

Log line:

Mon Jul 10 04:33:51 2024 “100”

{date:Date,EEE MMM dd HH:mm:ss yyyy} "{text:Num1}"{expression:Num1InHexa,js("scripts/convertHexToDec.js",column.Num1)}

Mon Jul 10 04:33:51 2024 100 74565

 Regular Expression

A regular expression, used to extract part of the data from another column
read more about regular expressions in the regular expressions help page

 {regexp,refIndex=index / refName=column_name;multiLine=true/false,(regular_expression_to_extract)}

OR for date/timestamp

{regexp,refIndex=index OR refName=column_name;columnType=date/timestamp;dateFormat=DATE_FORMAT_IN_DATA;dateUIFormat=DISPLAY_DATE_FORMAT;multiLine=true/false,(regular_expression_to_extract)}

refIndex/refName (mandatory): the zero-based index of the source column / the name of the source column

columnType (mandatory for date/timestamp only):
columnType=timestamp;dateFormat=<the desired format of the date>

columnType=date;
dateFormat=<the format of the date in the log to be extracted> 
dateUIFormat=<the format of the date desired to be displayed in XpoLog> 

multiLine (optional): indicates whether the record spreads over more than one line

expression (mandatory): the regular expression that will be extracted out of the value in the source column

See More Examples

Assuming you have the next record in your log, and you need to extract the error code (highlighted in the example) that is embedded in the message:

Mon Jul 10 04:33:51 2006 ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K
ORA-336 signalled during: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K...

["case_ID" : "1234", "created_at" : "2023-06-28 16:37:01"]

 {date:Date,EEE MMM dd HH:mm:ss yyyy}{regexp:Error Code,refName=Message;multiLine=true,(ORA-\d+)}{string:Message}

{text:Message}{regexp:created_at,refName=Message;columnType=date;dateFormat=yyyy-MM-dd HH:mm:ss,"created_at" : "([^"]+)}{regexp:case_ID,refName=Message;,case_ID" : "([^"]+)}

Mon Jul 10 04:33:51 2006 ORA-336 ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K
ORA-336 signalled during: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K...  

Properties

A set of key-value properties that can appear in a record

{properties,keysSep=[Keys_Separator];propSep=[Properties_Separator];,key1;key2;....;keyN}

keysSep (mandatory): the character/s that separate the key from the value

propSep (mandatory): the character/s that separate properties

Assuming you have the next record in your log:

2014-06-02 09:06:45,100 Type=INFO, Account=admin, ID=54, Team=all

{date:Date,yyyy-MM-dd HH:mm:ss,SSS} {properties: pairs,keysSep==;propSep=,;,Type;Account;ID;Team}

2014-06-02 09:06:45,100 INFO admin 54 all Type=INFO Account=admin ID=54 Team=all

  Json

A set of key-value pairs

{json,key1;key2;key3;....;keyN}

No special attributes except date.If the date is inside the JSON object the following parameters may be specified:
#_#date::dateFormat=[THE_DATE_FORMAT_AS_APPEARS_IN_THE_DATA],dateUIFormat=[THE_DATE_FORMAT_AS_DESIRED_TO_BE_DISPLAYED_IN_XPOLOG]

Note: #_#date:: should be used as is as the separator between the date key name and the date parameters

Note: If one of the keys of a json is a json object then the full path from the top json object should be specified using '->' to represent a level.

Note: In case that you would like to grant ftype to your json column, use the following syntax: JSON_KEY#_#ftype=[FTYPE]

Example: data->row->virtualMemory#_#ftype=virtualMemory

Note: In case that you would like to grant a different name to your json column (mostly needed during parsing of json within json, use the following syntax: JSON_KEY#_#name=[Column_Name]

Example: ;data->row->instanceCount#_#name=instanceCount

Granting both ftype and column name example: data->row->arguments#_#ftype=args#_#name=Arguments

Assuming you have the next record in your log:

2014-11-24 12:02:32,083 {"red":"#f00", "green":"#0f0", "blue":"#00f", "yellow":"#ff0","black":"#000"}

Date usage example:
{"name":"x","hostname":"y","version":"z","time":"2017-07-01T00:00:00.000"}

JSON in JSON example:
{"name":"name","hostname":"hostname","pid":1,"level":1,"http details":{"URL":"http://xpolog.com","method":"GET"},"msg":"Message","time":"2017-11-01 00:00:00.000"}

{date:Date,yyyy-MM-dd HH:mm:ss,SSS} {json,red;green;blue;yellow}





{json:JSON,time#_#date::dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSS,dateUIFormat=yyyy-MM-dd HH:mm:ss.SSS;name;hostname;version}

{json:JSON,time#_#date::dateFormat=yyyy-MM-dd HH:mm:ss.SSS;name;hostname;pid;level;http details->URL;http details->method;msg}

2014-11-24 12:02:32,083 #f00 #0f0 #00f #ff0 {"black":"#000"}

 Term

 a constant string that appears in a record and needs to be displayed in the log view

{term,TERM}

TERM is the constant string

No special attributes

 Assuming you have the next two records in your log:

03/07/2005 03:44:56 app1 IP=192.168.11.44 success (where IP is a source IP)
03/07/2005 03:44:57 app2 IP=192.168.12.33 failure (where IP is a destination IP)

 Multi-pattern:
{date,dd/MM/yyyy HH:mm:ss} {term,app1} IP={text:Source IP} {string}
{date,dd/MM/yyyy HH:mm:ss} {term,app2} IP={text:Destination IP} {string}

 03/07/2005 03:44:56 app1 192.168.11.44 success
03/07/2005 03:44:57 app2 192.168.12.33 failure

 Free Text

  any text you wish to display in the log view, usually used in multi-pattern logs to distinguish records

 {freetext,FREETEXT}

FREETEXT is the text that you wish to display in the log view.

No special attributes 

Assuming you have the next records in your log:

03/07/2005 03:44:56 $ success
03/07/2005 03:44:57 % failure

 Multi-pattern:

{date,dd/MM/yyyy HH:mm:ss} {freetext,Dollar}$ {string}
{date,dd/MM/yyyy HH:mm:ss} {freetext,Percentage}% {string}

Note: The freetext tag does not replace any text in the pattern, therefore it could be placed anywhere in the pattern and should not be separated with a space.

 03/07/2005 03:44:56 Dollar success
03/07/2005 03:44:57 Percentage failure

Value Mapping
Option I (manual mapping)

maps an original value from log to a converted value

{map,val1=convertedVal1;val2=convertedVal2;val3=convertedVal3;...;valN=convertedValN}

No special attributes

Assuming you have in the log under one of the columns the following values:
200,304,404,500 

Using the following will convert the values as follows:

map:status,200=OK;304=Forbidden;404=Page Not Found;500=Internal Error Server}

200 will be presented as 'OK'
304 will be presented as 'Forbidden'
404 will be presented as 'Page Not Found'
500 will be presented as 'Internal Error Server'

Value Mapping
Option II
(mapping based on an external file)

maps an original value from log to a converted value

{map,refIndex=ORIG_COL_INDEX,file:FULL_PATH_TO_FILE}

refIndex: numeric value which represents the index of the column in which its values will be converted (first column index is 0)

file:
Full path to the properties file that will be used to get the converted values 

Assuming you have in the log under one of the columns the following values:
200,304,404,500 

Using the following will convert the values as follows:

{map,refIndex=8,file:C:\XpoLogConfig\HTTP_VALUES.txt}

• Index 8 is the index of the original column in the log which its values will be converted

• C:\XpoLogConfig\HTTP_VALUES.txt is the file containing the following:
  200=OK
  304=Forbidden
  404=Page Not Found
  500=Internal Error Server

• A new column will be created with the converted values

200 will be presented as 'OK'
304 will be presented as 'Forbidden'
404 will be presented as 'Page Not Found'
500 will be presented as 'Internal Error Server'

Value Mapping
Option III (regular expression manual mapping)

maps an original value from log to a converted value

{regexp,refIndex=index | refName=column_name;columnType=map;mapping=val1:convertedVal1^val2:convertedVal2^...^valN:convertedValN,(regular_expression_to_extract)}

refIndex/refName (mandatory): the zero-based index of the source column / the name of the source column

columnType (mandatory)
  

mapping (mandatory)

expression (mandatory): the regular expression that will be extracted out of the value in the source column

Assuming you have the next record in your log, and you need to map the Logon Type (highlighted in the example) that is embedded in the message:

1400504701000*;*Microsoft-Windows-Security-Auditing*;*A logon was attempted using explicit credentials.

    Security ID:        S-1-0-0

    Logon ID:        0xdc01

    Logon Type: 3

{timestamp:Date,MM/dd/yyyy HH:mm:ss}*;*{text:Source}*;*{regexp:Logon Type,refName=description;columnType=map;mapping=2:Local^3:Network,Logon Type: ([\d]+).*}{string:Description}

 05/19/2014 16:05:01 Microsoft-Windows-Security-Auditing Network A logon was attempted using explicit credentials. Security ID:        S-1-0-0 Logon ID:        0xdc01 Logon Type: 3

 Block

 an optional string that does not appear in all records

 {block,start,emptiness=true}XXX{block,end,emptiness=true}

 start: indicates a block start

end: indicates a block end

emptiness: indicates whether the block's data could be missing

 first record = 64.236.16.52:8080,
second record = 64.236.16.52,
third record = 64.236.16.52:7001

 {string}{block,start,emptiness=true}:{string}{block,end,emptiness=true}

 Horizontal Tab

 a tab delimiter

 {tab}

 No special attributes

 one 123

 {string}{tab}{number}

first column = one,
second column = 123

 End of Line

 end of line, used in records that spread over more than one line

 {eol}

 No special attributes

name : logger
level : db0 name :

 {string}{eol}level : {priority,db0;db1}

first column = logger,
second column = db0

 End of Entity

 end of entity, used to mark the end of a record, improves the efficiency of the parsing process

 {eoe}

 No special attributes

name : logger
level : db0
name : logger
level : db1 name :

 string}{eol}level : {priority,db0;db1}{eoe}

first row = logger,db0
second row = logger,db1

MM - numeric month

01-25-1986  

{date,MM-dd-yyyy}

MMMMM - full textual month

25/July/1986

{date,dd/MMMMM/yyyy}

MMM - textual month

25/Jul/1986

{date,dd/MMM/yyyy}

dd - numeric day

 01:25:1986

{date,MM:dd:yyyy} 

EEEEE - full textual day 

 Friday 01-25-00

{date,EEEEE MM-dd-yy}

EEE - textual day

 Fri 01-25-00

 {date,EEE MM-dd-yy}

yy - 2 digit year 

 25/Jul/86

 {date,dd/MMM/yy}

 yyyy - 4 digit year

 25/Jul/1986

 {date,dd/MMM/yyyy}

 HH - 24 hour

 18:05:23 

 {date,HH:mm:ss}

hh - 12 hour

6:05:23 PM

{date,hh:mm:ss}

 a - AM/PM marker

6:05:23 PM

{date,hh:mm:ss a}

 mm - minute

 18-05-23

  {date,HH-mm-ss}

 ss - second

 18:05:23

 {date,HH:mm:ss}

 SSS - millisecond

 18:05:23 253

 {date,HH:mm:ss SSS}

 z - general time zone 

 18:05:23 EST

 {date,HH:mm:ss z}

 Z - RFC 822 time zone

 18:05:23 -0400

 {date,HH:mm:ss Z}

 X - ISO 8601 time zone

 18:05:23 -04:00

 {date,HH:mm:ss XXX}

 'TEXT' - a constant text that appears in the date string

 07-1986D25

 {date,MM-yyyy'D'dd}

Name

This attribute should always follow the tag name with a leading colon.

{string:Title}

{date:Start Date,dd/mm/yyyy}

{priority:Severity,DEBUG;INFO;ERROR}

{number:Status Code}

uiMessageLength 

 This attribute allows you to limit the length of data displayed in a specific column. If the data is longer than specified, it will be divided to several lines.

 {string:Title,uiMessageLength=20}

charsLength

 This attribute allows you to force the existence of a fixed number of characters in a string, even if there are less characters in the record.

 {string:Title,charsLength=10}

stopPattern

This attribute allows you to set a regular expression that will serve as the column's delimiter. This is useful in case there is no natural delimiter (such as space or a non-word character) between two columns.

{text,stopPattern=\d+.\d+.\d+}

masker

This attribute allows you to set a regular expression for masking the column's data. If a match is found for the column's value, then the matched part will be displayed as a string of asterisks ('*').

{string:UserID,masker=(.+)}

Log 1

2003-02-12 12:37:26 ContextConfig[/examples]: Missing application web.xml, using defaults only
2003-02-12 12:37:26 StandardManager[/examples]: Seeding random number generator class java.security.SecureRandom
2003-02-12 12:37:30 StandardManager[/examples]: Seeding of random number generator has been completed
2003-02-12 12:37:30 StandardWrapper[/examples:default]: Loading container servlet default
2003-02-12 12:37:30 StandardWrapper[/examples:invoker]: Loading container servlet invoker

{date,yyyy-MM-dd HH:mm:ss} {string}[/{string}]: {string}

Log2

127.0.0.1 - - [26/Dec/2001:19:49:23 +0200] "GET / HTTP/1.1" 200 1494
127.0.0.1 - - [26/Dec/2001:19:49:23 +0200] "GET /apache_pb.gif HTTP/1.1" 200 2326
127.0.0.1 - - [26/Dec/2001:19:52:48 +0200] "GET /examples/ HTTP/1.1" 404 277
127.0.0.1 - - [26/Dec/2001:19:54:37 +0200] "GET /examples/jsp/snp/snoop.jsp HTTP/1.1" 404 294
127.0.0.1 - - [28/Dec/2001:09:54:37 +0200] "GET /puga/main.html HTTP/1.1" 404 282

{string} - - [{date,dd/MMM/yyyy:HH:mm:ss Z} +0200] "{string}" {number} {number}

Log3

[Wed Dec 26 19:52:48 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/examples/
[Wed Dec 26 19:55:01 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/_vti_bin/owssvr.dll
[Wed Dec 26 19:55:01 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/msoffice/cltreq.asp

[{string} {date,EEE MMM dd HH:mm:ss yyyy}] [{priority,debug;info;warn;error;fatal}] [{string} {string}] {string}

 Log4

 28/02/03 20:23:16 ERR Critical error on section 34 on module 5 [Channel 9] 4.4.4.4
28/02/03 20:25:35 DBG information arrived to fusion zone, restoring states [Channel 39] 4.8.4.9
28/02/03 20:33:22 WRN port collision seeking another [Channel 19] 4.4.4.4
28/02/03 20:33:22 FLW DB connection open structure initiated [Channel 9] 4.23.12.5

{date,dd/MM/yy HH:mm:ss} {priority,DBG;FLW;WRN;ERR} {string} [{string}] {string} 

 Log5

 5 d MBGN Talk to port
9 f MLPT1 Port open
0 x MCOM Com port open

 {number}{tab}{string}{tab}{string}{tab}{string}

 Log6

5 d MBGN procId=123 Talk to port
9 f MLPT1 Port open
0 x MCOM procId=456 Com port open

{number:ID}TAB{text:Severity}TAB{text:Event ID}TAB{block,start,emptiness=true}procId={text:Proc ID}TAB{block,end,emptiness=true}{string:Message}{block,start,emptiness=true}TAB{block,end,emptiness=true}