Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

XpoLog enables users to use the Patterns language to normalize log records into tabular format.

...

Pattern Type  Syntax Attributes Examples   
Text in Log PatternResult in Log View 
Stringany string of characters, including multi line strings{string}No special attributes

Hello world. 

Hello world.

Hello
world. 

{string}

H{string} wor{string}

{string}

Hello world.  

first column = ello; second column = ld.

Hello world.

Textany single-line string of characters{text}No special attributesHello world. {text}Hello world.
Datea date string{date, MM-dd-yyyy}

UI Foramt: after date object is parsed, its possible to modify its display by using ';' separator and specifying the desired view format



timeDiff: specify time offset in milliseconds
 localelocale: specify the locale in which the log was written
localeView: specify the locale in which the log should be displayed.

Also, see Optional Identifiers for Date Pattern section below.

Date in Log: 10/Jan/2013:11:32:38

 

 




18:05:23

18:05:23

{date,locale=it,dd MM yyyy HH:mm:ss}


{date,locale=it; localeView=en,dd MM yyyy}

Pattern to parse date:
{date:Date,dd/MMM/yyyy:HH:mm:ss}

Pattern with different display:
{date:Date,dd/MMM/yyyy:HH:mm:ss;yyyy-MM-dd HH:mm:ss}





{date,timeDiff=1000,HH:mm:ss}

{date,timeDiff=-5000,HH:mm:ss}

 
10/Jan/2013:11:32:38


2013-01-10 11:32:38




Timestamp  a timestamp representing a date string {timestamp}product: XpoLog expects a timestamp in milliseconds, in case the timestamp is not in milliseconds use the product to add zeros at the end
  • 1388534400000
  • 1414973208
  • {timestamp,yyyy/MM/dd HH:mm:ss.SSS}  
  • {timestamp,product=1000,yyyy-MM-dd HH:mm:ss.SSS}
  •  2014/01/01 02:00:00.000
  • 2014/11/03 02:06:48.000
 Number a numeric string {number}No special attributes

Thread-1

Test 5 done 

calling to id 5667

 {string}-{number}

{string} {number} {string}

calling to id {number}

first column = Thread;second column = 1

first column = Test; second column = 5; third column = done

5667

Choice   a set of strings that can appear in a record

 {choice,value1;value2...,valueN}

Note: All optional choices should appear as a semicolon separated list inside the tag.

No special attributesfirst record = red,
second record = green,
third record = blue  
 {choice,red;green;blue}The matching choice of each record is displayed.
IP Address An IP address {ip}No special attributes 127.0.0.1  {ip}  127.0.0.1
Geo IP AddressA Geo IP address

 {geoip,type=country:region:city}

Note: All optional combinations of country, region and city are valid, for example: country:region, region:city, country etc.

No special attributes

66.249.71.162

193.35.249.128

{geoip:Remote Host,type=country:region:city}

{geoip:Remote Host,type=country}

66.249.71.162 United States CA Mountain View

193.35.249.128 United Kingdom


PriorityA set of priorities that can appear in a record

 {priority,priority1;priority2...,priorityN}

Note: All optional priorities should appear as a semi-colon separated list inside the tag.

No special attributesfirst record = DEBUG,
second record = INFO,
third record = ERROR
{priority,DEBUG;INFO;ERROR} The matching priority of each record is displayed.
ExpressionThe expression that will be used according to the source columns given

 {expression}

No special attributes

Assuming you have two columns in your log, and you want to create a new column in your log view, represents a combining of these two records (highlighted in the example):

Mon Jul 10 04:33:51 2006 100 300

{date:Date,EEE MMM dd HH:mm:ss yyyy} {num:Num1} {num:Num2}{expression,Num2-Num1}

 


Mon Jul 10 04:33:51 2006 100 300 200

 Regular ExpressionA regular expression, used to extract part of the data from another column
read more about regular expressions in the regular expressions help page
 {regexp,refIndex=index | refName=column_name;columnType=date/timestamp/number;dateUIFormat=DISPLAY_DATE_FORMATmultiLine=true/false,(regular_expression_to_extract)}

refIndex/refName (mandatory): the zero-based index of the source column / the name of the source column

columnType (mandatory for date/timestamp only):
columnType=timestamp;dateFormat=<the desired format of the date>
columnType=date;dateFormat=<the format of the date in the log to be extracted> 
dateUIFormat=<the format of the date desired to be displayed in XpoLog> 

 


multiLine (optional): indicates whether the record spreads over more than one line

expression (mandatory): the regular expression that will be extracted out of the value in the source column 


See More Examples

Assuming you have the next record in your log, and you need to extract the error code (highlighted in the example) that is embedded in the message:

Mon Jul 10 04:33:51 2006 ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K
ORA-336 signalled during: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K...
 {date:Date,EEE MMM dd HH:mm:ss yyyy}{regexp:Error Code,refName=Message;multiLine=true,(ORA-\d+)}{string:Message}Mon Jul 10 04:33:51 2006 ORA-336 ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K
ORA-336 signalled during: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K,
GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K...  
PropertiesA set of key-value properties that can appear in a record{properties,keysSep=[Keys_Separator];propSep=[Properties_Separator];,key1;key2;....;keyN}

keysSep (mandatory): the character/s that separate the key from the value

propSep (mandatory): the character/s that separate properties

Assuming you have the next record in your log:

2014-06-02 09:06:45,100 Type=INFO, Account=admin, ID=54, Team=all

{date:Date,yyyy-MM-dd HH:mm:ss,SSS} {properties: pairs,keysSep==;propSep=,;,Type;Account;ID;Team}2014-06-02 09:06:45,100 INFO admin 54 all Type=INFO Account=admin ID=54 Team=all
  JsonA set of key-value pairs{json,key1;key2;key3;....;keyN}

No special attributes except date.If the date is inside the JSON object the following parameters may be specified:
#_#date::dateFormat=[THE_DATE_FORMAT_AS_APPEARS_IN_THE_DATA],dateUIFormat=[THE_DATE_FORMAT_AS_DESIRED_TO_BE_DISPLAYED_IN_XPOLOG]

Note: #_#date:: should be used as is as the separator between the date key name and the date parameters

Note: If one of the keys of a json is a json object then the full path from the top jason object should be speicfied using '->' to represent a level.

 



Assuming you have the next record in your log:

2014-11-24 12:02:32,083 {"red":"#f00", "green":"#0f0", "blue":"#00f", "yellow":"#ff0","black":"#000"}

 


Date usage example:
{"name":"x","hostname":"y","version":"z","time":"2017-07-01T00:00:00.000"}

 


JSON in JSON example:
{"name":"name","hostname":"hostname","pid":1,"level":1,"http details":{"URL":"http://xpolog.com","method":"GET"},"msg":"Message","time":"2017-11-01 00:00:00.000"}

{date:Date,yyyy-MM-dd HH:mm:ss,SSS} {json,red;green;blue;yellow}





{json:JSON,time#_#date::dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSS,dateUIFormat=yyyy-MM-dd HH:mm:ss.SSS;name;hostname;version}

{json:JSON,time#_#date::dateFormat=yyyy-MM-dd HH:mm:ss.SSS;name;hostname;pid;level;http details->URL;http details->method;msg}
2014-11-24 12:02:32,083 #f00 #0f0 #00f #ff0 {"black":"#000"}
 Term a constant string that appears in a record and needs to be displayed in the log view

{term,TERM}

TERM is the constant string

No special attributes Assuming you have the next two records in your log:

03/07/2005 03:44:56 app1 IP=192.168.11.44 success (where IP is a source IP)
03/07/2005 03:44:57 app2 IP=192.168.12.33 failure (where IP is a destination IP)
 Multi-pattern:
{date,dd/MM/yyyy HH:mm:ss} {term,app1} IP={text:Source IP} {string}
{date,dd/MM/yyyy HH:mm:ss} {term,app2} IP={text:Destination IP} {string}
 03/07/2005 03:44:56 app1 192.168.11.44 success
03/07/2005 03:44:57 app2 192.168.12.33 failure
 Free Text  any text you wish to display in the log view, usually used in multi-pattern logs to distinguish records

 {freetext,FREETEXT}

FREETEXT is the text that you wish to display in the log view.

No special attributes Assuming you have the next records in your log:

03/07/2005 03:44:56 $ success
03/07/2005 03:44:57 % failure

 Multi-pattern:

{date,dd/MM/yyyy HH:mm:ss} {freetext,Dollar}$ {string}
{date,dd/MM/yyyy HH:mm:ss} {freetext,Percentage}% {string}

Note: The freetext tag does not replace any text in the pattern, therefore it could be placed anywhere in the pattern and should not be separated with a space.

 03/07/2005 03:44:56 Dollar success
03/07/2005 03:44:57 Percentage failure
Value Mapping
Option I (manual mapping)
maps an original value from log to a converted value

{map,val1=convertedVal1;val2=convertedVal2;val3=convertedVal3;...;valN=convertedValN}

No special attributesAssuming you have in the log under one of the columns the following values:
200,304,404,500 

Using the following will convert the values as follows:

map:status,200=OK;304=Forbidden;404=Page
Not Found;500=Internal Error Server}

200 will be presented as 'OK'
304 will be presented as 'Forbidden'
404 will be presented as 'Page Not Found'
500 will be presented as 'Internal Error Server'
Value Mapping
Option II
(mapping based on an external file)
maps an original value from log to a converted value

{map,refIndex=ORIG_COL_INDEX,file:FULL_PATH_TO_FILE}

refIndex: numeric value which represents the index of the column in which its values will be converted (first column index is 0)

file:
Full path to the properties file that will be used to get the converted values 

Assuming you have in the log under one of the columns the following values:
200,304,404,500 

Using the following will convert the values as follows:

{map,refIndex=8,file:C:\XpoLogConfig\HTTP_VALUES.txt}

  • Index 8 is the index of the original column in the log which its values will be converted
  • C:\XpoLogConfig\HTTP_VALUES.txt is the file containing the following:
    200=OK
    304=Forbidden
    404=Page Not Found
    500=Internal Error Server
  • A new column will be created with the converted values
200 will be presented as 'OK'
304 will be presented as 'Forbidden'
404 will be presented as 'Page Not Found'
500 will be presented as 'Internal Error Server'
Value Mapping
Option III (regular expression manual mapping)
maps an original value from log to a converted value{regexp,refIndex=index | refName=column_name;columnType=map;mapping=val1:convertedVal1^val2:convertedVal2^...^valN:convertedValN,(regular_expression_to_extract)}

refIndex/refName (mandatory): the zero-based index of the source column / the name of the source column

columnType (mandatory)
  

mapping (mandatory)

expression (mandatory): the regular expression that will be extracted out of the value in the source column

Assuming you have the next record in your log, and you need to map the Logon Type (highlighted in the example) that is embedded in the message:

1400504701000*;*Microsoft-Windows-Security-Auditing*;*A logon was attempted using explicit credentials.

    Security ID:        S-1-0-0

    Logon ID:        0xdc01

    Logon Type: 3

{timestamp:Date,MM/dd/yyyy HH:mm:ss}*;*{text:Source}*;*{regexp:Logon Type,refName=description;columnType=map;mapping=2:Local^3:Network,Logon Type: ([\d]+).*}{string:Description}




 05/19/2014 16:05:01 Microsoft-Windows-Security-Auditing Network A logon was attempted using explicit credentials. Security ID:        S-1-0-0 Logon ID:        0xdc01 Logon Type: 3
 Block an optional string that does not appear in all records {block,start,emptiness=true}XXX{block,end,emptiness=true} start: indicates a block start

end: indicates a block end

emptiness: indicates whether the block's data could be missing

 first record = 64.236.16.52:8080,
second record = 64.236.16.52,
third record = 64.236.16.52:7001
 {string}{block,start,emptiness=true}:{string}{block,end,emptiness=true} 
 Horizontal Tab a tab delimiter {tab} No special attributes one 123 {string}{tab}{number}first column = one,
second column = 123
 End of Line end of line, used in records that spread over more than one line

 {eol} No special attributesname : logger
level : db0 name :
 {string}{eol}level : {priority,db0;db1}first column = logger,
second column = db0
 End of Entity end of entity, used to mark the end of a record, improves the efficiency of the parsing process

 {eoe}

 No special attributesname : logger
level : db0
name : logger
level : db1 name :
 string}{eol}level : {priority,db0;db1}{eoe}

first row = logger,db0
second row = logger,db1

...

LogParsed RecordsData Pattern
Log 12003-02-12 12:37:26 ContextConfig[/examples]: Missing application web.xml, using defaults only
2003-02-12 12:37:26 StandardManager[/examples]: Seeding random number generator class java.security.SecureRandom
2003-02-12 12:37:30 StandardManager[/examples]: Seeding of random number generator has been completed
2003-02-12 12:37:30 StandardWrapper[/examples:default]: Loading container servlet default
2003-02-12 12:37:30 StandardWrapper[/examples:invoker]: Loading container servlet invoker
{date,yyyy-MM-dd HH:mm:ss} {string}[/{string}]: {string}
Log2127.0.0.1 - - [26/Dec/2001:19:49:23 +0200] "GET / HTTP/1.1" 200 1494
127.0.0.1 - - [26/Dec/2001:19:49:23 +0200] "GET /apache_pb.gif HTTP/1.1" 200 2326
127.0.0.1 - - [26/Dec/2001:19:52:48 +0200] "GET /examples/ HTTP/1.1" 404 277
127.0.0.1 - - [26/Dec/2001:19:54:37 +0200] "GET /examples/jsp/snp/snoop.jsp HTTP/1.1" 404 294
127.0.0.1 - - [28/Dec/2001:09:54:37 +0200] "GET /puga/main.html HTTP/1.1" 404 282
{string} - - [{date,dd/MMM/yyyy:HH:mm:ss Z} +0200] "{string}" {number} {number}
Log3[Wed Dec 26 19:52:48 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/examples/
[Wed Dec 26 19:55:01 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/_vti_bin/owssvr.dll
[Wed Dec 26 19:55:01 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/msoffice/cltreq.asp
[{string} {date,EEE MMM dd HH:mm:ss yyyy] [{priority,debug;info;warn;error;fatal}] [{string} {string}] {string}
 Log4 28/02/03 20:23:16 ERR Critical error on section 34 on module 5 [Channel 9] 4.4.4.4
28/02/03 20:25:35 DBG information arrived to fusion zone, restoring states [Channel 39] 4.8.4.9
28/02/03 20:33:22 WRN port collision seeking another [Channel 19] 4.4.4.4
28/02/03 20:33:22 FLW DB connection open structure initiated [Channel 9] 4.23.12.5
{date,dd/MM/yy HH:mm:ss} {priority,DBG;FLW;WRN;ERR} {string} [{string}] {string} 
 Log5 5 d MBGN Talk to port
9 f MLPT1 Port open
0 x MCOM Com port open
 {number}{tab}{string}{tab}{string}{tab}{string}
 Log65 d MBGN procId=123 Talk to port
9 f MLPT1 Port open
0 x MCOM procId=456 Com port open
 {number}{tab}{string}{tab}{block,start,emptiness=true}procId={string}{block,end,emptiness=true}{string}{tab}{string}