XpoLog enables users to use the Patterns language to normalize log records into tabular format.
...
Pattern Type | Syntax | Attributes | Examples | |||
---|---|---|---|---|---|---|
Text in Log | Pattern | Result in Log View | ||||
String | any string of characters, including multi line strings | {string} | No special attributes | Hello world. Hello world. Hello | {string} H{string} wor{string} {string} | Hello world. first column = ello; second column = ld. Hello world. |
Text | any single-line string of characters | {text} | No special attributes | Hello world. | {text} | Hello world. |
Date | a date string | {date, MM-dd-yyyy} | UI Foramt: after date object is parsed, its possible to modify its display by using ';' separator and specifying the desired view format
Also, see Optional Identifiers for Date Pattern section below. | Date in Log: 10/Jan/2013:11:32:38
18:05:23 {date,locale=it,dd MM yyyy HH:mm:ss}
| Pattern to parse date:
{date,timeDiff=-5000,HH:mm:ss} | |
Timestamp | a timestamp representing a date string | {timestamp} | product: XpoLog expects a timestamp in milliseconds, in case the timestamp is not in milliseconds use the product to add zeros at the end |
|
|
|
Number | a numeric string | {number} | No special attributes | Thread-1 Test 5 done calling to id 5667 | {string}-{number} {string} {number} {string} calling to id {number} | first column = Thread;second column = 1 first column = Test; second column = 5; third column = done 5667 |
Choice | a set of strings that can appear in a record | {choice,value1;value2...,valueN} Note: All optional choices should appear as a semicolon separated list inside the tag. | No special attributes | first record = red, second record = green, third record = blue | {choice,red;green;blue} | The matching choice of each record is displayed. |
IP Address | An IP address | {ip} | No special attributes | 127.0.0.1 | {ip} | 127.0.0.1 |
Geo IP Address | A Geo IP address | {geoip,type=country:region:city} Note: All optional combinations of country, region and city are valid, for example: country:region, region:city, country etc. | No special attributes | 66.249.71.162 193.35.249.128 | {geoip:Remote Host,type=country:region:city} {geoip:Remote Host,type=country} | 66.249.71.162 United States CA Mountain View 193.35.249.128 United Kingdom |
Priority | A set of priorities that can appear in a record | {priority,priority1;priority2...,priorityN} Note: All optional priorities should appear as a semi-colon separated list inside the tag. | No special attributes | first record = DEBUG, second record = INFO, third record = ERROR | {priority,DEBUG;INFO;ERROR} | The matching priority of each record is displayed. |
Expression | The expression that will be used according to the source columns given | {expression} | No special attributes | Assuming you have two columns in your log, and you want to create a new column in your log view, represents a combining of these two records (highlighted in the example): Mon Jul 10 04:33:51 2006 100 300 | {date:Date,EEE MMM dd HH:mm:ss yyyy} {num:Num1} {num:Num2}{expression,Num2-Num1}
| Mon Jul 10 04:33:51 2006 100 300 200 |
Regular Expression | A regular expression, used to extract part of the data from another column read more about regular expressions in the regular expressions help page | {regexp,refIndex=index | refName=column_name;columnType=date/timestamp/number;dateUIFormat=DISPLAY_DATE_FORMATmultiLine=true/false,(regular_expression_to_extract)} | refIndex/refName (mandatory): the zero-based index of the source column / the name of the source column columnType (mandatory for date/timestamp only):
multiLine (optional): indicates whether the record spreads over more than one line | Assuming you have the next record in your log, and you need to extract the error code (highlighted in the example) that is embedded in the message: Mon Jul 10 04:33:51 2006 ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K, GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K ORA-336 signalled during: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K, GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K... | {date:Date,EEE MMM dd HH:mm:ss yyyy}{regexp:Error Code,refName=Message;multiLine=true,(ORA-\d+)}{string:Message} | Mon Jul 10 04:33:51 2006 ORA-336 ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K, GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K ORA-336 signalled during: ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 3 ('/oradata/PROD/redo.log') SIZE 200K, GROUP 4 ('/oradata/PROD/redo.log') SIZE 200K... |
Properties | A set of key-value properties that can appear in a record | {properties,keysSep=[Keys_Separator];propSep=[Properties_Separator];,key1;key2;....;keyN} | keysSep (mandatory): the character/s that separate the key from the value propSep (mandatory): the character/s that separate properties | Assuming you have the next record in your log: 2014-06-02 09:06:45,100 Type=INFO, Account=admin, ID=54, Team=all | {date:Date,yyyy-MM-dd HH:mm:ss,SSS} {properties: pairs,keysSep==;propSep=,;,Type;Account;ID;Team} | 2014-06-02 09:06:45,100 INFO admin 54 all Type=INFO Account=admin ID=54 Team=all |
Json | A set of key-value pairs | {json,key1;key2;key3;....;keyN} | No special attributes except date.If the date is inside the JSON object the following parameters may be specified: Note: If one of the keys of a json is a json object then the full path from the top jason object should be speicfied using '->' to represent a level.
| Assuming you have the next record in your log: 2014-11-24 12:02:32,083 {"red":"#f00", "green":"#0f0", "blue":"#00f", "yellow":"#ff0","black":"#000"}
Date usage example:
JSON in JSON example: | {date:Date,yyyy-MM-dd HH:mm:ss,SSS} {json,red;green;blue;yellow} {json:JSON,time#_#date::dateFormat=yyyy-MM-dd'T'HH:mm:ss.SSS,dateUIFormat=yyyy-MM-dd HH:mm:ss.SSS;name;hostname;version} {json:JSON,time#_#date::dateFormat=yyyy-MM-dd HH:mm:ss.SSS;name;hostname;pid;level;http details->URL;http details->method;msg} | 2014-11-24 12:02:32,083 #f00 #0f0 #00f #ff0 {"black":"#000"} |
Term | a constant string that appears in a record and needs to be displayed in the log view | {term,TERM} TERM is the constant string | No special attributes | Assuming you have the next two records in your log: 03/07/2005 03:44:56 app1 IP=192.168.11.44 success (where IP is a source IP) 03/07/2005 03:44:57 app2 IP=192.168.12.33 failure (where IP is a destination IP) | Multi-pattern: {date,dd/MM/yyyy HH:mm:ss} {term,app1} IP={text:Source IP} {string} {date,dd/MM/yyyy HH:mm:ss} {term,app2} IP={text:Destination IP} {string} | 03/07/2005 03:44:56 app1 192.168.11.44 success 03/07/2005 03:44:57 app2 192.168.12.33 failure |
Free Text | any text you wish to display in the log view, usually used in multi-pattern logs to distinguish records | {freetext,FREETEXT} FREETEXT is the text that you wish to display in the log view. | No special attributes | Assuming you have the next records in your log: 03/07/2005 03:44:56 $ success 03/07/2005 03:44:57 % failure | Multi-pattern: {date,dd/MM/yyyy HH:mm:ss} {freetext,Dollar}$ {string} | 03/07/2005 03:44:56 Dollar success 03/07/2005 03:44:57 Percentage failure |
Value Mapping Option I (manual mapping) | maps an original value from log to a converted value | {map,val1=convertedVal1;val2=convertedVal2;val3=convertedVal3;...;valN=convertedValN} | No special attributes | Assuming you have in the log under one of the columns the following values: 200,304,404,500 | Using the following will convert the values as follows: | 200 will be presented as 'OK' 304 will be presented as 'Forbidden' 404 will be presented as 'Page Not Found' 500 will be presented as 'Internal Error Server' |
Value Mapping Option II (mapping based on an external file) | maps an original value from log to a converted value | {map,refIndex=ORIG_COL_INDEX,file:FULL_PATH_TO_FILE} | refIndex: numeric value which represents the index of the column in which its values will be converted (first column index is 0) file: | Assuming you have in the log under one of the columns the following values: 200,304,404,500 | Using the following will convert the values as follows:
| 200 will be presented as 'OK' 304 will be presented as 'Forbidden' 404 will be presented as 'Page Not Found' 500 will be presented as 'Internal Error Server' |
Value Mapping Option III (regular expression manual mapping) | maps an original value from log to a converted value | {regexp,refIndex=index | refName=column_name;columnType=map;mapping=val1:convertedVal1^val2:convertedVal2^...^valN:convertedValN,(regular_expression_to_extract)} | refIndex/refName (mandatory): the zero-based index of the source column / the name of the source column columnType (mandatory) mapping (mandatory) | Assuming you have the next record in your log, and you need to map the Logon Type (highlighted in the example) that is embedded in the message: 1400504701000*;*Microsoft-Windows-Security-Auditing*;*A logon was attempted using explicit credentials. Security ID: S-1-0-0 Logon ID: 0xdc01 Logon Type: 3 | {timestamp:Date,MM/dd/yyyy HH:mm:ss}*;*{text:Source}*;*{regexp:Logon Type,refName=description;columnType=map;mapping=2:Local^3:Network,Logon Type: ([\d]+).*}{string:Description} | 05/19/2014 16:05:01 Microsoft-Windows-Security-Auditing Network A logon was attempted using explicit credentials. Security ID: S-1-0-0 Logon ID: 0xdc01 Logon Type: 3 |
Block | an optional string that does not appear in all records | {block,start,emptiness=true}XXX{block,end,emptiness=true} | start: indicates a block start end: indicates a block end emptiness: indicates whether the block's data could be missing | first record = 64.236.16.52:8080, second record = 64.236.16.52, third record = 64.236.16.52:7001 | {string}{block,start,emptiness=true}:{string}{block,end,emptiness=true} | |
Horizontal Tab | a tab delimiter | {tab} | No special attributes | one 123 | {string}{tab}{number} | first column = one, second column = 123 |
End of Line | end of line, used in records that spread over more than one line | {eol} | No special attributes | name : logger level : db0 name : | {string}{eol}level : {priority,db0;db1} | first column = logger, second column = db0 |
End of Entity | end of entity, used to mark the end of a record, improves the efficiency of the parsing process | {eoe} | No special attributes | name : logger level : db0 name : logger level : db1 name : | string}{eol}level : {priority,db0;db1}{eoe} | first row = logger,db0 |
...
Log | Parsed Records | Data Pattern |
---|---|---|
Log 1 | 2003-02-12 12:37:26 ContextConfig[/examples]: Missing application web.xml, using defaults only 2003-02-12 12:37:26 StandardManager[/examples]: Seeding random number generator class java.security.SecureRandom 2003-02-12 12:37:30 StandardManager[/examples]: Seeding of random number generator has been completed 2003-02-12 12:37:30 StandardWrapper[/examples:default]: Loading container servlet default 2003-02-12 12:37:30 StandardWrapper[/examples:invoker]: Loading container servlet invoker | {date,yyyy-MM-dd HH:mm:ss} {string}[/{string}]: {string} |
Log2 | 127.0.0.1 - - [26/Dec/2001:19:49:23 +0200] "GET / HTTP/1.1" 200 1494 127.0.0.1 - - [26/Dec/2001:19:49:23 +0200] "GET /apache_pb.gif HTTP/1.1" 200 2326 127.0.0.1 - - [26/Dec/2001:19:52:48 +0200] "GET /examples/ HTTP/1.1" 404 277 127.0.0.1 - - [26/Dec/2001:19:54:37 +0200] "GET /examples/jsp/snp/snoop.jsp HTTP/1.1" 404 294 127.0.0.1 - - [28/Dec/2001:09:54:37 +0200] "GET /puga/main.html HTTP/1.1" 404 282 | {string} - - [{date,dd/MMM/yyyy:HH:mm:ss Z} +0200] "{string}" {number} {number} |
Log3 | [Wed Dec 26 19:52:48 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/examples/ [Wed Dec 26 19:55:01 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/_vti_bin/owssvr.dll [Wed Dec 26 19:55:01 2001] [error] [client 127.0.0.1] File does not exist: c:/devapp/apache/apache/htdocs/msoffice/cltreq.asp | [{string} {date,EEE MMM dd HH:mm:ss yyyy] [{priority,debug;info;warn;error;fatal}] [{string} {string}] {string} |
Log4 | 28/02/03 20:23:16 ERR Critical error on section 34 on module 5 [Channel 9] 4.4.4.4 28/02/03 20:25:35 DBG information arrived to fusion zone, restoring states [Channel 39] 4.8.4.9 28/02/03 20:33:22 WRN port collision seeking another [Channel 19] 4.4.4.4 28/02/03 20:33:22 FLW DB connection open structure initiated [Channel 9] 4.23.12.5 | {date,dd/MM/yy HH:mm:ss} {priority,DBG;FLW;WRN;ERR} {string} [{string}] {string} |
Log5 | 5 d MBGN Talk to port 9 f MLPT1 Port open 0 x MCOM Com port open | {number}{tab}{string}{tab}{string}{tab}{string} |
Log6 | 5 d MBGN procId=123 Talk to port 9 f MLPT1 Port open 0 x MCOM procId=456 Com port open | {number}{tab}{string}{tab}{block,start,emptiness=true}procId={string}{block,end,emptiness=true}{string}{tab}{string} |