Creating an XML-Based Scanner Configuration File

You can create an XML file to build an environment for scanning many servers, and per server, scanning many directories. The path to this XML file is placed in the Add Logs Directories Task, for adding multiple directories to XpoLog, and automating addition of directories.

DirectoryScanner XML General Structure

The following is the XML code of DirectoryScanner.

<DirectoryScanner>
<ScannerNode name="NAME_OF_FOLDER_IN_XPOLOG">
<Account name="ACCOUNT NAME" ... /> <!-- OPTIONAL -->
<ScanDirectories>
<ScanDirectory scanPath="DIRECTORY_PATH_ON_SOURCE">
<ScanConfiguration>
<!-- OPTIONAL CONF PARAMETERS (ScanFileFilter, ScanConfApplications) -->
</ScanConfiguration>
</ScanDirectory>
</ScanDirectories>
</ScannerNode>
</DirectoryScanner>

  The following table describes the general structure of DirectoryScanner. 

Tag Path

Mandatory/Optional

Description

Tag Path

Mandatory/Optional

Description

DirectoryScanner

Mandatory

 

DirectoryScanner/ScannerNode

Mandatory

The root folder that will be placed above its scanned directories.

DirectoryScanner/Account

Optional

Mandatory if XpoLog should connect to a remote server - Windows / UNIX (Creating an Account)

DirectoryScanner/ScanDirectories

Mandatory

 

DirectoryScanner/ScanDirectories/ScanDirectory

Mandatory

Contains the scanPath

DirectoryScanner/ScanDirectories/ScanDirectory/ScanConfiguration

Optional

 

DirectoryScanner/ScanDirectories/ScanDirectory/ScanConfiguration/ScanFileFilter

Optional

 

DirectoryScanner/ScanDirectories/ScanDirectory/ScanConfiguration/ScanConfApplications

Optional

 

 XML Reference

ScannerNode Parameter

Parameter

Mandatory/Optional

Description

Values

Parameter

Mandatory/Optional

Description

Values

 name

 Mandatory

The name of the root folder that will be placed above its scanned directories.
Leave the name empty to create all sub-directories under the parent folder with their original name from the source server.

 String

Example

<ScannerNode name="">

Account Parameters

Parameter

Mandatory/Optional

Description

Values

Parameter

Mandatory/Optional

Description

Values

 name

Mandatory

The account name

 String

 useEncrypt

Mandatory

Indicates whether or not account password will be encrypted

 Boolean

 isPublicKey

Mandatory (SSH Only)

If authentication is done by private key, should be FALSE. 
If user/password are used, it should be TRUE, and a path should be specified under    
privateKeyPath (see the following parameter).

 Boolean 

 privateKeyPath

Optional (SSH Only)

The path to the key, if authentication is done by private key

 

 isSystemAccount

 

Indicates whether or not account is a system account

 Boolean

 isScriptAPI

 

Indicates whether or not account is Script API

 Boolean

 isSSH

 

Indicates whether or not account is SSH

 Boolean

 isEditable

 

Indicates whether or not account can be edited in the XpoLog Address Book

 Boolean

 isCertificate

 

Indicates whether or not the account uses a certificate

 Boolean

 description

Optional

Description of the account

 

 classKey

Mandatory

Windows: xpolog.eye.media.auth.win.WinAuthenticationAccount
SSH: xpolog.eye.media.telsh.TelnetAccount

 

 certificateID

Optional

The ID of the certificate, if account uses a certificate (see isCertificate).

 String

 UserName

Mandatory

The username that the account uses to connect 

 String

 isDefault

 

 

 Boolean

 Port

Mandatory (SSH Only)

The port that will be used to establish the connection to the remote data source

Numeric

 TYPE_SCP_SFTP

Optional (SSH Only)

Indicates if the SSH account will use SCP or SFTP (default) protocol

String

 Password

Optional

The password that the account uses to connect. Optional only if SSH account uses Public/Private key

String

 NetAddress

Mandatory

The IP/hostname of the remote data source used in the account

String

Note 1: If a remote data source is scanned then an account to that source should be specified (it can be verified after execution under XpoLog > Tools > Address Book). In case an account for a specified machine already exists, XpoLog will automatically use it.

Note 2: In case XpoLog is running on a Windows machine, it is recommended to configure a service account on the Windows services panel and then all Windows network logs can be scanned as local without specifying an account in the ScannerNode (path may be \\<server-name>\<drive-name>$\...).

Windows Account Example

<Account name="ACCOUNT NAME" useEncrypt="false" isSystemAccount="false" isEditable="true" description="" classKey="xpolog.eye.media.auth.win.WinAuthenticationAccount" Password="PASSWORD" user="USERNAME" isDefault="false"/>

SSH Account Example

<Account name="ACCOUNT NAME" useEncrypt="false" privateKeyPath="" isSystemAccount="false" isScriptApi="false" isSSH="true" isPublicKey="false" isEditable="true" isCertificate="false" description="" classKey="xpolog.eye.media.telsh.TelnetAccount" certificateId="" UserName="USERNAME" TYPE_SCP_SFTP="SFTP" Port="22" Password="PASSWORD" NetAddress="IP"/>

 

Note: you should not use an id parameter in the account line in the scanner XML. In case an account already exists in XpoLog then based on the NetAdress and Name it will be matched and re-used. In case the account does not exist it will be created during the scanner execution.

ScanDirectory Parameter

Parameter

Mandatory/Optional

Description

Values

Parameter

Mandatory/Optional

Description

Values

 scanPath

 Mandatory

 The full path to the directory that is to be scanned

 Path

Examples

<ScanDirectory scanPath="C:\logs\"> (Windows Local)

<ScanDirectory scanPath="\\qaserver\C$\logs\"> (Windows Network)

<ScanDirectory scanPath="/opt/application/logs/"> (UNIX Local / Over SSH)

 

ScanConfiguration Parameters

Parameter

Mandatory/Optional

Description

Values

Parameter

Mandatory/Optional

Description

Values

 condenseLogsTree

 Optional

A "true" value indicates that folders containing only one sub-folder and without logs, will be omitted from the Folders and Logs tree.

 Boolean

 directoriesToHide

 Optional

A comma separated list of name expressions of folders that will not be added to the Folders and Logs tree;their sub-folders/logs will be added.

 

 fileSuffixesToIgnore

 Optional

Unite logs with different suffixes into one log type (advanced)

  

 numberOfThreads

 Optional

The number of threads to be used as part of the scanning operation

 Integer

 removeEmptyNodes

 Optional

In case there are no matching files under one of the Folders and Logs members, remove it from the Folders and Logs tree.

 Boolean

 subdirsScanLevel 

 Optional

The number of sub-directories to scan from the given directory. Default is unlimited; any number can be specified.

 Integer or "Unlimited"

 scanMethod

 Optional

0 = Use existing configuration (file names and content) and automatic matching.
1 = Use existing configuration (file names and content).
2 = Use existing configuration (file names only).

 0, 1, or 2

 namePatternLogic

 Optional

0 = Capture each file separately (without name pattern).
1 = Unite files with a similar names (apply name pattern automatically).
2 = Unite files with a similar suffix (apply name pattern only at the end of the file name).

 0, 1, or 2

 filesToInclude

 Optional

Define which files to include in the log scan:
Using wildcard: *.log,*.txt - include all files whose name ends with .log or .txt
Using a regular expression: regexp:\w+\.log - include all files whose name is constructed of word characters only and ends with .log, for example helloWorld.log

 

 filesToExclude

 Optional

Define which files to exclude in the log scan:
Using wildcard: *.zip,*.gz - exclude all files whose name ends with .zip or .gz
Using a regular expression: regexp:\w+\.tar\.gz - exclude all files whose name is constructed of word characters only and ends with .tar.gz, for example helloWorld.tar.gz

 

 directoriesToExclude

 Optional

Define which directories to exclude from the log scan:
Using wildcard: Apps* - exclude all directories whose name starts with Apps
Using a regular expression: regexp:\d\d\d\d-\d\d\-\d\d - exclude all directories whose name is a date, for example 2013-11-26

 

 directoriesToInclude

 Optional

Define which directories to include in the log scan:
Using wildcard: Apps* - include all directories whose name starts with Apps
Using a regular expression: regexp:\d\d\d\d-\d\d\-\d\d - include all directories whose name is a date, for example 2013-11-26

 

templatesToUse

Optional

The scan task will add only logs which were matched to one of the comma separated list of specified templates.

String

namePatternToApply

Optional

Automatically name the matched logs based on the given name pattern. Allowed identifiers are:

  • [PARENT_FOLDER n] - the name of n-th parent folder of the log

  • [CHILD_FOLDER n] - the name of n-th child folder of the root folder

  • [APPLICATION] - the name of the log's application

  • [SERVER] - the name of the log's server

  • [LOG] - the current name of the log

String

timeZone

Optional

Set the specified time zone on all matched logs

String

onlineLogsApplication

Optional

Comma separated list of application name(s) that the online logs will be tagged to once created

String

 assignedCollectionPolicy

Optional

The policy name of the collection policy that will be applied on the logs once created;
If the parameter does not exist - the default policy will be automatically applied.

String

fileSuffixesToIgnore

 Optional

Regular expression used to ignore part of the files names to define a name pattern

 String

 enableLogsIndex*

 Optional 

True – Online logs that are added under Folders and Logs will be indexed.
False – Disable indexing.

 Boolean

 enableLogsAnalytics*

 Optional

True – Online logs that are added under Folders and Logs will be analyzed by Analytics.
False – Disable Analytics.

Boolean

 addCollectors*

 Optional

True – All the logs that are added by the scanner task will be collected; default policy will be applied, unless a specific policy is specified.
False – Disables logs collection.

 Boolean

enableCollectedLogsAnalytics*

 Optional 

True – enables Analytics analysis on collected logs. Online logs state will be taken from the collection policy;
relevant only when addCollectors="true".
False – Disables Analytics analysis on collected log. 

 Boolean

 enableCollectedLogsIndex*

 Optional

True – enables indexing of collected logs. The online logs state will be taken from the collection policy;
relevant only when addCollectors="true".
False – Disables indexing of collected logs. 

 Boolean

 collectedLogsApplication*

Optional

Comma separated list of application name(s) that the collected logs will be tagged to once created;
relevant only when addCollectors="true".

String

* Properties relevant only to versions 4.4 and below

Example

<ScanConfiguration condenseLogsTree="false" directoriesToInclude="log,home" numberOfThreads="3" assignedCollectionPolicy="live" filesToInclude="*.log">

ScanFileFilter Parameters

Parameter

Mandatory/Optional

Description

Values

Parameter

Mandatory/Optional

Description

Values

 timeInterval/timeIntervalUnit

 Optional

The scan will add only log files with last updated time that is within the specified time interval per log type.

years, months, weeks, days, hours, mins

 maxNumberOfFiles

 Optional

The maximum number of log files that are added per log type.

 Integer

Examples

<ScanFileFilter timeInterval="3" timeIntervalUnit="months"/>

<ScanFileFilter maxNumberOfFiles="2"/>

ScanConfApplications Parameters

Parameter

Mandatory/Optional

Description

Values

Parameter

Mandatory/Optional

Description

Values

 applicationNamePattern

 Optional

 The pattern that is used to extract the application name. An application will be created as part of the scan process.

 

 applicationGroupNamePattern

 Optional

The pattern that is used to extract the application group name. An application will be created as part of the scan process that all its sub-application are tagged to.

 

Example

<ScanConfApplications applicationNamePattern="/\w+-(\w+).*" applicationGroupNamePattern="/(\w+)-\w+.*"/>


Templates:

Please use the following examples as templates and modify accordingly (multiple directories per host can be defined by adding more <scanDirecotry> entries / multiple hosts can be defined by adding multiple <scannerNode> entries:

Example 1 (scanner_example_Windows_logs_account_on_xpolog_service): scanner_example_Windows_logs_account_on_xpolog_service.xml

Example 2 (scanner_example_Windows_logs_using_windows_network_account): scanner_example_Windows_logs_using_windows_network_account.xml

Example 3 (scanner_example_Linux_local_logs): scanner_example_Linux_local_logs.xml

Example 4 (scanner_example_Linux_remote_logs): scanner_example_Linux_remote_logs.xml