Creating an XML-Based Scanner Configuration File

You can create an XML file to build an environment for scanning many servers, and per server, scanning many directories. The path to this XML file is placed in the Add Logs Directories Task, for adding multiple directories to XpoLog, and automating addition of directories.

DirectoryScanner XML General Structure

The following is the XML code of DirectoryScanner.

<DirectoryScanner>
<ScannerNode name="NAME_OF_FOLDER_IN_XPOLOG">
<Account name="ACCOUNT NAME" ... /> <!-- OPTIONAL -->
<ScanDirectories>
<ScanDirectory scanPath="DIRECTORY_PATH_ON_SOURCE">
<ScanConfiguration>
<!-- OPTIONAL CONF PARAMETERS (ScanFileFilter, ScanConfApplications) -->
</ScanConfiguration>
</ScanDirectory>
</ScanDirectories>
</ScannerNode>
</DirectoryScanner>

The following table describes the general structure of DirectoryScanner.

Tag Path	Mandatory/Optional	Description
DirectoryScanner	Mandatory
DirectoryScanner/ScannerNode	Mandatory	The root folder that will be placed above its scanned directories.
DirectoryScanner/Account	Optional	Mandatory if XpoLog should connect to a remote server - Windows / UNIX (Creating an Account)
DirectoryScanner/ScanDirectories	Mandatory
DirectoryScanner/ScanDirectories/ScanDirectory	Mandatory	Contains the scanPath
DirectoryScanner/ScanDirectories/ScanDirectory/ScanConfiguration	Optional
DirectoryScanner/ScanDirectories/ScanDirectory/ScanConfiguration/ScanFileFilter	Optional
DirectoryScanner/ScanDirectories/ScanDirectory/ScanConfiguration/ScanConfApplications	Optional

XML Reference

ScannerNode Parameter

Parameter	Mandatory/Optional	Description	Values
name	Mandatory	The name of the root folder that will be placed above its scanned directories. Leave the name empty to create all sub-directories under the parent folder with their original name from the source server.	String

Example

<ScannerNode name="">

Account Parameters

Parameter	Mandatory/Optional	Description	Values
name	Mandatory	The account name	String
useEncrypt	Mandatory	Indicates whether or not account password will be encrypted	Boolean
isPublicKey	Mandatory (SSH Only)	If authentication is done by private key, should be FALSE. If user/password are used, it should be TRUE, and a path should be specified under privateKeyPath (see the following parameter).	Boolean
privateKeyPath	Optional (SSH Only)	The path to the key, if authentication is done by private key
isSystemAccount		Indicates whether or not account is a system account	Boolean
isScriptAPI		Indicates whether or not account is Script API	Boolean
isSSH		Indicates whether or not account is SSH	Boolean
isEditable		Indicates whether or not account can be edited in the XpoLog Address Book	Boolean
isCertificate		Indicates whether or not the account uses a certificate	Boolean
description	Optional	Description of the account
classKey	Mandatory	Windows: xpolog.eye.media.auth.win.WinAuthenticationAccount SSH: xpolog.eye.media.telsh.TelnetAccount
certificateID	Optional	The ID of the certificate, if account uses a certificate (see isCertificate).	String
UserName	Mandatory	The username that the account uses to connect	String
isDefault			Boolean
Port	Mandatory (SSH Only)	The port that will be used to establish the connection to the remote data source	Numeric
TYPE_SCP_SFTP	Optional (SSH Only)	Indicates if the SSH account will use SCP or SFTP (default) protocol	String
Password	Optional	The password that the account uses to connect. Optional only if SSH account uses Public/Private key	String
NetAddress	Mandatory	The IP/hostname of the remote data source used in the account	String

Note 1: If a remote data source is scanned then an account to that source should be specified (it can be verified after execution under XpoLog > Tools > Address Book). In case an account for a specified machine already exists, XpoLog will automatically use it.

Note 2: In case XpoLog is running on a Windows machine, it is recommended to configure a service account on the Windows services panel and then all Windows network logs can be scanned as local without specifying an account in the ScannerNode (path may be \\<server-name>\<drive-name>$\...).

Windows Account Example

<Account name="ACCOUNT NAME" useEncrypt="false" isSystemAccount="false" isEditable="true" description="" classKey="xpolog.eye.media.auth.win.WinAuthenticationAccount" Password="PASSWORD" user="USERNAME" isDefault="false"/>

SSH Account Example

<Account name="ACCOUNT NAME" useEncrypt="false" privateKeyPath="" isSystemAccount="false" isScriptApi="false" isSSH="true" isPublicKey="false" isEditable="true" isCertificate="false" description="" classKey="xpolog.eye.media.telsh.TelnetAccount" certificateId="" UserName="USERNAME" TYPE_SCP_SFTP="SFTP" Port="22" Password="PASSWORD" NetAddress="IP"/>

Note: you should not use an id parameter in the account line in the scanner XML. In case an account already exists in XpoLog then based on the NetAdress and Name it will be matched and re-used. In case the account does not exist it will be created during the scanner execution.

ScanDirectory Parameter

Parameter	Mandatory/Optional	Description	Values
scanPath	Mandatory	The full path to the directory that is to be scanned	Path

Examples

<ScanDirectory scanPath="C:\logs\"> (Windows Local)

<ScanDirectory scanPath="\\qaserver\C$\logs\"> (Windows Network)

<ScanDirectory scanPath="/opt/application/logs/"> (UNIX Local / Over SSH)

ScanConfiguration Parameters

Parameter	Mandatory/Optional	Description	Values
condenseLogsTree	Optional	A "true" value indicates that folders containing only one sub-folder and without logs, will be omitted from the Folders and Logs tree.	Boolean
directoriesToHide	Optional	A comma separated list of name expressions of folders that will not be added to the Folders and Logs tree;their sub-folders/logs will be added.
fileSuffixesToIgnore	Optional	Unite logs with different suffixes into one log type (advanced)
numberOfThreads	Optional	The number of threads to be used as part of the scanning operation	Integer
removeEmptyNodes	Optional	In case there are no matching files under one of the Folders and Logs members, remove it from the Folders and Logs tree.	Boolean
subdirsScanLevel	Optional	The number of sub-directories to scan from the given directory. Default is unlimited; any number can be specified.	Integer or "Unlimited"
scanMethod	Optional	0 = Use existing configuration (file names and content) and automatic matching. 1 = Use existing configuration (file names and content). 2 = Use existing configuration (file names only).	0, 1, or 2
namePatternLogic	Optional	0 = Capture each file separately (without name pattern). 1 = Unite files with a similar names (apply name pattern automatically). 2 = Unite files with a similar suffix (apply name pattern only at the end of the file name).	0, 1, or 2
filesToInclude	Optional	Define which files to include in the log scan: Using wildcard: .log,.txt - include all files whose name ends with .log or .txt Using a regular expression: regexp:\w+\.log - include all files whose name is constructed of word characters only and ends with .log, for example helloWorld.log
filesToExclude	Optional	Define which files to exclude in the log scan: Using wildcard: .zip,.gz - exclude all files whose name ends with .zip or .gz Using a regular expression: regexp:\w+\.tar\.gz - exclude all files whose name is constructed of word characters only and ends with .tar.gz, for example helloWorld.tar.gz
directoriesToExclude	Optional	Define which directories to exclude from the log scan: Using wildcard: Apps* - exclude all directories whose name starts with Apps Using a regular expression: regexp:\d\d\d\d-\d\d\-\d\d - exclude all directories whose name is a date, for example 2013-11-26
directoriesToInclude	Optional	Define which directories to include in the log scan: Using wildcard: Apps* - include all directories whose name starts with Apps Using a regular expression: regexp:\d\d\d\d-\d\d\-\d\d - include all directories whose name is a date, for example 2013-11-26
templatesToUse	Optional	The scan task will add only logs which were matched to one of the comma separated list of specified templates.	String
namePatternToApply	Optional	Automatically name the matched logs based on the given name pattern. Allowed identifiers are: [PARENT_FOLDER n] - the name of n-th parent folder of the log [CHILD_FOLDER n] - the name of n-th child folder of the root folder [APPLICATION] - the name of the log's application [SERVER] - the name of the log's server [LOG] - the current name of the log	String
timeZone	Optional	Set the specified time zone on all matched logs	String
onlineLogsApplication	Optional	Comma separated list of application name(s) that the online logs will be tagged to once created	String
assignedCollectionPolicy	Optional	The policy name of the collection policy that will be applied on the logs once created; If the parameter does not exist - the default policy will be automatically applied.	String
fileSuffixesToIgnore	Optional	Regular expression used to ignore part of the files names to define a name pattern	String
enableLogsIndex*	Optional	True – Online logs that are added under Folders and Logs will be indexed. False – Disable indexing.	Boolean
enableLogsAnalytics*	Optional	True – Online logs that are added under Folders and Logs will be analyzed by Analytics. False – Disable Analytics.	Boolean
addCollectors*	Optional	True – All the logs that are added by the scanner task will be collected; default policy will be applied, unless a specific policy is specified. False – Disables logs collection.	Boolean
enableCollectedLogsAnalytics*	Optional	True – enables Analytics analysis on collected logs. Online logs state will be taken from the collection policy; relevant only when addCollectors="true". False – Disables Analytics analysis on collected log.	Boolean
enableCollectedLogsIndex*	Optional	True – enables indexing of collected logs. The online logs state will be taken from the collection policy; relevant only when addCollectors="true". False – Disables indexing of collected logs.	Boolean
collectedLogsApplication*	Optional	Comma separated list of application name(s) that the collected logs will be tagged to once created; relevant only when addCollectors="true".	String

* Properties relevant only to versions 4.4 and below

Example

<ScanConfiguration condenseLogsTree="false" directoriesToInclude="log,home" numberOfThreads="3" assignedCollectionPolicy="live" filesToInclude="*.log">

ScanFileFilter Parameters

Parameter	Mandatory/Optional	Description	Values
timeInterval/timeIntervalUnit	Optional	The scan will add only log files with last updated time that is within the specified time interval per log type.	years, months, weeks, days, hours, mins
maxNumberOfFiles	Optional	The maximum number of log files that are added per log type.	Integer

Examples

<ScanFileFilter timeInterval="3" timeIntervalUnit="months"/>

<ScanFileFilter maxNumberOfFiles="2"/>

ScanConfApplications Parameters

Parameter	Mandatory/Optional	Description	Values
applicationNamePattern	Optional	The pattern that is used to extract the application name. An application will be created as part of the scan process.
applicationGroupNamePattern	Optional	The pattern that is used to extract the application group name. An application will be created as part of the scan process that all its sub-application are tagged to.

Example

<ScanConfApplications applicationNamePattern="/\w+-(\w+).*" applicationGroupNamePattern="/(\w+)-\w+.*"/>

Templates:

Please use the following examples as templates and modify accordingly (multiple directories per host can be defined by adding more <scanDirecotry> entries / multiple hosts can be defined by adding multiple <scannerNode> entries:

Example 1 (scanner_example_Windows_logs_account_on_xpolog_service): scanner_example_Windows_logs_account_on_xpolog_service.xml

Example 2 (scanner_example_Windows_logs_using_windows_network_account): scanner_example_Windows_logs_using_windows_network_account.xml

Example 3 (scanner_example_Linux_local_logs): scanner_example_Linux_local_logs.xml

Example 4 (scanner_example_Linux_remote_logs): scanner_example_Linux_remote_logs.xml