Defining a Log Collection Policy

For each log added to XpoLog, a Log Collection Policy must be used for defining how XpoLog server should collect the log information into its repository, and how long the logs should be archived. This can be a default Collection Policy or a previously defined Collection Policy. You can also define a new Collection Policy.

The Log Collection Policy criteria can be defined in the following tabs:

• Members – for selecting the logs that are collected into the XpoLog repository using this policy 
• Storage – for defining where to store the log data, the maximum disk space that the policy can use for collecting data, how long to keep files in the archive directory before deleting them, and the email address of the administrator to notify when the maximum storage space is reached or if there is an error collecting data.
• Collection Schedule – for defining the frequency of bringing data into XpoLog
• Archiving – for defining the location of the archived data

Storage: XpoLog online (Indexed) Data is stored in a Binary, non readable format and cannot be read or decrypted only by XpoLog. In case data is being tampered, XpoLog immediately alerts on the issue.
Archive: XpoLog archived Data is stored in compressed flat files. XpoLog runs a standard checksum (SHA-1/MD5) on the archive repository. In case data is being tampered, XpoLog immediately alerts on the issue.

To define a new Log Collection Policy:

1. In the XpoLog Manager main menu, select Administration > Collection Policies. 
   The Collection Policies page opens.
2. Click the New Collection Policy button.
   The Add new collection policy page opens.
3. In Name, type the name of the Collection Policy.
4. In Description, type a short description of the Collection Policy.
5. Define the Collection Policy members. See Defining the Collection Policy Members section below.
6. Define the Collection Policy storage criteria. See Defining the Collection Policy Storage Criteria section below.
7. Define the archiving policy and security of the Collection Policy. See Defining Archiving below.
8. Click Save. 
   The Collection Policy is saved and can be used for adding logs and log directories.

Defining the Collection Policy Members

In the Members tab, you can select the logs that are to use the Collection Policy.

To define the Collection Policy members:

1. In the Collection Policies page, select the Members tab.
2. In the page that appears, select the checkboxes of the  logs that are to use this Collection Policy.

Defining the Collection Policy Storage Criteria

In the Storage tab, you can define where to store the collected data and other storage criteria.

To define the storage criteria: 

1. In the Collection Policies page, select the Storage tab.

2. In the page that appears, in Storage Path, browse to the location where to store the collected data. The default is the XpoLog internal data library.

3. In Max disk space, type the maximum disk space that the policy is allowed to utilize for collecting data, selecting the relevant units (MB or GB).
   Note: When the amount of data reaches the maximum, XpoLog stops collecting data. It does not delete any data.

4. In Delete files older than, specify at what age files are to be removed from the archive repository. 

5. In Send email notification on collection errors to, type the email address for notifying of collection errors or maximum disk space utilization. 

Defining the Collection Schedule

You can define the frequency of collecting data from the log: Daily, Weekly, Monthly, or Never. Depending on the frequency selected, parameters appear for specifying the collection schedule.

To define the collection schedule:

1. In the Collection Policies page, select the Collection Schedule tab.
2. In the page that appears, in Set Frequency, select the frequency of bringing data into the system: Never, Daily, Weekly, or Monthly.
3. Set the parameters that appear, as relevant.
4. Assigned Instance - In case XpoLog is running in a clustered environment, with more than one processor node, this option allows to determine which of the processors will be responsible for the collection policy.
5. Live Mode Collection Frequency - activating Live Mode in the search console immediately executes collection from all relevant sources in order to fetch matching log records, in near real time, to the console.
   The frequency of collection while Live Mode is active is determined here. By default, the collection will run every 10 seconds as long as Live Mode is active in search.
   Pay attention: the frequency set here determines the collection frequency of logs which are part of the collection policy during Live Mode, and the load that may be seen on the sources while active. It is recommended that users will be guided to activate Live Mode on specific logs/folders/servers, and not on the entire environment, to avoid unnecessary load on multiple sources.

Defining Archiving

Data stored in an archive is for longterm storage of data, and unlike Storage data, is unavailable to the user for searching and viewing. However, archive data can be restored and added to XpoLog as a local log.

Checksum algorithms for ensuring data integrity can be activated; supported types are SHA1 and MD5. The Checksum algorithm checks that there has been no data tampering. Execution of the checksum algorithm results in a signature, which is saved in a file location, so that the current signature can be compared with previous signatures. The Checksum result file location can be customized to any location that XpoLog can access (default is XpoLog internal repository.)

To define archiving of the Collection Policy:

1.  In the Collection Policies page, select the Archiving tab.
2. In the page that appears, in Archive Path, browse to the location where to archive the collected data.
3. Select the Enable Archiving checkbox to enable archiving collected logs; clear the checkbox to disable archiving.
4. In Delete archive files older than, select the age that logs are automatically deleted from the archive.
5. In Checksum algorithm, select a checksum algorithm for securing your archived logs: None, MD5, or SHA1.