Defining a Log Collection Policy

For each log added to XpoLog, a Log Collection Policy must be used for defining how XpoLog server should collect the log information into its repository, and how long the logs should be archived. This can be a default Collection Policy or a previously defined Collection Policy. You can also define a new Collection Policy.

The Log Collection Policy criteria can be defined in the following tabs:

• General –  for setting the collection policy name and optionally providing a description for the usage of this customized/default collection policy

• Logs – for selecting the logs that are collected into the XpoLog repository using this policy 

• Collection Schedule – for defining the frequency of bringing data into XpoLog

• Storage – for defining where to store the log data, the maximum disk space that the policy can use for collecting data, how long to keep files in the storage to be available for searches before deleting them, and the email address of the administrator to notify when the maximum storage space is reached or if there is an error collecting data.

• Archive – for defining the behavior of the archived data

• Apptags – for security aspects it is optional to associate a collection policy with group apptags.

Storage: XpoLog (Indexed) Data is stored in a Binary, non readable format and can be read/ decrypted only by XpoLog. In case data is being tampered, XpoLog immediately alerts on the issue.
Archive: XpoLog archived Data is stored in compressed flat files. XpoLog runs a standard checksum (SHA-1/256/MD5) on the archive repository. In case data is being tampered, XpoLog immediately alerts on the issue.

To define a new Log Collection Policy:

1. Click on Manager pane, the Left Navigation Panel opens. select Data > Collection Policies. 
   The Collection Policies console opens. The available options are: Add New Collection Policy | Edit | Duplicate | Delete.

2. Click the New Collection Policy button.
   The Add new collection policy page opens.

3. In Name, type the name of the Collection Policy.

4. In Description, type a short description of the Collection Policy.

5. In Logs, select the Collection Policy members. Use the checkbox and select the Folders and logs that will be associated with the new collection policy.

6. In Storage, define the Collection Policy storage criteria. See Defining the Collection Policy Storage Criteria section below.

7. Define the archiving policy and security of the Collection Policy. See Defining Archiving below.

8. Click Save. 
   The Collection Policy is saved and can be used for adding logs and log directories.

Defining the Collection Policy Members

In the Members tab, you can select the logs that are to use the Collection Policy.
In the Collection Policies page, select the Members tab:

1. In the page that appears, select the check boxes of the  logs that are to use this Collection Policy.

Defining the Collection Policy Storage Criteria

In the Storage tab, you can define where to store the collected data and other storage criteria.
In the Collection Policies page, select the Storage tab:

1. Storage Repository -  ether browse to the location where to store the collected data or type/paste the full path. Then press Set Path to save the new storage location. 

   • Note that the default is the XpoLog internal data directory( ${xpolog.root.path}data ). Also, it is recommended to use a fast storage for this location. 

2. Retention Policy - in the Delete files older than, specify at what age files are to be removed from the repository. 

3. Data Encryption - XpoLog stores its internal data in a propriety, non readable, model. It is possible to enhance it by activating an additional encryption on the repository (used algorithm AES CBC 128). By selecting an encryption algorithm, as of the next policy's execution data will be stored encrypted. Storing encrypted data causes an overhead when writing/reading data which may result in a performance decrease of data collection/index/search.

Defining the Collection Schedule

You can define the frequency of collecting data from the log: Daily, Weekly, Monthly, or Never. Depending on the frequency selected, parameters appear for specifying the collection schedule.
In the Collection Policies page, select the Collection Schedule tab:

1. Set Frequency - select the frequency of bringing data into the system: Never, Daily, Weekly, or Monthly.
   Set the parameters that appear, as relevant.

Defining Archiving

Data stored in an archive is for long term storage of data, and unlike Storage Data, is unavailable to the user for searching and viewing. However, archive data can be restored and added to XpoLog as a local log (note that it is a manual process).

Checksum algorithms for ensuring data integrity can be activated; supported types are SHA1 and MD5. The Checksum algorithm checks that there has been no data tampering. Execution of the checksum algorithm results in a signature, which is saved in a file location, so that the current signature can be compared with previous signatures. The Checksum result file location can be customized to any location that XpoLog can access (default is XpoLog internal repository).

In the Collection Policies page, select the Archive Location tab:

1. Archive Repository: 

   1. Select the Enable Archiving checkbox to enable archiving collected logs; clear the checkbox to disable archiving.
      If activated, all logs which are part of the collection policy will be archived. The system runs archive every hour automatically.

   2. By default, when activating archiving XPLG stores on the selected location the data as raw data in zip files in the structure of the XPLG Folders and Logs (folder = directory, log = zip file(s)).
      Select "Mirror Folders and Logs structure on file system" (recommended) if you wish that when moving directories/logs in the XPLG folders and logs console, the system will also move them on the file system in the archive location to "mirror" the folders and logs structure on file system.

   3. Select Local (recommended) or AWS S3 Bucket. In Archive Path, browse to the location where to archive the collected data, it is not mandatory to use a fast storage for this location.
      Note: Local repository is recommended for large volumes of data, if you're not using Local then you'll be asked to select an AWS S3 Bucket account. To create a new account follow the guidelines to Configuring an Amazon Web Services (AWS) S3 Bucket Account in the article: Creating An Account.

2. Setting the archiving policy:

   1. Archive all data, by default this option is not selected. If enabled, the first archive execution will archive ALL data which is currently stored in XpoLog - note that this may take a very long time in case the repository is large. If not selected, then archive will take place from the first time of execution going forward.

   2. In Delete archive files older than, select the age that logs are automatically deleted from the archive.
      
      

3. Archiving Security - In Checksum algorithm, select a checksum algorithm for securing your archived logs: None, MD5, SHA1 or SHA-256. 
   
   

AppTags

XpoLog instances support setting security groups and associating them by AppTags. Each AppTag group may have different set permissions.

In the Collection Policies page, select the AppTags tab:

1. There's a drop down list of all available AppTags. Use the selection boxes to choose all the AppTags you wish to associate with the collection policy.