Defining a Log Collection Policy

Owned by XPLG Team
Last updated: Nov 18, 2024 by XPLG Team
5 min read

For each log added to XPLG, a Collection Policy must be used for defining how to collect and store the log in the system repository, for how long to keep in searchable (indexed) and/or archived. This can be a default Collection Policy or any other custom defined Collection Policy.

The Log Collection Policy criteria's:

  • General –  setting the collection policy name and optionally providing a description for the usage of this customized/default collection policy

  • Logs – selecting the logs that are associated to the policy 

  • Collection Schedule – defining the frequency of bringing data into XPLG

  • Storage – defining the disk location of the collection policy where data will be stored and the retention policy.
    The storage is the “hot” repository of XPLG where data is stored indexed and searchable in XPLG Search console.

  • Archive – for defining the behavior of the archived data.
    The archive is the “cold” repository of XPLG where data is stored not indexed, usually on slower disks for long-term retention.

  • Apptags – The related tags that the policy will be part of.

Storage: XPLG Data is stored in a Segmented-Binary, non readable format and can only be read/decrypted by XPLG authenticated user. Data cannot be changed once it entered the processing engine and is stored as-is from entry to clean up. Regardless of any parsing/enrichment rules which run on top on the original data, the raw data itself remains untouched and may be exported as raw data from the system. In case data is being tampered, XPLG immediately alerts on the issue.
Archive: XPLG archived Data is stored in compressed flat files. XPLG runs a standard checksum (SHA-1/256/MD5) on the archive repository. In case data is being tampered, XPLG immediately alerts on the issue.

 

To define a new Log Collection Policy:

  1. Click on Manager pane, the Left Navigation Panel opens. select Data > Collection Policies. 
    The Collection Policies console opens. The available options are: Add New Collection Policy | Edit | Duplicate | Delete.

  2. Click the New Collection Policy button.
    The Add new collection policy page opens.

  3. In Name, type the name of the Collection Policy.

  4. In Description, type a short description of the Collection Policy.

  5. In Logs, select the Collection Policy members. Use the checkbox and select the Folders and logs that will be associated with the new collection policy.

  6. In Storage, define the Collection Policy storage criteria. See Defining the Collection Policy Storage Criteria section below.

  7. Define the archiving policy and security of the Collection Policy. See Defining Archiving below.

  8. Click Save. 
    The Collection Policy is saved and can be used for adding logs and log directories.

Defining the Collection Policy Members

In the Members tab, you can select the logs that are to use the Collection Policy.
In the Collection Policies page, select the Members tab:

  1. In the page that appears, select the check boxes of the  logs that are to use this Collection Policy.

Defining the Collection Policy Storage Criteria

In the Storage tab, you can define where to store the collected data and other storage criteria.
In the Collection Policies page, select the Storage tab:

  1. Storage Repository -  ether browse to the location where to store the collected data or type/paste the full path. Then press Set Path to save the new storage location. 

    • Note that the default is the XPLG internal data directory (${xpolog.root.path}data). Also, it is recommended to use a fast storage for this location as this is where most read/write operations are performed. 

  2. Retention Policy - in the Delete files older than, specify at what age files are to be removed from the repository. 

  3. Data Encryption - XPLG stores its internal data in a propriety, segmented binary non readable model and cannot be read on file system or by any other tool.

Defining the Collection Schedule

You can define the frequency of collecting data from the log: Daily, Weekly, Monthly, or Never. Depending on the frequency selected, parameters appear for specifying the collection schedule.
In the Collection Policies page, select the Collection Schedule tab:

  1. Set Frequency - select the frequency of bringing data into the system: Never, Daily, Weekly, or Monthly.
    Set the parameters that appear, as relevant.

Defining Archiving

Data stored in an archive is for long term storage of data, and unlike Storage Data, is unavailable to the user for searching and viewing. However, archive data can be restored and added to XPLG as a local log (note that it is a manual process).

Checksum algorithms for ensuring data integrity can be activated; supported types are SHA1 and MD5. The Checksum algorithm checks that there has been no data tampering. Execution of the checksum algorithm results in a signature, which is saved in a file location, so that the current signature can be compared with previous signatures. The Checksum result file location can be customized to any location that XPLG can access (default is XPLG internal repository).

In the Collection Policies page, select the Archive Location tab:

  1. Archive Repository: 

    1. Select the Enable Archiving checkbox to enable archiving collected logs; clear the checkbox to disable archiving.
      If activated, all logs which are part of the collection policy will be archived. The system runs archive every hour automatically.

    2. By default, when activating archiving XPLG stores on the selected location the data as raw data in zip files in the structure of the XPLG Folders and Logs (folder = directory, log = zip file(s)).
      Select "Mirror Folders and Logs structure on file system" (recommended) if you wish that when moving directories/logs in the XPLG folders and logs console, the system will also move them on the file system in the archive location to "mirror" the folders and logs structure on file system.

    3. Select Local (recommended) or AWS S3 Bucket. In Archive Path, browse to the location where to archive the collected data, it is not mandatory to use a fast storage for this location.
      Note: Local repository is recommended for large volumes of data, if you're not using Local then you'll be asked to select an AWS S3 Bucket account. To create a new account follow the guidelines to Configuring an Amazon Web Services (AWS) S3 Bucket Account in the article: Creating An Account.

  2. Setting the archiving policy:

    1. Archive all data, by default this option is not selected. If enabled, the first archive execution will archive ALL data which is currently stored in XPLG - note that this may take a very long time in case the repository is large. If not selected, then archive will take place from the first time of execution going forward.

    2. In Delete archive files older than, select the age that logs are automatically deleted from the archive.

  3. Archiving Security - In Checksum algorithm, select a checksum algorithm for securing your archived logs: None, MD5, SHA1 or SHA-256. 

AppTags

XPLG instances support setting security groups and associating them by AppTags. Each AppTag group may have different set permissions.

In the Collection Policies page, select the AppTags tab:

  1. There's a drop down list of all available AppTags. Use the selection boxes to choose all the AppTags you wish to associate with the collection policy.