Kafka

Configure XPLG to collect Kafka topics:

1. Background - 
   In case Kafka listener does not appear in the listeners management console. Follow these steps:

   1. Stop XPLG service.

   2. Enter the file system with XPLG user and enter the path /INSTALL_DIR/plugins/templates/ then copy the directory Kafka and its content and place it at the path /SHARED_CONF/plugins/user/

   3. Restart XPLG.

2. Go to Manager > left panel Data > Listen to Data. The Listeners management console opens. Kafka listener should be visible.

3. Add Kafka account, for each account the following should be configured:

4.  

   1. Name: the name of the Listener account

   2. Description: the description of the Listener account

   3. Listening Node: the XPLG node that will perform the subscription (display if XPLG runs in cluster mode).
      IMPORTANT: if you plan to process data from an occupied topic that contains allot of data, and your Kafka cluster has multiple partitions to manage the data it is highly recommended to select multiple XPLG nodes (simply user the CTRL key to multi-select cluster nodes that will be consuming data as part of the Kafka listener).
      
      

   4. Host: the Kafka server(s):port(s) that the listener receives topics from (semicolon separated). For example: SRV1:PORT;SRV2:PORT  

   5. Topics: the list of topics Kafka Listener will subscribe to (comma separated). For example: TP1,TP2,TP3
      You can also use a regular expression that represents all topics that the Kafka listener will subscribe to
       

   6. Advanced Settings:
      
      General Information:
      Enabled: determine whether this account is enabled or not
      
      Security Details (optional if Kafka requires authentication):
      User Name: the user name to use to subscribe to the Kafka topic(s)
      Password: the user name's password
      Auth Mechanism: PLAIN
      Security Protocol: SASL_PLAINTEXT
      Security Module: org.apache.kafka.common.security.plain.PlainLoginModule
      
      Advanced Kafka Details:
      Consumer Group Id: use xplg
      
      Dynamic Log Creation Configuration:
      Parent Folder: the parent folder which all logs from this listener will be place under in XPLG Folders and Logs tree
      Collection Policy: the collection policy which will be associated to all logs from this listener (used mainly for storage location and data retention)
      AppTags: the AppTags which will be associated to all the logs from this listener (used mainly for data enrichment)
      Log Name Prefix: a prefix which will be added to any of the the logs from this listener (used to easily distinguish between multiple listener accounts logs)
      Split by Source Device: 

      1. Do not split - by default, XPLG will not split the incoming data. All data will be stored under a single log in XPLG.

      2. Create log by unique IP / host name - XPLG will split the incoming data based on the source that sends it to different logs, the log name structure will be "Log_Name_Prefix Source_IP/Name"

      3. Create log by IP mask - XPLG will split the incoming data based on matched source to the configured IP mask that sends it to different logs, the log name structure will be "Log_Name_Prefix IP_Mask"

      4. Create log with Regular expression - XPLG will split the incoming data to different logs based on a regular expression that will be applied on the Message field. The part in the regular expression that will be used to determine the split should be in (round parentheses).
         Records that the regular expression does not return a value will be directed to the global Kafka log.

      Listener Data:
      Listener Data Location: the location which data will be stored to, by default XPLG stores it in its data directory
      Indexing Interval: the frequency in which received Kafka topics are indexed

5. Save the account.

6. Data received to the Kafka listener account will be created under the configured parent folder and will be available for searching, reporting and alerting.

Note: multiple listeners accounts may be configured. However, listeners which run on the same machine must listen on a different network port.