Send Data to XPLG using Fluent-Bit (agents)

Owned by XPLG Team
Last updated: Jun 05, 2024
2 min read

Background

XPLG contains built-in listeners (HTTP/S, Syslog UDP/TCP) and data can be easily shipped to XPLG cluster for processing and monitoring.

The most recommended agent to be used is Fluent Bit. Fluent Bit is an open source and multi-platform Log Forwarder which allows you to collect data/logs from different sources and send them to multiple destinations. Fluent Bit is one of the fastest, lightweight available in the market.

More information can be reviewed at - https://fluentbit.io/

Fluent bit supports HTTP/S output which sends the data/logs directly to XPLG listeners in JSON format.



XPLG Deployment

On the XPLG side, create a HTTP/S listener on the XPLG cluster, and make sure all required sources have access to XPLG cluster (XPLG listener IP/Port).

The listener endpoint URL should be copied from the listener definition and used in the fluent-bit.conf configuration file as the output endpoint (the target of shipped logs).



Windows Source Deployment

  1. Get the latest Fluent-bit zip file from https://docs.fluentbit.io/manual/installation/windows (fluent-bit-*-win64.zip).

  2. Extract it under C:\ drive on the source Windows machine. Rename the extracted directory to 'fluent-bit' (you should have a C:\fluent-bit\ directory)

  3. You may use this Windows-fluent-bit.conf file and just rename it to fluent-bit.conf and modify it based on your needs (override existing C:\fluent-bit\conf\fluent-bit.conf file):

    1. INPUT - the file contains 2 inputs:

      1. Windows Event Logs

      2. Custom Logs location (tail) - Configure the paths to the custom files to be shipped.

    2. OUTPUT - the file contains an HTTPS output:

      1. Configure the XPLG Listener IP/Name, Port and Token of your XPLG defined listener.

  4. Create a Windows service:

    1. Open a Command Prompt as Administrator to define the Windows service.

    2. To register Fluent Bit as a Windows service, you need to execute the following command on Command Prompt. Please be careful that a single space is required after binpath=.

      1. Create service: 
        sc create fluent-bit binpath= "\fluent-bit\bin\fluent-bit.exe -c \fluent-bit\conf\fluent-bit.conf" start= auto

      2. Add description:
        sc description fluent-bit "fluent-bit log shipper to XPLG"

    3. Start the service and check data is arriving to XPLG.





Linux Source Deployment

  1. Get the latest Fluent-bit package file fromhttps://docs.fluentbit.io/manual/installation/linux.

  2. Install it (sudo yum install fluent-bit)

  3. You may use this Linux-fluent-bit.conf file and just rename it to fluent-bit.conf and modify it based on your needs (override existing /etc/fluent-bit/conf/fluent-bit.conf file):

    1. INPUT - the file contains 1 input:

      1. Custom Logs location (tail) - Configure the paths to the custom files to be shipped.

    2. OUTPUT - the file contains an HTTPS output:

      1. Configure the XPLG Listener IP/Name, Port and Token of your XPLG defined listener.

  4. Start the service and check data is arriving to XPLG - sudo systemctl start fluent-bit