Use Fluent Bit to forward Windows logs to XPLG

Owned by Jonathan Wayne
Last updated: Sept 29, 2024
3 min read

 

This is a refined version of Send Data to XPLG using Fluent-Bit (agents)

 

 

What is Fluent Bit ?

Fluent Bit is a free, lightweight, open-source log dispatcher.
It gathers logs from various sources, processes them, and sends them out to different systems.
Fluent Bit is designed to be resource efficient, and easy to use.
Official documentation: fluentbit

 

How Does Fluent Bit Send Logs To XPLG?

We can install Fluent Bit on Windows or Linux machines.
FluentBit reads specific logs, processes them (applying filters if needed) and then sends the data out to XPLG.
The process of reading, processing, and sending logs is defined in a single conf file called fluent-bit.conf.

We will configure XPLG to listens and catch incoming data on a specific IP address , port and protocol like HTTP/S or Syslog (UDP/TCP).

 

How To Open A Port On XPLG (Listen For Incoming Data)?

The best practice is to send data via HTTPS. To set this up:

  1. Log in to the XPLG web interface.

  2. Navigate to the left panel: Data > Listen to Data.

  3. In the left section, select HTTP Listener, then click Add Account.

  4. Name the account, copy the provided URL (remember it),

  5. In a cluster environment, under Listening Node, you may choose the node that will receive the data (the IP to send to). You can assign to roles like PROC, Listener, or Master if needed.

About the listener: XPLG Data Listeners

How to open a Listener: HTTP/S

 

Where To Send The Data
We can now send data to XPLG using HTTP or HTTPS, via the specified IP, port, and token.
Please remember to note down this information, as it will be needed later for the Fluent Bit configuration file.

  1. HTTP: The port that XPLG is currently listening on. Default is 30303 (this can be changed or disabled).

  2. HTTPS: The port that XPLG is currently listening on. Default is 30443 (this can be changed).

  3. The IP address of the XPLG instance configured to listen.

  4. The auto-generated token. With the token, we can open multiple HTTP listeners on the same port, and it will help us identify the one that we want to reach. The token is specified in the listener settings and inside the access URL.

 

How to install Fluent Bit?

  • Install Fluent Bit on the source machine https://xpolog.atlassian.net/wiki/spaces/SK/pages/1581181726

Prerequisites Before Configuring Fluent Bit To Send Data:

  • Ensure there is network connectivity between the fluent-bit machine and XPLG.
    The connectivity should be from fluent-bit to XPLGs listener (chosen IP, Port, Protocol).


Windows Source Deployment - How to Configure Fluent Bit?

  1. Get the latest Fluent-bit zip file from https://docs.fluentbit.io/manual/installation/windows  (fluent-bit-*-win64.zip).

  2. Extract the files to the C:\ drive on the source Windows machine. Rename the extracted folder to 'fluent-bit', so you will have a directory path like C:\fluent-bit\.

  3. Downloaded a file from this link (Windows-fluent-bit.conf ) and rename it to fluent-bit.conf.

  4. Navigate to C:\fluent-bit and replace the existing fluent-bit.conf file with the one you downloaded overriding the existing file.

  5. Please edit the following lines to match your environment:

    1. This file takes as INPUT those logs:

      1. Windows Event Logs

      2. Custom Logs location (tail) - Configure the paths to the custom files to be shipped. (Modify the lines within square brackets in the file under the [INPUT] section.

    2. OUTPUT - the file contains an HTTPS output:

      1. Configure the XPLG Listener IP/Name, Port and Token of your XPLG defined listener. (Modify the lines within square brackets in the file under the [OUTPUT] section.

Create a Windows service:

  1. Open a Command Prompt as Administrator

  2. To register Fluent Bit as a Windows service, you need to execute the following command on Command Prompt. Please be careful that a single space is required after binpath=.

    1. Create service: 
      sc create fluent-bit binpath= "\fluent-bit\bin\fluent-bit.exe -c \fluent-bit\conf\fluent-bit.conf" start= auto

    2. Add description:
      sc description fluent-bit "fluent-bit log shipper to XPLG"

  3. Start the service and check data is arriving to XPLG.

 

Linux Source Deployment - How to Configure Fluent Bit?

  1. Get the latest Fluent-bit package file fromhttps://docs.fluentbit.io/manual/installation/linux .

  2. Install it (sudo yum install fluent-bit)

  3. You may use this Linux-fluent-bit.conf file and just rename it to fluent-bit.conf and modify it based on your needs (override existing /etc/fluent-bit/conf/fluent-bit.conf file):

    1. INPUT - the file contains 1 input:

      1. Custom Logs location (tail) - Configure the paths to the custom files to be shipped.

    2. OUTPUT - the file contains an HTTPS output:

      1. Configure the XPLG Listener IP/Name, Port and Token of your XPLG defined listener.

  4. Start the service and check data is arriving to XPLG - sudo systemctl start fluent-bit

 

Troubleshooting Tip
Run the command that starts Fluent Bit inside the command prompt and check for any output.
Windows: \fluent-bit\bin\fluent-bit.exe -c \fluent-bit\conf\fluent-bit.conf
Linux: Run systemctl status fluent-bit and check for the command being used to start the service
If the data is sent successfully, you will see an HTTP status=200.
If not, the output will display the reason for the failure.
Please pay close attention to the indentation when editing the fluent-bit.conf file.

Testing The Listener

Using the following URL format:

http://[IP]:[PORT]/logeye/api/logger.jsp?token=[TOKEN]&data={[JSON]}

or

https://[IP]:[PORT]/logeye/api/logger.jsp?token=[TOKEN]&data={[JSON]}

Please replace [IP], [PORT], [TOKEN], and [JSON] with your specific values.