What is Fluent Bit?

Fluent Bit is a free, lightweight, open-source, multi-platform log forwarding agent. It gathers logs from various sources, processes them, and forwards them to various destinations. It is designed to be resource-efficient and easy to use. Additional information can be found in the fluentbit website.

How Does Fluent Bit Send Logs To PortX?

We can install Fluent Bit on Windows or Linux machines. Fluent Bit reads specific logs, processes them (applying filters if needed), and then sends the log data to PortX. Fluent Bit can send data over HTTPS (which is the best practice), HTTP, and UDP and TCP Syslog.

Configuring a PortX HTTPS Listener

To configure an HTTP or HTTPS Listener on PortX:

1. Log in to PortX.

2. On the PortX menu, select Data > Listen to Data.

3. Under Listener Accounts, select HTTP Listener, then click Add Account.

4. Name the account, copy the Listener URL (you will need it later).

5. In a cluster environment, under Listening Node, you may choose the node that will receive the data (the IP to send to). You can assign to roles like PROC, Listener, or Master if needed.

The listener will use HTTP or HTTPS according to the protocol PortX is configured to use. For more information see XPLG Data Listeners and HTTP/S.

Installing Fluent Bit

Prerequisites

• Ensure there is network connectivity from the sending machine (where Fluent Bit will be installed) to the PortX machine (IP, Port, Protocol).

Deploying Fluent Bit on a Windows Machine

1. Get the latest Fluent-bit (fluent-bit-<version>-win64.zip) from https://docs.fluentbit.io/manual/installation/windows.

2. Extract the files to the C:\ drive on the source Windows machine. Rename the extracted folder to fluent-bit, so you will have a directory path named C:\fluent-bit\.

3. Download Windows-fluent-bit.conf - - copy it to C:\fluent-bit\conf\ and rename it to fluent-bit.conf, overriding the existing file.

Edit the Configuration File

1. In the [INPUT] section, configure Windows Event Log and Custom Logs paths (tail)
   Replace the tag [ENTER_ABSOLUTE_PATH_TO_LOGS] with the actual absolute path(s).

2. In the [OUTPUT] section, configure the Host, Port and URI to match the PortX listener URL you configured earlier.
   Replace the tags [XPLG_LISTENER_IP/NAME] , [XPLG_LISTENER_PORT] , [XPLG_LISTENER_TOKEN] with the correct information.

3. Make sure to change the tls On to tls Off for HTTP (non-secure) listener

Create a Windows Service

1. Open a Command Prompt as Administrator and run the followingsc commands listed below

Breakdown of Each Command:

• sc create fluent-bit → Creates a new Windows service named fluent-bit.

• binpath= "\fluent-bit\bin\fluent-bit.exe -c \fluent-bit\conf\fluent-bit.conf"

  • Specifies the full path to the Fluent Bit executable and its configuration file (fluent-bit.conf).

• start= auto → Sets the service to start automatically when the system boots.

• sc description fluent-bit → Updates the description of the fluent-bit service.

• "Fluent Bit: Log Forwarder to PortX" → Sets a friendly description that appears in the Windows Services Manager (services.msc).

• sc failure fluent-bit → Modifies the failure recovery settings for the fluent-bit service.

• reset= 86400 → Resets the failure count every 24 hours (86400 seconds).

• actions= → Defines what happens when the service fails:

  • restart/0 → Restart immediately on the 1st to 5th failures.

  • restart/900 → On the 6th failure, wait 900 seconds (15 minutes) before restarting.

  • restart/0 → Restart immediately on the 7th to 11th failures.

  • restart/900 → On the 12th failure, wait 900 seconds (15 minutes) before restarting.

  • This cycle repeats indefinitely.

3. Start the service and check data is arriving to PortX.

Deploying Fluent Bit on a Linux Machine

• Follow the instructions inhttps://docs.fluentbit.io/manual/installation/linux to install Fluent Bit on your distribution of Linux.

• Download Linux-fluent-bit.conf - copy it to /etc/fluent-bit/conf/ and rename it to fluent-bit.conf, overriding the existing file.

Edit the Configuration File

1. In the [INPUT] section, configure Custom Logs (tail)
   Replace the tag [ENTER_ABSOLUTE_PATH_TO_LOGS] with the actual absolute path(s).

2. In the [OUTPUT] section, configure the Host, Port and URI to match the PortX listener URL you configured earlier.
   Replace the tags [XPLG_LISTENER_IP/NAME] , [XPLG_LISTENER_PORT] , [XPLG_LISTENER_TOKEN] with the correct information.

3. Make sure to change the tls On to tls Off for HTTP (non-secure) listener

Running and Testing the Service

Linux

1. Start the service using the following command and check data is arriving to PortX.

Run the following command and check for the command being used to start the service. If the data is sent successfully, you will see an HTTP status=200 message. If not, the output will display the reason for the failure.

If Fluent Bit fails to start, check the configuration file for correct syntax and indentation. Refer to https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/format-schema for more information.
Note: that one common issues on Linux may be too many open files (default is limited to 1024 usually), if the Fluent-bit agent tails more than 1024 files it is recommended to configure a higher limitation to its service: edit the file /usr/lib/systemd/system/fluent-bit.service  and add under [Service] the following line, and then save and restart the service for verification:

Windows

Run Fluent Bit in the command prompt and check for any output.