Using Fluent Bit To Forward Logs to PortX
What is Fluent Bit?
Fluent Bit is a free, lightweight, open-source, multi-platform log forwarding agent. It gathers logs from various sources, processes them, and forwards them to various destinations. It is designed to be resource-efficient and easy to use. Additional information can be found in the fluentbit website.
How Does Fluent Bit Send Logs To PortX?
We can install Fluent Bit on Windows or Linux machines. Fluent Bit reads specific logs, processes them (applying filters if needed), and then sends the log data to PortX. Fluent Bit can send data over HTTPS (which is the best practice), HTTP, and UDP and TCP Syslog.
Configuring a PortX HTTPS Listener
To configure an HTTP or HTTPS Listener on PortX:
Log in to PortX.
On the PortX menu, select Data > Listen to Data.
Under Listener Accounts, select HTTP Listener, then click Add Account.
Name the account, copy the Listener URL (you will need it later).
In a cluster environment, under Listening Node, you may choose the node that will receive the data (the IP to send to). You can assign to roles like PROC, Listener, or Master if needed.
The listener will use HTTP or HTTPS according to the protocol PortX is configured to use. For more information see XPLG Data Listeners and HTTP/S.
Installing Fluent Bit
Prerequisites
Ensure there is network connectivity from the sending machine (where Fluent Bit will be installed) to the PortX machine (IP, Port, Protocol).
Deploying Fluent Bit on a Windows Machine
Get the latest Fluent-bit (
fluent-bit-<version>-win64.zip
) from https://docs.fluentbit.io/manual/installation/windows.Extract the files to the
C:\
drive on the source Windows machine. Rename the extracted folder tofluent-bit
, so you will have a directory path namedC:\fluent-bit\
.Download
Windows-fluent-bit.conf
- - copy it toC:\fluent-bit\conf\
and rename it tofluent-bit.conf
, overriding the existing file.
Edit the Configuration File
In the
[INPUT]
section, configure Windows Event Log and Custom Logs paths (tail
)
Replace the tag [ENTER_ABSOLUTE_PATH_TO_LOGS] with the actual absolute path(s).In the
[OUTPUT]
section, configure the Host, Port and URI to match the PortX listener URL you configured earlier.
Replace the tags [XPLG_LISTENER_IP/NAME] , [XPLG_LISTENER_PORT] , [XPLG_LISTENER_TOKEN] with the correct information.Make sure to change the
tls On
totls Off
for HTTP (non-secure) listener
Create a Windows Service
Open a Command Prompt as Administrator and run the following
sc
commands listed below
sc create fluent-bit binpath= "\fluent-bit\bin\fluent-bit.exe -c \fluent-bit\conf\fluent-bit.conf" start= auto
sc description fluent-bit "Fluent Bit: Log Forwarder to PortX"
sc failure fluent-bit reset= 86400 actions= restart/0/restart/0/restart/0/restart/0/restart/0/restart/900/restart/0/restart/0/restart/0/restart/0/restart/0/restart/900
Breakdown of Each Command:
Create the Main Fluent Bit Service (sc create)
sc create fluent-bit
→ Creates a new Windows service named fluent-bit.binpath= "\fluent-bit\bin\fluent-bit.exe -c \fluent-bit\conf\fluent-bit.conf"
Specifies the full path to the Fluent Bit executable and its configuration file (
fluent-bit.conf
).
start= auto
→ Sets the service to start automatically when the system boots.
Set the Service Description (sc description)
sc description fluent-bit
→ Updates the description of the fluent-bit service."Fluent Bit: Log Forwarder to PortX"
→ Sets a friendly description that appears in the Windows Services Manager (services.msc
).
Configure Service Recovery (Restart on Failure - sc failure)
sc failure fluent-bit
→ Modifies the failure recovery settings for the fluent-bit service.reset= 86400
→ Resets the failure count every 24 hours (86400 seconds).actions=
→ Defines what happens when the service fails:restart/0
→ Restart immediately on the 1st to 5th failures.restart/900
→ On the 6th failure, wait 900 seconds (15 minutes) before restarting.restart/0
→ Restart immediately on the 7th to 11th failures.restart/900
→ On the 12th failure, wait 900 seconds (15 minutes) before restarting.This cycle repeats indefinitely.
Start the service and check data is arriving to PortX.
Deploying Fluent Bit on a Linux Machine
Follow the instructions inhttps://docs.fluentbit.io/manual/installation/linux to install Fluent Bit on your distribution of Linux.
Download
Linux-fluent-bit.conf
- copy it to/etc/fluent-bit/conf/ and rename it to fluent-bit.conf
, overriding the existing file.
Edit the Configuration File
In the
[INPUT]
section, configure Custom Logs (tail
)
Replace the tag [ENTER_ABSOLUTE_PATH_TO_LOGS] with the actual absolute path(s).In the
[OUTPUT]
section, configure the Host, Port and URI to match the PortX listener URL you configured earlier.
Replace the tags [XPLG_LISTENER_IP/NAME] , [XPLG_LISTENER_PORT] , [XPLG_LISTENER_TOKEN] with the correct information.Make sure to change the
tls On
totls Off
for HTTP (non-secure) listener
Running and Testing the Service
Linux
Start the service using the following command and check data is arriving to PortX.
sudo systemctl start fluent-bit
Run the following command and check for the command being used to start the service. If the data is sent successfully, you will see an HTTP status=200
message. If not, the output will display the reason for the failure.
systemctl status fluent-bit
If Fluent Bit fails to start, check the configuration file for correct syntax and indentation. Refer to https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/format-schema for more information.
Note: that one common issues on Linux may be too many open files (default is limited to 1024 usually), if the Fluent-bit agent tails more than 1024 files it is recommended to configure a higher limitation to its service: edit the file /usr/lib/systemd/system/fluent-bit.service
and add under [Service]
the following line, and then save and restart the service for verification:
Windows
Run Fluent Bit in the command prompt and check for any output.