Send data from Kubernetes to XPLG using Fluent-Bit
DaemonSet vs Sidecar
Overview
Fluent Bit is a lightweight and resource-efficient log processor designed for high-performance environments such as containers and microservices. It's written in C, making it significantly faster and lighter than alternatives like Fluentd, and is widely adopted for log collection, transformation, and forwarding in Kubernetes clusters.
https://docs.fluentbit.io/manual
Fluent Bit Deployment Strategies in Kubernetes
Fluent Bit supports two primary deployment models in Kubernetes: DaemonSet and Sidecar. Each serves different logging use cases and architectural needs.
The next pages explain all main methods, their configuration, and when to use each:
DaemonSet
What is a DaemonSet?
A DaemonSet is a native Kubernetes workload that ensures exactly one copy of a specific Pod runs on each node in the cluster that matches the Pod's node selection criteria.
When a new node is added to the cluster, the DaemonSet controller automatically schedules a Pod onto it. When a node is removed, the associated Pod is cleaned up.
What is the DaemonSet Controller in Kubernetes?
The DaemonSet controller is a built-in Kubernetes control loop (part of the Kubernetes Controller Manager) that manages the lifecycle of DaemonSet resources.
https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#:~:text=elasticsearch%20Docker%20image%3A-,controllers/daemonset.yaml,-apiVersion%3A%20apps DeamonSet controller
https://kubernetes.io/docs/concepts/architecture/#kube-controller-manager:~:text=interference%2C%20and%20deadlines.-,kube%2Dcontroller%2Dmanager,-Control%20plane%20component kube-controller-manager
https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/#:~:text=on%20all%20nodes.-,How%20Daemon%20Pods%20are%20scheduled,-A%20DaemonSet%20can How Daemon Pods are scheduled
Why Deploy Fluent Bit as a DaemonSet?
Deploying Fluent Bit as a DaemonSet is the canonical and officially recommended pattern for cluster-wide log aggregation in Kubernetes. This approach offers both technical efficiency and operational simplicity.
Key Advantages:
Cluster-wide visibility: One Pod per node ensures all local container logs are collected—including
/var/log/containers/*.logand other node-level files.Lightweight & scalable: Fluent Bit is optimized for minimal resource usage. One lightweight instance per node avoids central bottlenecks.
Handles node churn automatically: You declare the DaemonSet once; the Kubernetes controller ensures every new node gets a Fluent Bit Pod.
Easy deployment: Integrates cleanly with tools like Helm for repeatable and manageable installations.
Perfect for stdout/stderr logs: Ideal in environments where apps write logs to standard output, as is common with containerized workloads.
Enhanced with metadata: When combined with RBAC and the Kubernetes filter, Fluent Bit enriches logs with pod, namespace, and label metadata—making it powerful for search and analysis in systems like XPLG, Elasticsearch, or any SIEM.
Reference: Fluent Bit Kubernetes Deployment Guide (Official Docs)
Full deployment guide here:
https://xpolog.atlassian.net/wiki/spaces/XPOL/pages/2400813104/FluentBit+on+K8s+DemonSet+Deployment?atlOrigin=eyJpIjoiMWVkNTQwYTFmZTRkNGYxNzhhYThhYzZjM2I5YTk5OTMiLCJwIjoiYyJ9
SideCar
What is the SideCar pattern?
Definition – A sidecar is a second container that lives inside the same Pod as your main application.
Lifecycle – Because it shares the Pod:
It starts, stops, is scheduled, and scales exactly with the app container.
Both containers share Linux namespaces (network, IPC, cgroup limits) and the same volumes.
Understanding Kubernetes Pods and Immutability
In Kubernetes, a Pod is the smallest deployable unit and represents one or more containers running together. Crucially, Kubernetes Pods are immutable once created:
You cannot directly change the containers, configuration, or specifications of an already running Pod.
Any modifications require the Pod to be deleted and recreated.
Why immutable?
Immutability ensures consistent states, predictable deployments, reliable updates, and simplifies troubleshooting and debugging.
Use Fluent-Bit as a sidecar when you need fine-grained control, strict data separation, or your application writes proprietary log files.
For cluster-wide stdout logging with minimal overhead, stick with the DaemonSet pattern.
Full deployment guide here:
For example, I have Xpolog deployed on this k8s cluster:
Run next command:
kubectl get service xpolog-serviceis used to view details about a xpolog-service.
Here’s the breakdown of the output:
Column | Value | Meaning |
|---|---|---|
NAME |
| The name of the service you're querying. |
TYPE |
| Service type, in this case trying to provision a cloud load balancer (for external access). |
CLUSTER-IP |
| Internal IP within the Kubernetes cluster. Other pods can reach this via |
EXTERNAL-IP |
| This should show an external/public IP — but it's |
PORT(S) |
|
|
30303is the Service port (inside the cluster). Default xpolog port31114is the NodePort (exposed on all nodes at that port). In Kubernetes, when you use a Service of type NodePort/LoadBalancer, the system must assign a port from a special reserved range, not just any number. By default, Kubernetes only allows NodePort values between:30000–32767
You need to expose XpoLog inside the cluster:
Exposing = creating a route from outside (or inside) the cluster to your app.
Main Types of Exposure in Kubernetes
Type | Accessible From | Used For |
|---|---|---|
| Inside the cluster only | Pod-to-pod/internal traffic |
| Outside via | Basic external access |
| Outside via public IP | Cloud-based clusters (AWS, GCP, etc.) |
nano xpolog-service.yamlapiVersion: v1
kind: Service
metadata:
name: xpolog-service
namespace: default
spec:
type: NodePort # Minikube/kind don’t have cloud LBs
selector:
app: xpolog
ports:
- name: http-listener
port: 30303 # cluster-internal port
targetPort: 30303 # container port
nodePort: 31114 # node-ip:31114 from outside
kubectl apply -f xpolog-service.yaml
kubectl get svc xpolog-serviceYou exposed your app this way:
Your app (XPLG) runs inside a pod.
It listens on port
30303(inside the pod).You expose it to the world via a NodePort service.
For the next deployment, I’ll use just the cluster-internal port: 30303
Summary
In Kubernetes, log collection is a critical part of observability and monitoring. Fluent-Bit is a powerful, efficient tool for collecting and forwarding logs to systems like XPLG. This guide walked through the two most common deployment strategies for Fluent-Bit in Kubernetes:
🔹 DaemonSet Deployment
Runs one Fluent-Bit instance per node.
Collects logs from all containers via the node’s filesystem (
/var/log/containers/*.log).Ideal for large-scale clusters, logs written to stdout/stderr, and centralized configurations.
Can be deployed manually or using Helm, which simplifies lifecycle and configuration management.
🔹 Sidecar Deployment
Deploys a Fluent-Bit container alongside each application container.
Collects logs directly via a shared volume (e.g.,
/var/log/app/).Offers fine-grained, application-specific control and isolation.
Requires modifying pod or deployment specifications and recreating the pods.
Useful for custom log formats, compliance isolation, or debugging specific services.
Conclusion
Choosing between DaemonSet and Sidecar deployment depends on your use case:
Use Case | Recommended Approach |
|---|---|
Cluster-wide centralized logging | ✅ DaemonSet |
Per-app log routing or customization | ✅ Sidecar |
Minimal resource usage | ✅ DaemonSet |
Isolation & debugging per service | ✅ Sidecar |
Dynamic configuration via Helm | ✅ DaemonSet + Helm |
Kubernetes Pods are immutable. To use the sidecar pattern, you must update the Pod or Deployment configuration and recreate the Pod. For most production environments, DaemonSet with Helm offers the best balance between simplicity and scalability.
Both methods are valid, production-ready solutions. The right choice depends on your architecture, compliance, and operational needs. Use this guide as a foundation to confidently deploy Fluent-Bit into your Kubernetes-based logging pipeline.
Full guides here:
References
Fluent-Bit Documentation
https://docs.fluentbit.io/manual/
Official documentation for configuring, deploying, and extending Fluent-Bit.Fluent-Bit Helm Chart Repository
https://github.com/fluent/helm-charts/tree/main/charts/fluent-bit
Community-maintained Helm chart for Fluent-Bit, including values.yaml structure and options.Kubernetes Official Documentation
https://kubernetes.io/docs/home/
General reference for Kubernetes concepts like Pods, DaemonSets, Deployments, ConfigMaps, and Volumes.Kubernetes DaemonSet Controller
https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/
Detailed guide on how DaemonSets work and when to use them.Kubernetes Sidecar Pattern
https://kubernetes.io/docs/concepts/workloads/pods/#using-sidecar-containers
Explanation of the sidecar container pattern, ideal for log collection or service augmentation.Helm Official Documentation
https://helm.sh/docs/
For installing, upgrading, templating, and rolling back Kubernetes applications with Helm.