DaemonSet vs Sidecar

Overview

Fluent Bit is a lightweight, high-performance log processor and forwarder. In Kubernetes, it is commonly used to collect logs from containers, enrich them with metadata, and forward them to external systems such as XPLG.

There are two main deployment strategies:

DaemonSet (cluster-wide log collection)
Sidecar (per-pod log collection)

This page explains both methods, their configuration, and when to use each.

For this guide, I have xpolog deployed on this cluster:

Just need to expose XpoLog inside the cluster:

nano xpolog-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: xpolog-service
  namespace: default
spec:
  type: NodePort               # Minikube/kind don’t have cloud LBs
  selector:
    app: xpolog
  ports:
  - name: http-listener
    port: 30303                # cluster-internal port
    targetPort: 30303          # container port
    nodePort: 31114            # node-ip:31114 from outside

kubectl apply -f xpolog-service.yaml
kubectl get svc xpolog-service

Run xpolog and take token from http listener:

Deployment Options

1. DaemonSet (Regular)

Description:
Runs one Fluent Bit pod per node. It tails logs from all containers on the node using the path /var/log/containers/*.log.

Advantages:

Centralized and scalable
Lightweight (1 pod per node)
Easy to manage using Helm
Ideal for clusters where logs go to stdout/stderr

2. SideCar

This deployment method is per application pod.

Understanding Kubernetes Pods and Immutability

In Kubernetes, a Pod is the smallest deployable unit and represents one or more containers running together. Crucially, Kubernetes Pods are immutable once created:

You cannot directly change the containers, configuration, or specifications of an already running Pod.

Any modifications require the Pod to be deleted and recreated.

Why immutable?

Immutability ensures consistent states, predictable deployments, reliable updates, and simplifies troubleshooting and debugging.

Deployment guide:

Regular (DemonSet)

Example cluster: Simple k8s cluster with 3 renning pods:

Step 1: Create Namespace

kubectl create namespace logging

RBAC & ServiceAccount for Fluent Bit

nano fluentbit-rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: logging
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources: ["pods", "namespaces"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: logging

kubectl apply -f fluentbit-rbac.yaml

Step 2: Create Fluent Bit ConfigMap

Create a file named fluentbit-config.yaml

nano fluentbit-config.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  # ---- main ----
  fluent-bit.conf: |
    [SERVICE]
        Flush        5
        Daemon       Off
        Log_Level    info
        Parsers_File parsers.conf
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020

    @INCLUDE inputs.conf
    @INCLUDE filters.conf
    @INCLUDE outputs.conf

  # ---- input ----
  inputs.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  # ---- filter ----
  filters.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log
        Keep_Log            Off

  # ---- output ----
  outputs.conf: |
    [OUTPUT]
        Name              http
        Match             *
        Host              10.98.116.173
        Port              30303
        URI               /logeye/api/logger.jsp?token=5693be0-bd51-41f5-8968-15f6f946af53
        Format            json_lines
        Json_date_key     time
        Json_date_format  iso8601
        Header            X-Xpolog-Sender  local-k8s
        Retry_Limit       5

  # ---- parser ----
  parsers.conf: |
    [PARSER]
        Name             docker
        Format           json
        Time_Key         time
        Time_Format      %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep        On
        Decode_Field_As  escaped log do_next
        Decode_Field_As  json    log

Apply it:

kubectl apply -f fluentbit-config.yaml

Step 3: Deploy Fluent Bit DaemonSet

Create a file named fluent-bit-daemonset.yaml:

nano fluent-bit-daemonset.yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels: { app.kubernetes.io/name: fluent-bit }
spec:
  selector:
    matchLabels: { app.kubernetes.io/name: fluent-bit }
  template:
    metadata:
      labels: { app.kubernetes.io/name: fluent-bit }
    spec:
      serviceAccountName: fluent-bit
      tolerations:
      - operator: Exists              # run on every node, even control-plane
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:2.2.2
        resources:
          requests: { cpu: "50m", memory: "100Mi" }
          limits:   { cpu: "100m", memory: "200Mi" }
        volumeMounts:
        - name: config
          mountPath: /fluent-bit/etc
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        livenessProbe:
          httpGet:
            path: /api/v1/health
            port: 2020
          initialDelaySeconds: 10
          periodSeconds: 60
      volumes:
      - name: config
        configMap:
          name: fluent-bit-config
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

Fluent Bit is pulled from the Docker Hub public registry.

Apply it:

kubectl apply -f fluent-bit-daemonset.yaml

Step 4: Verify Deployment

kubectl get pods -n logging
kubectl logs -n logging <fluent-bit-pod-name>

Helm Deployment (DaemonSet with Helm)

Helm is Package Manager for Kubernetes

Step 1: Install Helm (if not already installed)

Helm Installation Guide

Step 2: Add Fluent Helm Repository

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

Step 3: Create Custom `values.yaml`

Create values.yaml:

# ------------------------------------------------------------------------------
# values.yaml  – override file for chart fluent/fluent-bit
# ------------------------------------------------------------------------------

############################
# Global
############################
fullnameOverride: fluent-bit          # produces "fluent-bit" resources
image:
  repository: fluent/fluent-bit
  tag: 2.2.2                          # same image you used manually
  pullPolicy: IfNotPresent

############################
# Namespace / RBAC
############################
serviceAccount:
  create: true
  name: fluent-bit                    # must match DaemonSet spec
rbac:
  create: true

############################
# Resources & scheduling
############################
resources:
  requests:
    cpu: 50m
    memory: 100Mi
  limits:
    cpu: 100m
    memory: 200Mi

tolerations:                          # run on any tainted node (incl. control-plane)
  - operator: Exists

############################
# DaemonSet
############################
kind: DaemonSet
podLabels:
  app.kubernetes.io/name: fluent-bit

############################
# Fluent Bit configuration
############################
## The chart lets you provide raw Fluent Bit files via `.config.[service|inputs|filters|outputs|parsers]`
config:
  service: |
    [SERVICE]
        Flush        5
        Daemon       Off
        Log_Level    info
        Parsers_File parsers.conf
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020

  inputs: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filters: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log
        Keep_Log            Off

  outputs: |
    [OUTPUT]
        Name              http
        Match             *
        Host              10.98.116.173
        Port              30303
        URI               /logeye/api/logger.jsp?token=5693be0-bd51-41f5-8968-15f6f946af53
        Format            json_lines
        Json_date_key     time
        Json_date_format  iso8601
        Header            X-Xpolog-Sender  local-k8s
        Retry_Limit       5

  parsers: |
    [PARSER]
        Name             docker
        Format           json
        Time_Key         time
        Time_Format      %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep        On
        Decode_Field_As  escaped log do_next
        Decode_Field_As  json    log

############################
# Volumes – identical to manual manifest
############################
volumes:
  - name: varlog
    hostPath:
      path: /var/log
  - name: varlibdockercontainers
    hostPath:
      path: /var/lib/docker/containers
volumeMounts:
  - name: varlog
    mountPath: /var/log
    readOnly: true
  - name: varlibdockercontainers
    mountPath: /var/lib/docker/containers
    readOnly: true

############################
# Health check
############################
livenessProbe:
  httpGet:
    path: /api/v1/health
    port: 2020
  initialDelaySeconds: 10
  periodSeconds: 60

Step 4: Deploy Fluent Bit with Helm

helm install fluent-bit fluent/fluent-bit \
  -n logging --create-namespace \
  -f values.yaml

Step 5: Verify Deployment

kubectl get pods -n logging helm status fluent-bit -n logging

Sidecar Deployment

Proper Way to Add Fluent Bit Sidecar to an Existing App

The correct way to add Fluent Bit sidecar containers involves these steps:

Obtain current Pod configuration
Edit configuration to include Fluent Bit sidecar
Reapply configuration (delete old Pod, create new one automatically)

Prerequisites

# one-time XpoLog listener token
XPOLOG_TOKEN=5693be0-bd51-41f5-8968-15f6f946af53
# XpoLog service must already exist (from the earlier steps)
kubectl get svc xpolog-service

Step 1: Export Existing Pod Configuration

Export existing Pod definition to a YAML file:

kubectl get pod apache-deploy-xxxxx -o yaml > apache-pod.yaml

Step 2: Modify Pod YAML to Include Fluent Bit Sidecar

Edit apache-pod.yaml in your text editor.

Add Fluent Bit Container:

Under spec.containers, add the sidecar container:

- name: fluent-bit-sidecar
  image: fluent/fluent-bit:latest
  volumeMounts:
    - name: applogs
      mountPath: /var/log/app
    - name: config
      mountPath: /fluent-bit/etc/

Explanation:

Adds Fluent Bit container alongside your main app container.
Shares log volume (applogs) between containers.
Fluent Bit configuration (config) is mounted from a ConfigMap.

Add Volumes:

Make sure your Pod has these volumes under spec.volumes:

volumes:
  - name: applogs
    emptyDir: {}
  - name: config
    configMap:
      name: fluent-bit-sidecar-config

Note:
Your main container (Apache) must mount /var/log/app for log collection.

Example of main container (apache) definition:

- name: apache
  image: httpd:latest
  volumeMounts:
    - name: applogs
      mountPath: /var/log/app

✅ Step 3: Create Fluent Bit ConfigMap (if not exists)

fluent-bit-sidecar-config.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-sidecar-config
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush 5
        Daemon Off
        Log_Level info

    [INPUT]
        Name tail
        Path /var/log/app/*.log
        Tag app.logs

    [OUTPUT]
        Name http
        Match app.logs
        Host <XPLG_HTTP_LISTENER_IP>  # replace with your XPLG IP
        Port <PORT>                   # replace with your XPLG Port
        Format json

Full yaml file:

###############################################################################
# 0) (Optional) Namespace — delete this block if you prefer “default”
###############################################################################
apiVersion: v1
kind: Namespace
metadata:
  name: apache
---
###############################################################################
# 1) Fluent Bit configuration — holds per-app settings
###############################################################################
apiVersion: v1
kind: ConfigMap
metadata:
  name: apache-fluentbit-config
  namespace: apache        # ← change if you removed/changed the namespace
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        5
        Log_Level    info
        Parsers_File parsers.conf

    @INCLUDE inputs.conf
    @INCLUDE outputs.conf

  inputs.conf: |
    [INPUT]
        Name              tail
        Path              /var/log/app/*.log
        Tag               apache
        Parser            none              # httpd writes plain text
        Refresh_Interval  5

  outputs.conf: |
    [OUTPUT]
        Name      http
        Match     *
        Host      10.98.116.173   # XpoLog service
        Port      30303
        URI       /logeye/api/logger.jsp?token=5693be0-bd51-41f5-8968-15f6f946af53
        Format    json_lines
        Header    X-Xpolog-Sender  apache-sidecar
        Retry_Limit 5

  parsers.conf: |
    # (none needed for plain text; kept to satisfy Parsers_File directive)
---
###############################################################################
# 2) Deployment with TWO containers: Apache + Fluent Bit sidecar
###############################################################################
apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache-deploy
  namespace: apache
  labels: { app: apache }
spec:
  replicas: 1
  selector:
    matchLabels: { app: apache }
  template:
    metadata:
      labels: { app: apache }
    spec:
      # Shared volume: app writes logs, FB reads them
      volumes:
      - name: applogs
        emptyDir: {}
      - name: fluentbit-config
        configMap:
          name: apache-fluentbit-config
      containers:
      # -------- 2.1  MAIN APACHE CONTAINER --------
      - name: apache
        image: httpd:latest
        # Redirect stdout & stderr to /var/log/app/stdout.log
        command: ["/bin/sh","-c"]
        args:
          - |
            httpd-foreground 2>&1 | tee /var/log/app/stdout.log
        volumeMounts:
        - name: applogs
          mountPath: /var/log/app
      # -------- 2.2  FLUENT BIT SIDECAR --------
      - name: fluent-bit
        image: fluent/fluent-bit:2.2.2
        resources:
          requests: { cpu: 25m, memory: 50Mi }
          limits:   { cpu: 50m,  memory: 100Mi }
        volumeMounts:
        - name: applogs
          mountPath: /var/log/app           # same path as the app
        - name: fluentbit-config
          mountPath: /fluent-bit/etc        # default FB config dir
          readOnly: true

Apply it:

kubectl apply -f fluent-bit-sidecar-config.yaml

✅ Step 4: Delete the Old Pod and Apply Updated YAML

Because Pods cannot be edited directly, you must delete and re-apply the modified YAML:

kubectl delete pod apache-deploy-xxxxx kubectl apply -f apache-pod.yaml

If you use a Deployment (highly recommended), just edit the Deployment YAML and run:

kubectl apply -f deployment.yaml

Kubernetes automatically manages rolling updates with zero downtime.

✅ Step 5: Verification

Confirm your Pod runs with two containers:

kubectl get pods

Check both containers:

kubectl describe pod apache-deploy-xxxxx

Summary

In Kubernetes, log collection is a critical part of observability and monitoring. Fluent Bit is a powerful, efficient tool for collecting and forwarding logs to systems like XPLG. This guide walked through the two most common deployment strategies for Fluent Bit in Kubernetes:

🔹 DaemonSet Deployment

Runs one Fluent Bit instance per node.
Collects logs from all containers via the node’s filesystem (/var/log/containers/*.log).
Ideal for large-scale clusters, logs written to stdout/stderr, and centralized configurations.
Can be deployed manually or using Helm, which simplifies lifecycle and configuration management.

🔹 Sidecar Deployment

Deploys a Fluent Bit container alongside each application container.
Collects logs directly via a shared volume (e.g., /var/log/app/).
Offers fine-grained, application-specific control and isolation.
Requires modifying pod or deployment specifications and recreating the pods.
Useful for custom log formats, compliance isolation, or debugging specific services.

🧠 Conclusion

Choosing between DaemonSet and Sidecar deployment depends on your use case:

Use Case	Recommended Approach

Use Case	Recommended Approach
Cluster-wide centralized logging	✅ DaemonSet
Per-app log routing or customization	✅ Sidecar
Minimal resource usage	✅ DaemonSet
Isolation & debugging per service	✅ Sidecar
Dynamic configuration via Helm	✅ DaemonSet + Helm

Kubernetes Pods are immutable. To use the sidecar pattern, you must update the Pod or Deployment configuration and recreate the Pod. For most production environments, DaemonSet with Helm offers the best balance between simplicity and scalability.

Both methods are valid, production-ready solutions. The right choice depends on your architecture, compliance, and operational needs. Use this guide as a foundation to confidently deploy Fluent Bit into your Kubernetes-based logging pipeline.

References

Fluent Bit Documentation
Fluent Bit Documentation | Fluent Bit: Official Manual
Official documentation for configuring, deploying, and extending Fluent Bit.
Fluent Bit Helm Chart Repository
helm-charts/charts/fluent-bit at main · fluent/helm-charts
Community-maintained Helm chart for Fluent Bit, including values.yaml structure and options.
Kubernetes Official Documentation
Kubernetes Documentation
General reference for Kubernetes concepts like Pods, DaemonSets, Deployments, ConfigMaps, and Volumes.
Kubernetes DaemonSet Controller
DaemonSet
Detailed guide on how DaemonSets work and when to use them.
Kubernetes Sidecar Pattern
Pods
Explanation of the sidecar container pattern, ideal for log collection or service augmentation.
Helm Official Documentation
Helm | Docs Home
For installing, upgrading, templating, and rolling back Kubernetes applications with Helm.

Send data to XPLG using Fluent-Bit (Kubernetes)