Background
Kubernetes (K8S) / OpenShift are container-orchestration system for automating application deployment, scaling, and management.
As K8S / OpenShift are dynamic environments that automates deployment, scaling, and operations of application containers across clusters of hosts it constantly changing. Therefore, it is not possible to collect data by engaging connections to the different NODES/PODS/CONTAINERS as they constantly changes as well.
In order to get logs data processed from such dynamic environments, it is required to dynamically send the logs from K8S / OpenShift cluster to XpoLog in real time while running - whenever a new container is created in the cluster it immediately ships the logs to XpoLog.
The procedure requires an automated deployment of a lightweight log forwarder that will be automatically deployed and managed by the K8S / OpenShift cluster and send the container’s logs to XpoLog for processing/monitoring.
Recommended agent
XpoLog contains built-in listeners (HTTP/S, Syslog UDP/TCP) and data can be easily shipped to XpoLog cluster for processing and monitoring.
The most recommended agent to be used is Fluent Bit. Fluent Bit is an open source and multi-platform Log Forwarder which allows you to collect data/logs from different sources and send them to multiple destinations. It's fully compatible with Docker and K8S / OpenShift environments. Fluent Bit is one of the fastest, lightweight available in the market.
More information can be reviewed at - https://fluentbit.io/
Fluent bit supports HTTP/S output which sends the data/logs directly to XpoLog listeners in JSON format.
XpoLog Deployment
On the XpoLog side, create a HTTP/S listener on the cluster.
The listener URL should be copied from the listener definition and used in the K8S / OpenShift configuration as the output (the source when logs will be shipped to).
K8S / OpenShift Deployment
Before getting started it is important to understand how Fluent Bit will be deployed. K8S / OpenShift manages a cluster of NODES, so our log agent tool will need to run on every NODE to collect logs from every POD, hence Fluent Bit is deployed as a DaemonSet (a POD that runs on every NODE of the cluster).When Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich each entry with the following information (metadata):
- POD Name
- POD ID
- Container Name
- Container ID
- Labels
- Annotations
- Fluent bit will be deployed as a DaemonSet so that it will run on every node of the K8S / OpenShift cluster.
- Create a ConfigMap (YAML based configuration settings) that will be used by our Fluent bit DaemonSet with the required configuration to forward all containers logs from the running Node to XpoLog (input: /var/log/containers/* output: the required HTTP/S URL – IP/PORT/TOKEN to XpoLog cluster).
- The Fluent bit POD will be automatically deployed by the K8S / OpenShift cluster on every running node and forward data to XpoLog.
- Data Enrichment - the Fluent bit enriches logs with K8S / OpenShift metadata (POD NAME, POD ID, Container Name, Container ID, Labels, NAMESPACE, etc.) by a built-in filter plugin that ‘talks’ to the K8S / OpenShift API Server to retrieve relevant information.
High Level Deployment Diagram
