Background
Kubernetes (K8s) / OpenShift are container-orchestration system for automating application deployment, scaling, and management.
Kubernetes (K8s) was originally designed by Google, and is now maintained by the Cloud Native Computing Foundation, and OpenShift is developed by Red Hat and managed by Kubernetes on a foundation of Red Hat Enterprise Linux - so basically they are very similar with regards to this article and both are based on Kubernetes (K8s).
As K8s / OpenShift are dynamic environments that automates deployment, scaling, and operations of application containers across clusters of hosts it constantly changing. Therefore, it is not possible to collect data by engaging connections to the different NODES/PODS/CONTAINERS as they constantly changes as well.
...
In general the steps are:
- Create the namespace, service account and the access rights of the Fluent Bit deployment.
- Define the Fluent Bit configuration.
- Launch the DaemonSet of Fluent Bit.
- Fluent bit will be deployed as a DaemonSet so that it will run on every node of the K8s / OpenShift cluster.
The Fluent bit version which is supported by XpoLog is version 1.0.4. The Fluent Bit Docker image is available at the docker hub https://hub.docker.com/r/fluent/fluent-bit (fluent/fluent-bit:1.0.4 or laterits is recommended to use latest stable docker image), download it and deploy it to your K8s / OpenShift cluster. - Define the Fluent Bit configuration
Create a configMap that will be used by our Fluent bit instances with the required configuration to forward all containers logs from the running Node to XpoLog (input: /var/log/containers/* output: the required HTTP/S URL – IP/PORT/TOKEN to XpoLog cluster).
Use the this yaml as the configuration of your Fluent Bit. All you have to change in this file is update the http OUTPUT part, and replace the placeholders (XPOLOG_IP, XPOLOG_PORT, XPOLOG_HTTP_LISTENER_TOKEN) with the HTTP Listener details which you had created in XpoLog. - Launch the DaemonSet of Fluent Bit that on each node startup, a pod which executes the FluentBit docker will be started also.
Example yaml can be found here - note, you'll need to modify it according to your cluster details. - The Fluent bit POD will be automatically deployed by the K8s / OpenShift cluster on every running node and forward data to XpoLog.
- Data Enrichment - the Fluent bit enriches logs with K8s / OpenShift metadata (POD NAME, POD ID, Container Name, Container ID, Labels, NAMESPACE, etc.) by a built-in filter plugin that ‘talks’ to the K8s / OpenShift API Server to retrieve relevant information.
...