Background
Kubernetes (K8SK8s) / OpenShift are container-orchestration system for automating application deployment, scaling, and management.
Kubernetes (K8SK8s) was originally designed by Google, and is now maintained by the Cloud Native Computing Foundation, and OpenShift is developed by Red Hat and managed by Kubernetes on a foundation of Red Hat Enterprise Linux - so basically they are very similar with regards to this article and both are based on Kubernetes (K8SK8s).
As K8S K8s / OpenShift are dynamic environments that automates deployment, scaling, and operations of application containers across clusters of hosts it constantly changing. Therefore, it is not possible to collect data by engaging connections to the different NODES/PODS/CONTAINERS as they constantly changes as well.
In order to get logs data processed from such dynamic environments, it is required to dynamically send the logs from K8S K8s / OpenShift cluster to XpoLog in real time while running - whenever a new container is created in the cluster it immediately ships the logs to XpoLog.
The procedure requires an automated deployment of a lightweight log forwarder that will be automatically deployed and managed by the K8S K8s / OpenShift cluster and send the container’s logs to XpoLog for processing/monitoring.
...
The most recommended agent to be used is Fluent Bit. Fluent Bit is an open source and multi-platform Log Forwarder which allows you to collect data/logs from different sources and send them to multiple destinations. It's fully compatible with Docker and K8S and K8s / OpenShift environments. Fluent Bit is one of the fastest, lightweight available in the market.
...
On the XpoLog side, create a HTTP/S listener on the XpoLog cluster, and make sure the K8s nodes has access to XpoLog cluster (XpoLog IP/Ports).
The listener URL should be copied from the listener definition and used in the K8S K8s / OpenShift configuration as the output (the source when logs will be shipped to).
K8S K8s / OpenShift Deployment
Before getting started it is important to understand how Fluent Bit will be deployed. K8S K8s / OpenShift manages a cluster of NODES, so our log agent tool will need to run on every NODE to collect logs from every POD, hence Fluent Bit is deployed as a DaemonSet (a POD that runs on every NODE of the cluster). When Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich each entry with the following information (metadata): POD Name
...
, POD ID
...
, Container Name
...
, Container ID
...
, Labels
...
, Annotations, etc.
...
- Fluent bit will be deployed as a DaemonSet so that it will run on every node of the
...
- K8s / OpenShift cluster.
...
- The Fluent bit version which is supported by XpoLog is version 1.0.4. The Fluent Bit Docker image is available at the docker hub https://hub.docker.com/r/fluent/fluent-bit (fluent/fluent-bit:1.0.4), download it and deploy it to your K8s / OpenShift cluster.
- Configure the DaemonSet that on each node startup, a pod which executes the FluentBit docker will be started also.
- Create a ConfigMap (YAML based configuration settings) that will be used by our Fluent bit DaemonSet with the required configuration to forward all containers logs from the running Node to XpoLog (input: /var/log/containers/* output: the required HTTP/S URL – IP/PORT/TOKEN to XpoLog cluster).
...
- Use the this yaml as the configuration of your Fluent Bit. All you have to change in this file is update the http OUTPUT part, and replace the placeholders (XPOLOG_IP, XPOLOG_PORT, XPOLOG_HTTP_LISTENER_TOKEN) with the HTTP Listener details which you had created in XpoLog.
- The Fluent bit POD will be automatically deployed by the
...
- K8s / OpenShift cluster on every running node and forward data to XpoLog.
...
- Data Enrichment - the Fluent bit enriches logs with
...
- K8s / OpenShift metadata (POD NAME, POD ID, Container Name, Container ID, Labels, NAMESPACE, etc.) by a built-in filter plugin that ‘talks’ to the
...
- K8s / OpenShift API Server to retrieve relevant information.
High Level Deployment Diagram
...