Adding Data from K8s / OpenShift (agents)

Owned by Omry Koschitzky
Last updated: Jan 11, 2023
6 min read

Background

Kubernetes (K8s) / OpenShift are container-orchestration system for automating application deployment, scaling, and management. 
Kubernetes (K8s) was originally designed by Google, and is now maintained by the Cloud Native Computing Foundation, and OpenShift is developed by Red Hat and managed by Kubernetes on a foundation of Red Hat Enterprise Linux - so basically they are very similar with regards to this article and both are based on Kubernetes (K8s).
As K8s / OpenShift are dynamic environments that automates deployment, scaling, and operations of application containers across clusters of hosts it constantly changing. Therefore, it is not possible to collect data by engaging connections to the different NODES/PODS/CONTAINERS as they constantly changes as well.

In order to get logs data processed from such dynamic environments, it is required to dynamically send the logs from K8s / OpenShift cluster to XpoLog in real time while running - whenever a new container is created in the cluster it immediately ships the logs to XpoLog.
The procedure requires an automated deployment of a lightweight log forwarder that will be automatically deployed and managed by the K8s / OpenShift cluster and send the container’s logs to XpoLog for processing/monitoring.



Recommended agent

XpoLog contains built-in listeners (HTTP/S, Syslog UDP/TCP) and data can be easily shipped to XpoLog cluster for processing and monitoring.

The most recommended agent to be used is Fluent Bit. Fluent Bit is an open source and multi-platform Log Forwarder which allows you to collect data/logs from different sources and send them to multiple destinations. It's fully compatible with Docker and K8s / OpenShift environments. Fluent Bit is one of the fastest, lightweight available in the market.

More information can be reviewed at - https://fluentbit.io/

Fluent bit supports HTTP/S output which sends the data/logs directly to XpoLog listeners in JSON format.



XpoLog Deployment

On the XpoLog side, create a HTTP/S listener on the XpoLog cluster, and make sure the K8s nodes has access to XpoLog cluster (XpoLog IP/Ports).

The listener URL should be copied from the listener definition and used in the K8s / OpenShift configuration as the output (the source when logs will be shipped to).



K8s / OpenShift Deployment

Before getting started it is important to understand how Fluent Bit will be deployed. K8s / OpenShift manages a cluster of NODES, so our log agent tool will need to run on every NODE to collect logs from every POD, hence Fluent Bit is deployed as a DaemonSet (a POD that runs on every NODE of the cluster). When Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich each entry with the following information (metadata): POD Name, POD ID, Container Name, Container ID, Labels, Annotations, etc.

In general the steps are:

  • Create the namespace, service account and the access rights of the Fluent Bit deployment.

  • Define the Fluent Bit configuration.

  • Launch the DaemonSet of Fluent Bit.



  • Fluent bit will be deployed as a DaemonSet so that it will run on every node of the K8s / OpenShift cluster.
    The Fluent Bit Docker image is available at the docker hub https://hub.docker.com/r/fluent/fluent-bit (its is recommended to use latest stable docker image), download it and deploy it to your K8s / OpenShift cluster.

  • Define the Fluent Bit configuration
    Create a configMap that will be used by our Fluent bit instances with the required configuration to forward all containers logs from the running Node to XpoLog (input: /var/log/containers/* output: the required HTTP/S URL – IP/PORT/TOKEN to XpoLog cluster).
    Use the this yaml as the configuration of your Fluent Bit. All you have to change in this file is update the http OUTPUT part, and replace the placeholders (XPOLOG_IP, XPOLOG_PORT, XPOLOG_HTTP_LISTENER_TOKEN) with the HTTP Listener details which you had created in XpoLog.

  • Launch the DaemonSet of Fluent Bit that on each node startup, a pod which executes the FluentBit docker will be started also.
    Example yaml can be found here - note, you'll need to modify it according to your cluster details.



  • The Fluent bit POD will be automatically deployed by the K8s / OpenShift cluster on every running node and forward data to XpoLog.

  • Data Enrichment - the Fluent bit enriches logs with K8s / OpenShift metadata (POD NAME, POD ID, Container Name, Container ID, Labels, NAMESPACE, etc.) by a built-in filter plugin that ‘talks’ to the K8s / OpenShift API Server to retrieve relevant information.

Contact us for more information (useful general guide to deploy Fluent Bit).



High Level Deployment Diagram





Background

Kubernetes (K8s) / OpenShift are container-orchestration system for automating application deployment, scaling, and management. 
Kubernetes (K8s) was originally designed by Google, and is now maintained by the Cloud Native Computing Foundation, and OpenShift is developed by Red Hat and managed by Kubernetes on a foundation of Red Hat Enterprise Linux - so basically they are very similar with regards to this article and both are based on Kubernetes (K8s).
As K8s / OpenShift are dynamic environments that automates deployment, scaling, and operations of application containers across clusters of hosts it constantly changing. Therefore, it is not possible to collect data by engaging connections to the different NODES/PODS/CONTAINERS as they constantly changes as well.

In order to get logs data processed from such dynamic environments, it is required to dynamically send the logs from K8s / OpenShift cluster to XpoLog in real time while running - whenever a new container is created in the cluster it immediately ships the logs to XpoLog.
The procedure requires an automated deployment of a lightweight log forwarder that will be automatically deployed and managed by the K8s / OpenShift cluster and send the container’s logs to XpoLog for processing/monitoring.



Recommended agent

XpoLog contains built-in listeners (HTTP/S, Syslog UDP/TCP) and data can be easily shipped to XpoLog cluster for processing and monitoring.

The most recommended agent to be used is Fluent Bit. Fluent Bit is an open source and multi-platform Log Forwarder which allows you to collect data/logs from different sources and send them to multiple destinations. It's fully compatible with Docker and K8s / OpenShift environments. Fluent Bit is one of the fastest, lightweight available in the market.

More information can be reviewed at - https://fluentbit.io/

Fluent bit supports HTTP/S output which sends the data/logs directly to XpoLog listeners in JSON format.



XpoLog Deployment

On the XpoLog side, create a HTTP/S listener on the XpoLog cluster, and make sure the K8s nodes has access to XpoLog cluster (XpoLog IP/Ports).

The listener URL should be copied from the listener definition and used in the K8s / OpenShift configuration as the output (the source when logs will be shipped to).



K8s / OpenShift Deployment

Before getting started it is important to understand how Fluent Bit will be deployed. K8s / OpenShift manages a cluster of NODES, so our log agent tool will need to run on every NODE to collect logs from every POD, hence Fluent Bit is deployed as a DaemonSet (a POD that runs on every NODE of the cluster). When Fluent Bit runs, it will read, parse and filter the logs of every POD and will enrich each entry with the following information (metadata): POD Name, POD ID, Container Name, Container ID, Labels, Annotations, etc.

In general the steps are:

  • Create the namespace, service account and the access rights of the Fluent Bit deployment.

  • Define the Fluent Bit configuration.

  • Launch the DaemonSet of Fluent Bit.



  • Fluent bit will be deployed as a DaemonSet so that it will run on every node of the K8s / OpenShift cluster.
    The Fluent Bit Docker image is available at the docker hub https://hub.docker.com/r/fluent/fluent-bit (its is recommended to use latest stable docker image), download it and deploy it to your K8s / OpenShift cluster.

  • Define the Fluent Bit configuration
    Create a configMap that will be used by our Fluent bit instances with the required configuration to forward all containers logs from the running Node to XpoLog (input: /var/log/containers/* output: the required HTTP/S URL – IP/PORT/TOKEN to XpoLog cluster).
    Use the this yaml as the configuration of your Fluent Bit. All you have to change in this file is update the http OUTPUT part, and replace the placeholders (XPOLOG_IP, XPOLOG_PORT, XPOLOG_HTTP_LISTENER_TOKEN) with the HTTP Listener details which you had created in XpoLog.

  • Launch the DaemonSet of Fluent Bit that on each node startup, a pod which executes the FluentBit docker will be started also.
    Example yaml can be found here - note, you'll need to modify it according to your cluster details.



  • The Fluent bit POD will be automatically deployed by the K8s / OpenShift cluster on every running node and forward data to XpoLog.

  • Data Enrichment - the Fluent bit enriches logs with K8s / OpenShift metadata (POD NAME, POD ID, Container Name, Container ID, Labels, NAMESPACE, etc.) by a built-in filter plugin that ‘talks’ to the K8s / OpenShift API Server to retrieve relevant information.

Contact us for more information (useful general guide to deploy Fluent Bit).



High Level Deployment Diagram