How to stream data from Azure Event Hubs using a Kafka Listener
Azure Event Hubs provides an Apache Kafka endpoint on an event hub, which enables users to connect to the event hub using the Kafka protocol.
To stream the data into XPLG, we can configure a Kafka listener and by that, connect to your Azure event hub and pull the data.
Microsoft Azure Event Hubs supports the default Kafka protocol ports, which are TLS 9092 and non-TLS 9093.
Relevant Microsoft articles:
https://learn.microsoft.com/en-us/azure/event-hubs/azure-event-hubs-kafka-overview
Kafka and Event Hubs conceptual mapping
Kafka Concept | Event Hubs Concept |
|---|---|
Cluster | Namespace |
Topic | An event hub |
Partition | Partition |
Consumer Group | Consumer Group |
Offset | Offset |
Azure Side
namespace / Event Hub → which translates to Cluster / Topic:
For generating the private key, navigate to access policy, create a new key and copy the connection string private key:
A connection string–primary key from azure console should have the following structure:
Endpoint=sb://xplg-test-demo.servicebus.windows.net/;SharedAccessKeyName=nick;SharedAccessKey=pz9twEESO0qhdFThJeYPsmNd3EDKJDsdfsdfsdfSvpcY=;EntityPath=xplgazurekafka
XPLG Side
Open an outbound firewall rule on XpoLog server, port 9093 (The default Apache Kafka TLS port is 9093, and the default non-TLS port is 9092).
Navigate to PortX → Data → Listen to Data and create a new Kafka Listener:
Properties that need to be adjusted in order to use the Apache Kafka protocol:
Host: NAMESPACENAME.servicebus.windows.net:9093 / Cluster Address
Topic: EventHub
User Name: $ConnectionString
Password: The private key that was generated by Azure. Please find a sample string (including the “Endpoint=“ in the beginning):
Endpoint=sb://xplg-test-demo.servicebus.windows.net/;SharedAccessKeyName=nick;SharedAccessKey=pz9twEESO0qhdFThJeYPsmNd3EDKJDsdfsdfsdfSvpcY=;EntityPath=xplgazurekafka
Auth Mechanism: PLAIN
Security Protocol: SASL_SSL
Security Module: org.apache.kafka.common.security.plain
All values are mandatory to establish the connection.