Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

Announcements
Join us in Toronto Sept 9th for Qlik's AI Reality Tour! Register Now
cancel
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable

What is the best component to use to read messages from Azure Event Hub in real-time?

We need to read data published to Azure Event Hub in real-time. Which Talend component is best suited to fulfill this requirement?

Labels (2)
5 Replies
Anonymous
Not applicable
Author

I've not done this, however from what I understand Azure Event Hub is compliant with AMQP (https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-amqp-protocol-guide). As such you can use Talend ESB and the cAMQP component.

Anonymous
Not applicable
Author

I am trying to use cAMQP component with cConnectionFactory. However, I am running into connection issue due to firewall blocking the port (Connection refused). Per Azure documentation, Event Hubs support AMQP over websockets. Is there a way to connect to Event Hub using AMQP over websockets, so that we don't have to open up firewall for AMQP specific ports for TCP?

Anonymous
Not applicable
Author

I'm afraid I am not an expert in this and have never tried. However Apache Camel do provide a Websocket component. Maybe this will be of use to you

https://camel.apache.org/websocket.html

Anonymous
Not applicable
Author

Thank you for your feedback. I am working on opening firewall, as we do not have Talend experts who can help create custom components that can communicate via AMQP protocol over Camel websockets. I was hoping that with so many standard components available in Talend, we would have a standard component to connect to Event hub, which is similar to Service Bus. I tried using MQconnectionFactory in combination with cAMQP component on a machine without firewall restriction and I keep getting amqp authorization failure. The configuration on Event Hub seems sufficient, as the same SAS credentials work on a .Net stub provided by Microsoft. Its surprising that there are several examples of using different components, but I could not find a single example explaining connecting Talend to Azure Event Hub. Has anyone ever successfully connected to Azure Event Hub using Talend? If so, can you please advise if there are any other components or pre-requisites needed to get it to work?  

I followed example on Talend documentation site (https://help.talend.com/reader/Cu21OB3bCNfqp6r4JCCNzQ/kkt0YENIDNzdc21~v50_1w) and used below values for connection factory

MQ Server : AMQP 1.0

Host: "aaaa-eventhub.servicebus.windows.net/;SharedAccessKeyName=<name>;SharedAccessKey=<SASprimaryKey>"

Port: 5671 (tried both 5671 and 5672 as number and string)

Anonymous
Not applicable
Author

You do not need to create a new component to use the Apache Camel components. All you need to do is to use the cMessagingEndpoint component. The Apache Camel components are simply configured using URIs. This document explains it...

https://help.talend.com/reader/M0piPAcrxN~73Yo0IsrZtQ/_F6Lk7R_Q5b5k3zruFp~6A

 

With regard to shortage of examples, I can request that an example is written. But I do not believe it will be produced immediately. If you manage to get this working, then I welcome an example from you which I will happily publish with you as the author 🙂

 

Sorry I can't be of immediate help, but I will look at getting a demonstration produced