topic Re: Confluent Kafka connection in Talend Studio

Confluent Kafka connection

suresh_gopalan — Fri, 15 Nov 2024 23:15:40 GMT

Hi,

Env: Talend Cloud /Talend Fabric 8.0

I am trying to confligure tKafkaInput component to read messages from our Confluent Kafka Topic

I selected version Kafka 2.2.1 from the drop down

Given Broker list, Topic Name, Consumer Group ID

We are using SASL/PLAIN for security.

Where do we set this?

What are the minimum set of properties needed to set (assuming it is in Advanced Settings) to read messages from Confluent Kafka Topic?

How to set them here?

I saw a 2 year old post

https://community.talend.com/s/question/0D53p00007vCx5kCAC/how-to-define-a-jaas-config-file-for-use-with-kafka-in-talend-big-data?language=en_US

In this, the suggestion is to create a custom component. Now also, Is this the only solution, so NO support out of the box?

Please help.

Re: Confluent Kafka connection

Anonymous — Tue, 01 Feb 2022 18:20:24 GMT

These precise settings may not work for you, but this will be the way to attempt this. I have to admit that I haven't done this and don't have a system to test this on. So I am passing this on as something that I would try if I could....

Go to your Advanced Settings and look at the Kafka Properties box. Then experiment with the following properties....

bootstrap.servers=localhost:9092

ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

ssl.truststore.location=/path/to/truststore

ssl.truststore.password=password

ssl.protocol=TLS

security.protocol=SASL_SSL

sasl.mechanism=PLAIN

sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="admin-secret"

Re: Confluent Kafka connection

suresh_gopalan — Tue, 01 Feb 2022 23:09:56 GMT

Talend Support helped. Below settings worked

Created a config file named "KafkaClient.conf" with below content and save it to a local folder

KafkaClient {

org.apache.kafka.common.security.plain.PlainLoginModule required

username="<your api key>"

password="<your secret>";

};

1) In tKafkaInput's Basic Settings tab, give Broker List, Topic Name and Consumer Group ID

2) In Advanced settings -> Kafka properties, add the following Names, Values

"security.protocol" , "SASL_SSL"

"sasl.mechanism", "PLAIN"

In Advanced settings -> Use Specific JVM Arguments, enter below

-Djava.security.auth.login.config=<File Path>/KafkaClient.conf

3) Run the job

If you are NOT sure about the security config of Confluent Kafka Account then run the job in debug mode and see the properties. By default Talend expect Kerberos config. To overcome we need to set the properties in Advanced settings.

I am attaching few screenshots for reference

Re: Confluent Kafka connection

suresh_gopalan — Tue, 01 Feb 2022 23:25:36 GMT

See screenshots for kafka input settings

Re: Confluent Kafka connection

suresh_gopalan — Tue, 01 Feb 2022 23:27:58 GMT

Re: Confluent Kafka connection

suresh_gopalan — Tue, 01 Feb 2022 23:28:15 GMT

Re: Confluent Kafka connection

suresh_gopalan — Tue, 01 Feb 2022 23:28:32 GMT

Re: Confluent Kafka connection

suresh_gopalan — Tue, 01 Feb 2022 23:28:50 GMT

Re: Confluent Kafka connection

Anonymous — Wed, 02 Feb 2022 09:32:40 GMT

@Suresh Gopalan thanks for posting this solution!