How to define a jaas config file for use with Kafka in Talend Big Data

[adbdkb]

I am trying to read from Kfka topics and trying to set up the job.  I have defined the jaas config file as below.  When I try to run the job I get the following exception.    How should my config file be defined for this to work?

KafkaClient {
security.protocol=SASL_SSL
sasl.method=PLAIN
org.apache.kafka.common.security.plain.PlainLoginModule required \
username="user" \
password="password";
};

Exception in component tKafkaInput_1 (Test)

org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:793)

at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:644)

at big_data_poc.test_0_1.Test.tKafkaInput_1Process(Test.java:848)

To see the whole post, download it here

[adbdkb]

Hi,  anyone has any pointers / solutions?

Appreciate any help to get this resolved.

Thanks

[spruett]

I was able to get this to work.

KafkaClient {
	org.apache.kafka.common.security.plain.PlainLoginModule required 
	username="XXXXX" 
	password="XXXXX";
	com.sun.security.auth.module.Krb5LoginModule required	
	security.protocol=SASL_SSL
	ssl.endpoint.identification.algorithm=https 
	sasl.mechanism=PLAIN;
};

Unfortunately, I then ran into further issues with Kerberos.  In order to supply your JAAS config path, you have to select the "Use kerberos authentication", and we are not using Kerberos.

[vboppudisfx]

Any updates on how to resolve this error?

I am also in similar situation. I am trying to connect to concluent cloud and running into issues while using SASL_SSL security provided by client. 

Does anyone has experience connecting to Confluent could ? I am able to connect using python script but running into issues using Talend . here is sample python script i used.

------

from confluent_kafka import Producer
def delivery_report(err, msg):
""" Called once for each message produced to indicate delivery result.
Triggered by poll() or flush(). """
if err is not None:
print('Message delivery failed: {}'.format(err))
else:
print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition()))

p = Producer({
'bootstrap.servers': 'SASL_SSL://servernameort',
'sasl.mechanism': 'PLAIN',
'security.protocol': 'SASL_SSL',
'sasl.username': 'usename',
'sasl.password': 'password',

'ssl.ca.location':'/Users/certifi/cacert.pem' # not provided by confluent
})

p.produce('test_topic', value='Test---', callback=delivery_report)

 ----------

[spruett]

I was not able to make this work with the current Talend components, because they assume the use of Kerberos, which we do not use.  I ended up making my own components.

[vboppudisfx]

Thanks for your reply @spruett  

Can you share details on how to create new components to make it work ?

Thank  you and appreciate your help.  

[spruett · Accepted solution]

There is ample documentation on building custom Talend components.  You might want to note that there are two was to do this: the old way, which the Kafka components use, and the new way called the Talend Component Kit.

What I did was to start by copying the existing tKafka... Talend component folders that I found here:

[Install Dir]\studio\plugins\org.talend.designer.components.bigdata_7.2.1.20190619_1114\components

I then began renaming folders and files, and customized the code according to my needs.  I added another checkbox in front of the "Use kerberos authentication" to "Use SASL/PLAIN" authentication so I could pick that and ignore the kerberos stuff.  So that I didn't even need a JAAS config file, I just put fields for Username and Password into the component properties form.  Important properties that would need to be set for this authentication would be: "security.protocol", "sasl.mechanism", and "sasl.jaas.config".  In the code, they might look something like this:

<%=cid%>_kafkaProperties.setProperty("security.protocol", "SASL_SSL");
<%=cid%>_kafkaProperties.setProperty("sasl.mechanism", "PLAIN");
<%=cid%>_kafkaProperties.put("sasl.jaas.config", "org.apache.kafka.common.security.plain.PlainLoginModule required username=\"" + <%=tLWKafkaInputUtil.getSaslUsername()%> + "\" password=\"" + <%=tLWKafkaInputUtil.getSaslPassword()%> + "\";" );

[vboppudisfx]

Thank you @spruett  . Now i am able to establish connection to confluent kafka.