Talend Spark Stream Kerberos enabled Job not working with error IllegalStateException: Expected SaslMessage not have SASL enabled

[wei_guo]

A Talend Spark Stream Job configured with Yarn cluster mode and Kerberos enabled is encountering issues and failing to execute, presenting the following errors: 

YarnClusterScheduler- Lost executor 3 on me-worker1.xxx.co.id: Unable to create executor due to Unable to register with external shuffle server due to : java.lang.IllegalStateException: Expected SaslMessage, received something else (maybe your client does not have SASL enabled?)

jaas.conf content =
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/tmp/adm.keytab"
principal="cld_adm@XXX.CO.ID"
doNotPrompt=true;
};

KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/tmp/adm.keytab"
principal="cld_adm@XXX.CO.ID"
doNotPrompt=true;
};

Cause

The external shuffle service within YARN is configured to mandate SASL authentication; however, the Spark executor is either improperly configured to use SASL or is transmitting an incompatible message.

Common causes includes:

1. The absence or inaccuracy of SASL configuration in either spark-defaults.conf or YARN's configuration file (yarn-site.xml).
2. A version discrepancy between Spark and YARN, where the shuffle service necessitates a particular SASL protocol unsupported by the Spark client.
3. Improper security configurations, including the omission of Kerberos credentials or incorrect settings for spark.authenticate and associated properties.

This error results in the executor's failure to register with the shuffle service, prompting the YarnClusterScheduler to mark it as lost.

Resolution

Ensure that the external shuffle service is enabled, and that the SASL settings are in accordance with YARN's configurations:

spark.authenticate true
( #not necessary 
spark.network.crypto.enabled true
spark.network.crypto.saslFallback true
)

Environment

• Talend Data Integration