Spark as YARN Client

[Anonymous]

Hello everyone,
I'm currently managing a Hortonworks HDP cluster and I'm testing Talend Big Data licence evaluation.
I would like to test the Spark components with my cluster.
When I set in the Spark connection the "YARN Client" client, the component is asking me a .jar as configuration file.
Do you know where I can find/generate such a jar file?
Thank you.
Regards,
Orlando

[Anonymous]

Hi Orlando,

When I set in the Spark connection the "YARN Client" client, the component is asking me a .jar as configuration file.

For this field, you need to browse to the local jar file that contains the configuration of the Yarn service to be used.
The configuration files that must present in this jar file are: core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml.
 Note: You can press F1 (focus your mouse on tSparkConnection)to find the related component reference of tSparkConnection.

Best regards
Sabrina

[Anonymous]

Hi,
Even i have tried building the jar file but its not working for me. Any luck for you?
If so could you please help me how to build the jar file??
I have these 4 xml files on the edge node. i packaged them in jar file. But its not working. Anything else i have to do?
Regards,
Shanker

[Anonymous]

Hi Shanker,
I didn't manage to get it work. I'm managing a Hortonwork's distribution of Hadoop.
I had to change ${hdp.version} to 2.2.4.2-2 (or your version here) in the mapred-site.xml.
Now my job is deployed but failed after a few seconds with this error :
Exception in thread "main" java.lang.IllegalArgumentException: Invalid ContainerId: container_e157_1443206132389_0129_01_000001
...
Caused by: java.lang.NumberFormatException: For input string: "e157"
...
... 11 more
I'm still looking for a solution.
Cheers,
Orlando