topic Spark as YARN Client in Talend Studio

Spark as YARN Client

Anonymous — Sat, 16 Nov 2024 11:16:44 GMT

Hello everyone,
I'm currently managing a Hortonworks HDP cluster and I'm testing Talend Big Data licence evaluation.
I would like to test the Spark components with my cluster.
When I set in the Spark connection the "YARN Client" client, the component is asking me a .jar as configuration file.
Do you know where I can find/generate such a jar file?
Thank you.
Regards,
Orlando

Re: Spark as YARN Client

Anonymous — Thu, 26 Mar 2015 08:02:06 GMT

Hi Orlando,

When I set in the Spark connection the "YARN Client" client, the component is asking me a .jar as configuration file.

For this field, you need to browse to the local jar file that contains the configuration of the Yarn service to be used.
The configuration files that must present in this jar file are: core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml.
Note: You can press F1 (focus your mouse on tSparkConnection)to find the related component reference of tSparkConnection.

Best regards
Sabrina

Re: Spark as YARN Client

Anonymous — Thu, 16 Jul 2015 22:48:37 GMT

Hi,
Even i have tried building the jar file but its not working for me. Any luck for you?
If so could you please help me how to build the jar file??
I have these 4 xml files on the edge node. i packaged them in jar file. But its not working. Anything else i have to do?
Regards,
Shanker

Re: Spark as YARN Client

Anonymous — Mon, 28 Sep 2015 15:08:17 GMT

Hi Shanker,
I didn't manage to get it work. I'm managing a Hortonwork's distribution of Hadoop.
I had to change ${hdp.version} to 2.2.4.2-2 (or your version here) in the mapred-site.xml.
Now my job is deployed but failed after a few seconds with this error :
Exception in thread "main" java.lang.IllegalArgumentException: Invalid ContainerId: container_e157_1443206132389_0129_01_000001
...
Caused by: java.lang.NumberFormatException: For input string: "e157"
...
... 11 more
I'm still looking for a solution.
Cheers,
Orlando