topic Re: Oozie and Namenode HA in Talend Studio

Oozie and Namenode HA

Anonymous — Sat, 16 Nov 2024 10:59:49 GMT

Hi,
I am using a HDP cluster with a HDFS set in High Availability.
I figured out how to configure HDFS and Hive components (in a previous post: thank you again for this)
But how could I set HA properties (dfs.nameservice,etc.) when I deploy it with Oozie?
Folowing to this page : https://help.talend.com/search/all?query=Setting+reusable+Hadoop+properties&content-lang=en, I need to add those properties in the Hadoop cluster metadata (setp 4 in the linked tutorial), but I think it's not possible with Talend 5.6. Am I right?
Maybe I should set up this configuration somewhere else in Talend 5.6?
Thank you for any help on this.
Bests,
Orlando

Re: Oozie and Namenode HA

amarouni — Tue, 13 Oct 2015 09:18:22 GMT

Hi,
You can set the HA Hadoop properties on the Oozie connection directly. Start by creating a Hadoop connection (Metadata -> Hadoop Cluster), then add an Oozie service. The Oozie wizard will let you add any Hadoop properties to use with Oozie (check the attached screenshot).

Best Regards,

Re: Oozie and Namenode HA

Anonymous — Wed, 14 Oct 2015 10:09:41 GMT

Hi amarouni,
Thank you for your answer.
I tried this solution, but the Oozie namenode setting inside the job keep using the namenode URL set in the cluster metadata (not the Oozie metadata which is the URL of the Oozie endpoint). I tried to set the namenode URL to the "nameservice", but I can't deploy anything with the error "unknown host", which is normal since I can't set the properties (dfs.nameservice= etc.) in the cluster.
Orlando