topic HDFS HA in tHDFS components in Talend Studio

HDFS HA in tHDFS components

Anonymous — Sat, 16 Nov 2024 11:04:13 GMT

Hi everyone,
I'm using a Hadoop cluster with a HDFS in HA mode set up.
How could I set up the tHDFS*components to always point to the active namenode ?
I saw this page https://help.talend.com/search/all?query=Enabling+the+HDFS+High+Availability+feature+in+the+Studio&content-lang=en but it describes how to configure the tHive components and it doesn't seem to work with tHDFS components. What am I missing? Do I need to change any configuration in the metadata of my Hadoop cluster in the Studio ?
Thank you.
Orlando

Re: HDFS HA in tHDFS components

Anonymous — Wed, 23 Sep 2015 17:00:39 GMT

Hi,
The solution is the same than Hive but for the Namenode URI, you should define "hdfs://nameservice1".
Can you please let us know if that fixes the issue ?
Cheers,
Rémy.

Re: HDFS HA in tHDFS components

Anonymous — Thu, 24 Sep 2015 08:07:01 GMT

Hi Rémy,
You're right. The solution is the same than Hive.
Actually, I found out that my problem was coming from my Oozie configuration. I got an error saying that Hostname "nameservice1" was not found the while deploying to Oozie.
I had to add the parameters (dfs.nameservice, ...) in the hadoop properties of the cluster in the metadata.
This solution is working with the new Talend Big Data 6.0.
Thanks for your help.
Regards,
Orlando