topic Re: Automatic Namenode failover - When using tHDFS connection in Talend Studio

Automatic Namenode failover - When using tHDFS connection

Anonymous — Sat, 16 Nov 2024 02:34:44 GMT

Friends,

I am new to Talend, so might be a basic question. I am using Talend 7.0.1

When there is a failover of the NameNode server, I am trying to understand the possibilities on how to make the Talend job still work (with out any changes). To make this possible, How do we define the HDFS connection? Should we do any Parameters setting? What configuration to be taken care?

Thanks for taking time, to help me understand the solution

Regards

Giri

Re: Automatic Namenode failover - When using tHDFS connection

Anonymous — Tue, 28 Apr 2020 05:28:55 GMT

Hi Giri
If you have a connectivity issue to HDFS, you can't do any operations related to HDFS. Do you want the job continue to perform other business processing even though there is a connectivity failover on tHDFSConnection?

Regards
Shong

Re: Automatic Namenode failover - When using tHDFS connection

Anonymous — Tue, 28 Apr 2020 07:08:10 GMT

Hi Shong.

Thanks for your reply.

It is not a connectivity issue. tHDFS is configured with primary/active namenode. When there is a swtich (Cloudera) on Namenode server passive/secondary becoming active node. We have to change the configuration of the Talend job to point to the new name node.

I am trying to understand, if there is a way where we can build the Talend job so that, we dont have to change the configuration every time there is a change in the Namenode.

Regards

Giri

Re: Automatic Namenode failover - When using tHDFS connection

Anonymous — Tue, 28 Apr 2020 09:22:58 GMT

yes, you should configure the parameter with context variables, so that you can load the value of variables from file/DB in the beginning of job, or specify the value of variable before you execute the job script out of studio, don't need to open studio and modify the job.

Re: Automatic Namenode failover - When using tHDFS connection

Anonymous — Tue, 28 Apr 2020 09:49:03 GMT

We are already performing this. Every time there is a name node change, we have to modify this param entries. This option is happening frequently, so it is not wise to change the file, where ever there is a switch. I am trying to understand if there is any other option, where we dont have to change anything in the talend job/param/config file.