Error in HDFSOutput outside Open Studio in Linux/cron

[SJ3]

Hi,

I am pulling some log files and putting it under a Hadoop folder. I can run this job in the big data open studio but when I try to run it in cron/linux, it throws following error:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addDeprecations([Lorg/apache/hadoop/conf/Configuration$DeprecationDeltaV
at org.apache.hadoop.hdfs.HdfsConfiguration.addDeprecatedKeys(HdfsConfiguration.java:66)

I thought may be this was because of different Hadoop versions so I checked I have following HDFS version:

HDFS

2.7.1.2.4

then I tried to change the versions of Hadoop under HDFSOutput component

But I couldn't find the same version. Now I am not sure how I can make this job work on linux/cron. Any suggestions?

Thanks!

SJ

[SJ3]

I guess no one had this issue before 

[Anonymous]

Hello,

The bigdata job works perfectly fine in studio but not in cmd(.sh file)? Are you using tHDFSconnection and tHBASEconnection in the same Job? Could you please indicate on which talend build version you got this issue?

Best regards

Sabrina

[SJ3 · Accepted solution]

I am using version 6.3.1. And I was able to solve this problem by not using any context variable under HDFS components. I have also used same HDFS connections

in tHDFSOutput and tHDFSList, which I have created under Hadoop Cluster - Metadata options. Thanks for your help though Sabrina!

SJ