topic running same map reduce job called from different project in Talend Studio

running same map reduce job called from different project

rohit_sharma21 — Thu, 04 Apr 2019 18:35:58 GMT

HI,

I have a scenario where I have a generic map reduce job that is getting called from different project at the same time. but job is getting failed due to error "org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://nameservice1/tmp/test_abc_out_job/tMROutput_tMap_11/out_ww already exists"

as I have 4 jobs that will call this one MR at same time is causing this issue. can someone help me to fix this,

Thanks

Rohit

Re: running same map reduce job called from different project

Anonymous — Thu, 04 Apr 2019 19:02:58 GMT

Hi,

Why don't you add an additional directory at end based on the job you are calling? This will make sure that you can run 4 jobs in parallel since directory structure will be different for each job.

The last part of the directory can be populated from a context variable.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂