running same map reduce job called from different project

[rohit_sharma21]

HI,

I have a scenario where I have a generic map reduce job that is getting called from different project at the same time. but job is getting failed due to error "org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://nameservice1/tmp/test_abc_out_job/tMROutput_tMap_11/out_ww already exists"

as I have 4 jobs that will call this one MR at same time is causing this issue. can someone help me to fix this,

Thanks

Rohit

[Anonymous · Accepted solution]

Hi,

    Why don't you add an additional directory at end based on the job you are calling? This will make sure that you can run 4 jobs in parallel since directory structure will be different for each job.

    The last part of the directory can be populated from a context variable.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂