Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

Announcements
Discover how organizations are unlocking new revenue streams: Watch here
cancel
Showing results for 
Search instead for 
Did you mean: 
KarthikGs
Creator
Creator

How to set the number of mappers in Talend

Hi,
I'm setting the number of mappers in big data batch using specific JVM arguments. by passing this parameter "-Dmapred.map.tasks=200".
When calling this big data job from standard job, the number of mappers running are shown as "2".
Is there a way to set the number of mappers in my scenario?
I'm retrieving data from Hbase table BTW.
Thanks
Labels (2)
5 Replies
Anonymous
Not applicable

Hi,
Have you tried to set this when launching the jobs with the "hadoop" command to see if it works?
Best regards
Sabrina
KarthikGs
Creator
Creator
Author

Hi,
Have you tried to set this when launching the jobs with the "hadoop" command to see if it works?
Best regards
Sabrina

I'm running my job from TAC. So before loading my job in to talend console i have set the desirable mappers while designing my application itself. Is there any way to increase the nmuber of mappers.
vapukov
Master II
Master II

The default number of map tasks per job. Ignored when mapred.job.tracker is "local".
and because:
Resolved it, TAC and local server were on the same cluster.
Added this property in JVM parameters:
-Djava.security.auth.login.config =/opt/mapr/conf/mapr.login.conf


default value - exactly 2
KarthikGs
Creator
Creator
Author

So how can we increase the number of mappers?
KarthikGs
Creator
Creator
Author

Hi,
Have you tried to set this when launching the jobs with the "hadoop" command to see if it works?
Best regards
Sabrina

Any findings on how to increase the mappers in my mapreduce job in big data batch? I have added 
-Dmapred.map.tasks=200
  -Dmapred.reduce.tasks=50.
But no effect. I cannot rise a ticket for product support for this small issue.