topic Re: MapReduce job is failing... in Talend Studio

MapReduce job is failing...

Anonymous — Sat, 16 Nov 2024 11:59:50 GMT

ok, finally I was able to design MapReduce job and was also able to get configuration right to run it on hadoop. But for some reason, the job is failing for some reason.
Job is really simple - I've Invoice file stored on hdfs of size 1.5 GB approx, all I'm trying to do is make a copy of it...
tHDFSInput ----> tMap ----> tHDFSOutput

Job starts execution and when job is submitted to mapreduce - I get following error - Can someone please help me understand this issue.

Starting job MapRSample01 at 17:18 17/06/2013.

13/06/17 17:18:14 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/06/17 17:18:15 INFO mapred.FileInputFormat: Total input paths to process : 1
13/06/17 17:18:15 WARN conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
13/06/17 17:18:15 INFO mapred.JobClient: Running job: job_201306171252_0006
13/06/17 17:18:16 INFO mapred.JobClient: map 0% reduce 0%
13/06/17 17:18:26 INFO mapred.JobClient: Task Id : attempt_201306171252_0006_m_000000_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:389)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
13/06/17 17:18:26 INFO mapred.JobClient: Task Id : attempt_201306171252_0006_m_000001_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:389)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
13/06/17 17:18:26 INFO mapred.JobClient: Task Id : attempt_201306171252_0006_m_000002_0, Status : FAILED
13/06/17 17:18:47 INFO mapred.JobClient: Job Counters
13/06/17 17:18:47 INFO mapred.JobClient: Failed map tasks=1
13/06/17 17:18:47 INFO mapred.JobClient: Launched map tasks=16
13/06/17 17:18:47 INFO mapred.JobClient: Data-local map tasks=16
13/06/17 17:18:47 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=102806
13/06/17 17:18:47 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=0
13/06/17 17:18:47 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/06/17 17:18:47 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/06/17 17:18:47 INFO mapred.JobClient: Job Failed: NA
13/06/17 17:18:47 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs (authIMPLE) cause:java.io.IOException: Job failed!
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1323)
at talenddemosjava.maprsample01_1_0.MapRSample01$1.run(MapRSample01.java:2415)
at talenddemosjava.maprsample01_1_0.MapRSample01$1.run(MapRSample01.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at talenddemosjava.maprsample01_1_0.MapRSample01.tHDFSInput_1Process(MapRSample01.java:2369)
at talenddemosjava.maprsample01_1_0.MapRSample01.run(MapRSample01.java:2572)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at talenddemosjava.maprsample01_1_0.MapRSample01.runJobInTOS(MapRSample01.java:2494)
at talenddemosjava.maprsample01_1_0.MapRSample01.main(MapRSample01.java:2465)
Job MapRSample01 ended at 17:18 17/06/2013.

Re: MapReduce job is failing...

Anonymous — Mon, 17 Jun 2013 15:30:41 GMT

Hello,
Hard to define the cause here. Could you please have a look at the job which fails on the jobtracker, browsing http://HOSTNAME:50030? Maybe you'll find additional logs.
I see you are executing the job as 'hdfs'. Are the data you are trying to read owned by 'hdfs'?

Re: MapReduce job is failing...

Anonymous — Mon, 17 Jun 2013 17:59:22 GMT

yes data is owned by hdfs... looking for log on hdfs. Will post it if I've any questions regarding it.
Thanks,
Nishad Joshi

Re: MapReduce job is failing...

Anonymous — Tue, 18 Jun 2013 12:35:34 GMT

Looks like everything is good as far as job setting is concerned - But now I'm getting one error -
Starting job MapRSample02 at 16:58 18/06/2013.

13/06/18 16:58:27 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/06/18 16:58:28 INFO mapred.JobClient: Cleaning up the staging area hdfs://CentOS1:8020/user/hdfs/.staging/job_201306171252_0030
13/06/18 16:58:28 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs (authIMPLE) causerg.apache.hadoop.mapred.InvalidJobConfException: Output directory not set in JobConf.
13/06/18 16:58:28 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs (authIMPLE) causerg.apache.hadoop.mapred.InvalidJobConfException: Output directory not set in JobConf.
org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set in JobConf.
at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:103)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:937)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)

Can someone please help me understand this problem?

Re: MapReduce job is failing...

Anonymous — Tue, 18 Jun 2013 12:47:04 GMT

"Output directory not set in JobConf" - is this something I should setup on hadoop cluster?

Re: MapReduce job is failing...

Anonymous — Tue, 18 Jun 2013 15:05:39 GMT

No, it deals with the output folder in the tHDFSOutput component. Please be sure it's not empty.

Re: MapReduce job is failing...

Anonymous — Wed, 19 Jun 2013 18:48:34 GMT

Unlike tHDFSInput, tHDFSOutput component does not have option to mention output file. I've attached screen shot - am I missing something here?
Ideally, if I need to design MapReduce job I should have the ability to mention the output file on hdfs.
detailed explanation on this would really help.... thank you very much for all the help.

Re: MapReduce job is failing...

Anonymous — Fri, 21 Jun 2013 06:09:25 GMT

Hi,
can anyone please help me with this?
Thanks,
Nishad Joshi.

Re: MapReduce job is failing...

Anonymous — Fri, 21 Jun 2013 16:25:43 GMT

Hello,
A MapReduce job is going to generate as much files as there are reducers in your map reduce job. These files are going to be named part-r-00000, part-r-00001, part-r-00002, ... That's why you can't set a file, but a folder, in which the files are going to be written.
Is it clearer?
HTH.

Re: MapReduce job is failing...

Anonymous — Sat, 22 Jun 2013 15:12:39 GMT

Thanks rdubois. Ok, makes sense... then If I've a job which is simply trying to make a copy of existing file...
tHDFSInput ---- tMap ----- tHDFSOutput
The job should create part0001, part0002, .... etc. on output folder. But for some reason the job is failing again and again.... tried all the possible combination with settings.
When I verified log carefully - I found following two errors.
1) Error in configuring object
2) ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs (authIMPLE) cause:java.io.IOException: Job failed!
Is this the actual problem with the job? If you can help me understand the problem with this log(pasted above), it will be a great help.

Re: MapReduce job is failing...

Anonymous — Mon, 24 Jun 2013 13:10:23 GMT

The error you send is not very meaningful. Let's try to troubleshoot. Does the output folder already exist before you execute the job? If it does, have you selected "Overwrite" in the tHDFSOutput drop down list?
Has the user which executes the MapReduce job the permissions to write into the output folder on hdfs?

Re: MapReduce job is failing...

Anonymous — Mon, 24 Jun 2013 13:18:36 GMT

Does the output folder already exist before you execute the job?
Yes
If it does, have you selected "Overwrite" in the tHDFSOutput drop down list?
Yes
Has the user which executes the MapReduce job the permissions to write into the output folder on hdfs?
Not sure. I've Talend installed on my Master node (CentOS1) for which "root" is the admin user. When I submit MapReduce job designed on this machine does this mean "The user" is root? coz on Hadoop side - hdfs is the owner

Re: MapReduce job is failing...

Anonymous — Mon, 24 Jun 2013 13:33:52 GMT

If you try to write your result into /user/username/output within HDFS, try a hadoop fs -ls /user/username to know what are the permissions and the owner of this folder. Finally, be sure the user which executes the job has the correct permissions according to what you have seen before.

Re: MapReduce job is failing...

Anonymous — Mon, 24 Jun 2013 14:13:21 GMT

This is what I got

bash-4.1$ hadoop fs -ls
Found 3 items
drwx------ - hdfs supergroup 0 2013-06-24 18:17 .staging
drwxr-xr-x - hdfs supergroup 0 2013-06-24 17:53 oozie-oozi
drwxr-xr-x - hdfs supergroup 0 2013-06-24 18:17 output

Re: MapReduce job is failing...

Anonymous — Tue, 25 Jun 2013 08:41:18 GMT

Which username did you set in your username parameter in the hadoop configuration tab?

Re: MapReduce job is failing...

Anonymous — Tue, 25 Jun 2013 09:07:55 GMT

hdfs.. please see the attached screen shot.

Re: MapReduce job is failing...

Anonymous — Tue, 25 Jun 2013 11:12:43 GMT

Hi rdubois,
Just to see if my hadoop configurations are correct, I tried executing WordCount program on it. It executed without any issues.

Re: MapReduce job is failing...

Anonymous — Tue, 25 Jun 2013 12:46:51 GMT

ok. Could you please send the whole exception you get?

Re: MapReduce job is failing...

Anonymous — Tue, 25 Jun 2013 12:59:39 GMT

Starting job MapRSample01 at 17:23 25/06/2013.

13/06/25 17:23:57 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/06/25 17:23:58 INFO mapred.FileInputFormat: Total input paths to process : 1
13/06/25 17:23:58 WARN conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
13/06/25 17:23:58 INFO mapred.JobClient: Running job: job_201306241745_0021
13/06/25 17:23:59 INFO mapred.JobClient: map 0% reduce 0%
13/06/25 17:24:09 INFO mapred.JobClient: Task Id : attempt_201306241745_0021_m_000002_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130) at org.apache

To see the whole post, download it here
OriginalPost.pdf

Re: MapReduce job is failing...

Anonymous — Tue, 25 Jun 2013 13:34:38 GMT

Ok, now could you please find the log of the failed tasks in your jobtracker?