Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Announcements
Note: You may notice some temporary visual or styling issues in the Community. Our vendor is actively investigating.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable
‎2013-06-17 09:19 AM

MapReduce job is failing...

ok, finally I was able to design MapReduce job and was also able to get configuration right to run it on hadoop. But for some reason, the job is failing for some reason.
Job is really simple - I've Invoice file stored on hdfs of size 1.5 GB approx, all I'm trying to do is make a copy of it...
tHDFSInput ----> tMap ----> tHDFSOutput

Job starts execution and when job is submitted to mapreduce - I get following error - Can someone please help me understand this issue.

Starting job MapRSample01 at 17:18 17/06/2013.

13/06/17 17:18:14 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/06/17 17:18:15 INFO mapred.FileInputFormat: Total input paths to process : 1
13/06/17 17:18:15 WARN conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
13/06/17 17:18:15 INFO mapred.JobClient: Running job: job_201306171252_0006
13/06/17 17:18:16 INFO mapred.JobClient: map 0% reduce 0%
13/06/17 17:18:26 INFO mapred.JobClient: Task Id : attempt_201306171252_0006_m_000000_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:389)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
13/06/17 17:18:26 INFO mapred.JobClient: Task Id : attempt_201306171252_0006_m_000001_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:389)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
at org.apache.hadoop.mapred.Child$4.run(Child.java:270)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.ja
13/06/17 17:18:26 INFO mapred.JobClient: Task Id : attempt_201306171252_0006_m_000002_0, Status : FAILED
13/06/17 17:18:47 INFO mapred.JobClient: Job Counters
13/06/17 17:18:47 INFO mapred.JobClient: Failed map tasks=1
13/06/17 17:18:47 INFO mapred.JobClient: Launched map tasks=16
13/06/17 17:18:47 INFO mapred.JobClient: Data-local map tasks=16
13/06/17 17:18:47 INFO mapred.JobClient: Total time spent by all maps in occupied slots (ms)=102806
13/06/17 17:18:47 INFO mapred.JobClient: Total time spent by all reduces in occupied slots (ms)=0
13/06/17 17:18:47 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/06/17 17:18:47 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/06/17 17:18:47 INFO mapred.JobClient: Job Failed: NA
13/06/17 17:18:47 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs (auth0683p000009M9p6.pngIMPLE) cause:java.io.IOException: Job failed!
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1323)
at talenddemosjava.maprsample01_1_0.MapRSample01$1.run(MapRSample01.java:2415)
at talenddemosjava.maprsample01_1_0.MapRSample01$1.run(MapRSample01.java:1)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
at talenddemosjava.maprsample01_1_0.MapRSample01.tHDFSInput_1Process(MapRSample01.java:2369)
at talenddemosjava.maprsample01_1_0.MapRSample01.run(MapRSample01.java:2572)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at talenddemosjava.maprsample01_1_0.MapRSample01.runJobInTOS(MapRSample01.java:2494)
at talenddemosjava.maprsample01_1_0.MapRSample01.main(MapRSample01.java:2465)
Job MapRSample01 ended at 17:18 17/06/2013.

Labels (3)
Labels
1,471 Views
0 Likes
24 Replies
Anonymous
Not applicable
‎2013-06-17 11:30 AM
Author

Hello,
Hard to define the cause here. Could you please have a look at the job which fails on the jobtracker, browsing http://HOSTNAME:50030? Maybe you'll find additional logs.
I see you are executing the job as 'hdfs'. Are the data you are trying to read owned by 'hdfs'?
892 Views
0 Likes
Anonymous
Not applicable
‎2013-06-17 01:59 PM
Author

yes data is owned by hdfs... looking for log on hdfs. Will post it if I've any questions regarding it.
Thanks,
Nishad Joshi
892 Views
0 Likes
Anonymous
Not applicable
‎2013-06-18 08:35 AM
Author

Looks like everything is good as far as job setting is concerned - But now I'm getting one error -
Starting job MapRSample02 at 16:58 18/06/2013.

13/06/18 16:58:27 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13/06/18 16:58:28 INFO mapred.JobClient: Cleaning up the staging area hdfs://CentOS1:8020/user/hdfs/.staging/job_201306171252_0030
13/06/18 16:58:28 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs (auth0683p000009M9p6.pngIMPLE) cause0683p000009MA5A.pngrg.apache.hadoop.mapred.InvalidJobConfException: Output directory not set in JobConf.
13/06/18 16:58:28 ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs (auth0683p000009M9p6.pngIMPLE) cause0683p000009MA5A.pngrg.apache.hadoop.mapred.InvalidJobConfException: Output directory not set in JobConf.
org.apache.hadoop.mapred.InvalidJobConfException: Output directory not set in JobConf.
at org.apache.hadoop.mapred.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:103)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:937)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:896)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
Can someone please help me understand this problem?
892 Views
0 Likes
Anonymous
Not applicable
‎2013-06-18 08:47 AM
Author

"Output directory not set in JobConf" - is this something I should setup on hadoop cluster?
892 Views
0 Likes
Anonymous
Not applicable
‎2013-06-18 11:05 AM
Author

No, it deals with the output folder in the tHDFSOutput component. Please be sure it's not empty.
892 Views
0 Likes
Anonymous
Not applicable
‎2013-06-19 02:48 PM
Author

Unlike tHDFSInput, tHDFSOutput component does not have option to mention output file. I've attached screen shot - am I missing something here?
Ideally, if I need to design MapReduce job I should have the ability to mention the output file on hdfs.
detailed explanation on this would really help.... thank you very much for all the help.
0683p000009MAEN.jpg 0683p000009MAIR.jpg
892 Views
0 Likes
Anonymous
Not applicable
‎2013-06-21 02:09 AM
Author

Hi,
can anyone please help me with this?
Thanks,
Nishad Joshi.
892 Views
0 Likes
Anonymous
Not applicable
‎2013-06-21 12:25 PM
Author

Hello,
A MapReduce job is going to generate as much files as there are reducers in your map reduce job. These files are going to be named part-r-00000, part-r-00001, part-r-00002, ... That's why you can't set a file, but a folder, in which the files are going to be written.
Is it clearer?
HTH.
892 Views
0 Likes
Anonymous
Not applicable
‎2013-06-22 11:12 AM
Author

Thanks rdubois. Ok, makes sense... then If I've a job which is simply trying to make a copy of existing file...
tHDFSInput ---- tMap ----- tHDFSOutput
The job should create part0001, part0002, .... etc. on output folder. But for some reason the job is failing again and again.... tried all the possible combination with settings.
When I verified log carefully - I found following two errors.
1) Error in configuring object
2) ERROR security.UserGroupInformation: PriviledgedActionException as:hdfs (auth0683p000009M9p6.pngIMPLE) cause:java.io.IOException: Job failed!
Is this the actual problem with the job? If you can help me understand the problem with this log(pasted above), it will be a great help.
892 Views
0 Likes