Do not input private or sensitive data. View Qlik Privacy & Cookie Policy.
Skip to main content

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Announcements
Talend Cloud AWS EU Scheduled Outage: Starting Tues 26 May 21:00 CEST with expected completion Wed 27 May 01:00 CEST
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Anonymous
Not applicable
‎2013-05-07 03:55 AM

Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Hi,
I am very new to Talend. Is there a way to schedule Hadoop Jobs (e.g. HIVE, PIG etc.) using OOZIE plugin.
Can you please show me a example.
Regards,
Shouvanik

Labels (1)
Labels
2,725 Views
0 Likes
49 Replies
Anonymous
Not applicable
‎2013-05-08 03:58 AM
Author

Hi,
I suspect that the hadoop version running locally in my machine is of a incompatible version with respect to the version required by Talend.
Can you please tell me what Hadoop version is compatible with Talend Hadoop client version?

You can check the online guide Supported+systems+and+databases and Supported+Hadoop+distribution+versions.
Best regards
Sabrina
549 Views
0 Likes
Anonymous
Not applicable
‎2013-05-08 09:17 AM
Author

Thanks.
But how to include them inside Talend? I am confused here.
Regards,
Shouvanik
549 Views
0 Likes
Anonymous
Not applicable
‎2013-05-13 05:51 AM
Author

Hi,
I am able to schedule and run jobs using OOZIE plugin of Talend.
The main problem was finding out which version of hadoop is compatible with which version of oozie.
Now, I am having different outcomes with different jobs using Talend
1. A simple job "HDFSJob" which has a big data component called tHDFSConnection_1
This job runs smoothly and getting succeeded.

2. A job "test_job" which prints a message "Hello World"
This job runs, but after sometime, I get a error in Talend console which is
Deploying job to Hadoop...
Deployment complete!
Job is running ...
Job killed!
Main class , exit code
Job failed, error message, exit code ]
And when I look at oozie log, i get the following output
2013-05-13 14:18:04,586 INFO ActionStartXCommand:539 - USER GROUP TOKEN[] APP JOB ACTION Start action with user-retry state : userRetryCount , userRetryMax , userRetryInterval
2013-05-13 14:18:04,593 WARN ActionStartXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION Action status=DONE
2013-05-13 14:18:04,594 WARN ActionStartXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION Action updated in DB!
2013-05-13 14:18:04,807 INFO ActionStartXCommand:539 - USER GROUP TOKEN[] APP JOB ACTION Start action with user-retry state : userRetryCount , userRetryMax , userRetryInterval
2013-05-13 14:18:05,100 WARN JavaActionExecutor:542 - USER GROUP TOKEN[] APP JOB ACTION credentials is null for the action
2013-05-13 14:18:05,418 WARN JobClient:667 - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2013-05-13 14:18:05,486 WARN JobClient:784 - No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
2013-05-13 14:18:05,799 INFO JavaActionExecutor:539 - USER GROUP TOKEN[] APP JOB ACTION checking action, external ID status
2013-05-13 14:18:05,810 WARN ActionStartXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION Action status=RUNNING
2013-05-13 14:18:05,811 WARN ActionStartXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION Action updated in DB!
2013-05-13 14:18:15,524 INFO StatusTransitService$StatusTransitRunnable:539 - USER GROUP Acquired lock for
2013-05-13 14:18:15,529 INFO StatusTransitService$StatusTransitRunnable:539 - USER GROUP Running coordinator status service from last instance time = 2013-05-13T08:47Z
2013-05-13 14:18:15,531 INFO StatusTransitService$StatusTransitRunnable:539 - USER GROUP Running bundle status service from last instance time = 2013-05-13T08:47Z
2013-05-13 14:18:15,532 INFO StatusTransitService$StatusTransitRunnable:539 - USER GROUP Released lock for
2013-05-13 14:18:16,431 INFO PauseTransitService:539 - USER GROUP Acquired lock for
2013-05-13 14:18:16,455 INFO PauseTransitService:539 - USER GROUP Released lock for
2013-05-13 14:18:26,624 INFO CallbackServlet:539 - USER GROUP TOKEN APP JOB ACTION callback for action
2013-05-13 14:18:26,773 INFO JavaActionExecutor:539 - USER GROUP TOKEN[] APP JOB ACTION action completed, external ID
2013-05-13 14:18:26,824 WARN JavaActionExecutor:542 - USER GROUP TOKEN[] APP JOB ACTION Launcher ERROR, reason: Main class , exit code
2013-05-13 14:18:26,922 INFO ActionEndXCommand:539 - USER GROUP TOKEN[] APP JOB ACTION ERROR is considered as FAILED for SLA
2013-05-13 14:18:26,951 INFO ActionStartXCommand:539 - USER GROUP TOKEN[] APP JOB ACTION Start action with user-retry state : userRetryCount , userRetryMax , userRetryInterval
2013-05-13 14:18:26,951 WARN ActionStartXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION Action status=DONE
2013-05-13 14:18:26,952 WARN ActionStartXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION Action updated in DB!
2013-05-13 14:18:27,012 WARN CoordActionUpdateXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION E1100: Command precondition does not hold before execution, , Error Code: E1100


And when I check the job tracker log, I get the following log
2013-05-13 14:18:05,367 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Creating password for identifier: owner=root, renewer=mr token, realUser=root, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0
2013-05-13 14:18:05,387 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Creating password for identifier: owner=root, renewer=mr token, realUser=root, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0
2013-05-13 14:18:05,734 INFO org.apache.hadoop.mapred.JobInProgress: job_201305131138_0013: nMaps=1 nReduces=0 max=-1
2013-05-13 14:18:05,734 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201305131138_0013
2013-05-13 14:18:05,735 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201305131138_0013
2013-05-13 14:18:05,742 INFO org.apache.hadoop.mapred.JobTracker: Job job_201305131138_0013 added successfully for user 'root' to queue 'default'
2013-05-13 14:18:05,743 INFO org.apache.hadoop.mapred.AuditLogger: USER=root IP=127.0.0.1 OPERATION=SUBMIT_JOB TARGET=job_201305131138_0013 RESULT=SUCCESS
2013-05-13 14:18:05,794 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Creating password for identifier: owner=root, renewer=mr token, realUser=root, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0
2013-05-13 14:18:05,928 INFO org.apache.hadoop.mapred.JobInProgress: jobToken generated and stored with users keys in /tmp/hadoop-root/mapred/system/job_201305131138_0013/jobToken
2013-05-13 14:18:05,935 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201305131138_0013 = 5. Number of splits = 1
2013-05-13 14:18:05,935 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201305131138_0013_m_000000 has split on node:/default-rack/localhost.localdomain
2013-05-13 14:18:05,935 INFO org.apache.hadoop.mapred.JobInProgress: job_201305131138_0013 LOCALITY_WAIT_FACTOR=1.0
2013-05-13 14:18:05,936 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201305131138_0013 initialized successfully with 1 map tasks and 0 reduce tasks.
2013-05-13 14:18:08,450 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_SETUP) 'attempt_201305131138_0013_m_000002_0' to tip task_201305131138_0013_m_000002, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:59737'
2013-05-13 14:18:14,468 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201305131138_0013_m_000002_0' has completed task_201305131138_0013_m_000002 successfully.
2013-05-13 14:18:14,473 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201305131138_0013_m_000000_0' to tip task_201305131138_0013_m_000000, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:59737'
2013-05-13 14:18:14,475 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201305131138_0013_m_000000
2013-05-13 14:18:20,518 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201305131138_0013_m_000000_0' has completed task_201305131138_0013_m_000000 successfully.
2013-05-13 14:18:20,524 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_CLEANUP) 'attempt_201305131138_0013_m_000001_0' to tip task_201305131138_0013_m_000001, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:59737'
2013-05-13 14:18:26,545 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201305131138_0013_m_000001_0' has completed task_201305131138_0013_m_000001 successfully.
2013-05-13 14:18:26,567 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201305131138_0013 has completed successfully.
2013-05-13 14:18:26,567 INFO org.apache.hadoop.mapred.JobInProgress$JobSummary: jobId=job_201305131138_0013,submitTime=1368434885696,launchTime=1368434885935,firstMapTaskLaunchTime=1368434894473,firstJobSetupTaskLaunchTime=1368434888450,firstJobCleanupTaskLaunchTime=1368434900524,finishTime=1368434906567,numMaps=1,numSlotsPerMap=1,numReduces=0,numSlotsPerReduce=1,user=root,queue=default,status=SUCCEEDED,mapSlotSeconds=13,reduceSlotsSeconds=0,clusterMapCapacity=2,clusterReduceCapacity=2
2013-05-13 14:18:26,618 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131138_0013_m_000000_0'
2013-05-13 14:18:26,624 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131138_0013_m_000001_0'
2013-05-13 14:18:26,625 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131138_0013_m_000002_0'
2013-05-13 14:18:26,619 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/home/user/hadoop-1.0.0/logs/history/job_201305131138_0013_1368434885696_root_oozie%3Alauncher%3AT%3Djava%3AW%3Dtest%5Fjob%3AA%3Dtest%5Fjob%3AID%3D000 to file:/home/user/hadoop-1.0.0/logs/history/done/version-1/localhost_1368425312972_/2013/05/13/000000
2013-05-13 14:18:26,637 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/home/user/hadoop-1.0.0/logs/history/job_201305131138_0013_conf.xml to file:/home/user/hadoop-1.0.0/logs/history/done/version-1/localhost_1368425312972_/2013/05/13/000000
2013-05-13 14:18:26,733 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Creating password for identifier: owner=root, renewer=mr token, realUser=root, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0
Can you please help?
Regards,
Shouvanik
549 Views
0 Likes
Anonymous
Not applicable
‎2013-05-15 06:15 AM
Author

Hi,
I am having a issue here.
I have installed Talend Open Studio for Big data ver 5.2.1 on windows. I have created a simple job. It consists of a simple HBASE connection. When I run the job, no error occurs.
Then I try to submit and run the job using Talend OOZIE plugin. I get the following error
Deploying job to Hadoop...
Deployment failed!
Can not access Hadoop File System with user root!
Server IPC version 7 cannot communicate with client version 4

I have checked that hadoop jars inside "C:\Users\shouvanik.haldar\Downloads\TOS_BD-r95165-V5.2.1\TOS_BD-r95165-V5.2.1\plugins\org.talend.designer.components.bigdata_5.2.1.r95165\components\tHBaseConnection" are different from the hadoop jars present in the server.
What should I do? Please help
Regards,
Shouvanik
549 Views
0 Likes
Anonymous
Not applicable
‎2013-05-15 08:01 AM
Author

Hi Sabrina,
Can you please reply to my latest post?
Regards,
Shouvanik
549 Views
0 Likes
Anonymous
Not applicable
‎2013-05-16 01:21 AM
Author

Hi,
I have checked that hadoop jars inside "C:\Users\shouvanik.haldar\Downloads\TOS_BD-r95165-V5.2.1\TOS_BD-r95165-V5.2.1\plugins\org.talend.designer.components.bigdata_5.2.1.r95165\components\tHBaseConnection" are different from the hadoop jars present in the server.

Sorry for the late response. I suspect the .jar are not compatible. As a workaround, you can use Talend open studio for BigData 5.3.0 in which there is a "Custom" distribution option.
See the related reference tHBaseConnection and Connecting to a custom Hadoop distribution
In addition, please check supported Hadoop version Supported+Hadoop+distribution+versions
Best regards
Sabrina
549 Views
0 Likes
Anonymous
Not applicable
‎2013-05-16 03:22 AM
Author

Hi Sabrina,
Currently we have a constraint in that we have to stick to this version. Can you please tell me a workaround so that I can remove this error from version 5.2.2.
Please help.
Regards,
Shouvanik
549 Views
0 Likes
Anonymous
Not applicable
‎2013-05-16 03:43 AM
Author

Hi,
What is your hadoop version? I have see the version Talend open studio for bigdata 5.2.2 r99271 and the .jar file is "hbase-0.92.1-cdh4.0.1"
Best regards
Sabrina
549 Views
0 Likes
Anonymous
Not applicable
‎2013-05-16 04:12 AM
Author

Hi Sabrina,
I am using Talend Open Studio for Big data version 5.2.1 (TOS_BD-r95165-V5.2.1).
The hadoop version on the server(RHEL box) shows the following when I give command : hadoop version
Hadoop 2.0.0-cdh4.2.0
Subversion file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.2.0/src/hadoop-common-project/hadoop-common -r 8bce4bd28a464e0a92950c50ba01a9deb1d85686
Compiled by jenkins on Fri Feb 15 11:13:32 PST 2013
From source with checksum 3eefc211a14ac7b6e764d6ded2eeeb26
Regards,
Shouvanik
549 Views
0 Likes
Anonymous
Not applicable
‎2013-05-16 04:32 AM
Author

Hi Sabrina,
I also tried the below.
Copied all hadoop jars from server location inside -> C:\Users\shouvanik.haldar\Downloads\TOS_BD-r95165-V5.2.1\TOS_BD-r95165-V5.2.1\plugins\org.talend.designer.components.bigdata_5.2.1.r95165\components\tHBaseConnection folder in addition to all the jars residing there.
Still I am getting the error as
Deploying job to Hadoop...
Deployment failed!
Can not access Hadoop File System with user root!
Server IPC version 7 cannot communicate with client version 4

Here I am testing tHBaseConnection_1 component only.

Do I also need to change the index.xml file inside
C:\Users\shouvanik.haldar\Downloads\TOS_BD-r95165-V5.2.1\TOS_BD-r95165-V5.2.1\lib\java folder?
Please help.
549 Views
0 Likes