topic Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE in Talend Studio

Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Sat, 16 Nov 2024 12:02:14 GMT

Hi,
I am very new to Talend. Is there a way to schedule Hadoop Jobs (e.g. HIVE, PIG etc.) using OOZIE plugin.
Can you please show me a example.
Regards,
Shouvanik

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Tue, 07 May 2013 08:46:45 GMT

Hi,
For community version(Talend Open Studio for BigData), we support for OOZIE.
As a newbie, I encourage you to read the online manual of How+to+run+a+Job+on+a+remote+HDFS+server with four chapters How+to+set+HDFS+connection+details, How+to+run+a+Job+on+the+HDFS+server, How+to+schedule+the+executions+of+a+Job and How+to+monitor+Job+execution+status. Hope it is helpful for you.
Best regards
Sabrina

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Tue, 07 May 2013 11:11:24 GMT

Hi,
Are these tutorials for community edition? I have a job. I just want to schedule it using OOZIE. What are the steps. It will be good if you can give me the steps. I could not follow from the links you had shared.
Regards,
Shouvanik

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Tue, 07 May 2013 11:23:46 GMT

Hi,
It is community edition tutorials on talend help center. The references provide tips and tricks to teach you how to schedule Hadoop Jobs (e.g. HIVE, PIG etc.) using OOZIE, what's more, it is for free.
You can log in Talend Help Center to see the related references using your forum account. If you cannot log in, please read the notice forum Upcoming Maintenance.
Best regards
Sabrina

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Tue, 07 May 2013 11:58:04 GMT

Hi Sabrina,
Do we always have to create a connection to HDFS before scheduling a job? Can't we schedule any Talend job using OOZIE.
Like, I have a example job, which reads data from a fixed format input file and puts into HBASE.
Next, I export the job. Now, in OOZIE tab when I am clicking on run, I get the following error.

Deploying job to Hadoop...
Deployment failed!
Can not access Hadoop File System with user user!
Protocol org.apache.hadoop.hdfs.protocol.ClientProtocol version mismatch. (client = 61, server = 63)

Please help
Regards,
Shouvanik

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Tue, 07 May 2013 12:19:11 GMT

Hi
Before you can run or schedule executions of a Job on an HDFS server, you need first to define the HDFS connection details either in the Oozie scheduler view or in the studio preference settings, and specify the path where your Job will be deployed. see How to set HDFS connection details.
Shong

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Tue, 07 May 2013 12:54:03 GMT

Hi,
Please help me out with the above HDFS issue.
I am using Talend 5.2 Open Studio for Big data and its a community edition. It does not have all the features.
Please help me
Thanks,
Shouvanik

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Tue, 07 May 2013 16:03:25 GMT

Hi,
You need to open the View which is not open by default in the Studio.
Here is the procedure to open the Talend Oozie view :
In your Menu = Window > Show view > Talend Oozie.
Then the View is open and appears beside your RUN view; you can set up your connection to Oozie and then run.
(A valid PATH on your hadoop cluster need to exist).
Regards;
Christophe

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Wed, 08 May 2013 04:27:21 GMT

Hi Shouvanik,
To make it more clear, attach the screenshot in forum.
Best regards
Sabrina

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Wed, 08 May 2013 05:34:47 GMT

Hi,
@cantoine: Opening Talend OOZIE view window is not a problem here. The issue is: I want to schedule and run a Talend job using OOZIE workflow scheduler plugin provided with Talend Open studio (screenshot of the product attached.).
Is it possible to do that? Why do I have to connect to HDFS always remotely in order to do that.
@sabrina: I am attaching the screenshot of the job, which I created from the example given. Can I schedule it with OOZIE?

Moreover, when I am trying to schedule a job and click on RUN button, I get the following error.
Deploying job to Hadoop...
Deployment failed!
Can not access Hadoop File System with user root!
Protocol org.apache.hadoop.hdfs.protocol.ClientProtocol version mismatch. (client = 61, server = 63)
What should be going inside the field for PATH inside OOZIE view?
I suspect that the hadoop version running locally in my machine is of a incompatible version with respect to the version required by Talend.
Can you please tell me what Hadoop version is compatible with Talend Hadoop client version?

Please help me.
Regards,
Shouvanik

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Wed, 08 May 2013 07:58:36 GMT

Hi,

I suspect that the hadoop version running locally in my machine is of a incompatible version with respect to the version required by Talend.
Can you please tell me what Hadoop version is compatible with Talend Hadoop client version?

You can check the online guide Supported+systems+and+databases and Supported+Hadoop+distribution+versions.
Best regards
Sabrina

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Wed, 08 May 2013 13:17:52 GMT

Thanks.
But how to include them inside Talend? I am confused here.
Regards,
Shouvanik

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Mon, 13 May 2013 09:51:00 GMT

Hi,
I am able to schedule and run jobs using OOZIE plugin of Talend.
The main problem was finding out which version of hadoop is compatible with which version of oozie.
Now, I am having different outcomes with different jobs using Talend
1. A simple job "HDFSJob" which has a big data component called tHDFSConnection_1
This job runs smoothly and getting succeeded.

2. A job "test_job" which prints a message "Hello World"
This job runs, but after sometime, I get a error in Talend console which is
Deploying job to Hadoop...
Deployment complete!
Job is running ...
Job killed!
Main class , exit code
Job failed, error message, exit code ]
And when I look at oozie log, i get the following output
2013-05-13 14:18:04,586 INFO ActionStartXCommand:539 - USER GROUP TOKEN[] APP JOB ACTION Start action with user-retry state : userRetryCount , userRetryMax , userRetryInterval
2013-05-13 14:18:04,593 WARN ActionStartXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION Action status=DONE
2013-05-13 14:18:04,594 WARN ActionStartXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION Action updated in DB!
2013-05-13 14:18:04,807 INFO ActionStartXCommand:539 - USER GROUP TOKEN[] APP JOB ACTION Start action with user-retry state : userRetryCount , userRetryMax , userRetryInterval
2013-05-13 14:18:05,100 WARN JavaActionExecutor:542 - USER GROUP TOKEN[] APP JOB ACTION credentials is null for the action
2013-05-13 14:18:05,418 WARN JobClient:667 - Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
2013-05-13 14:18:05,486 WARN JobClient:784 - No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
2013-05-13 14:18:05,799 INFO JavaActionExecutor:539 - USER GROUP TOKEN[] APP JOB ACTION checking action, external ID status
2013-05-13 14:18:05,810 WARN ActionStartXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION Action status=RUNNING
2013-05-13 14:18:05,811 WARN ActionStartXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION Action updated in DB!
2013-05-13 14:18:15,524 INFO StatusTransitService$StatusTransitRunnable:539 - USER GROUP Acquired lock for
2013-05-13 14:18:15,529 INFO StatusTransitService$StatusTransitRunnable:539 - USER GROUP Running coordinator status service from last instance time = 2013-05-13T08:47Z
2013-05-13 14:18:15,531 INFO StatusTransitService$StatusTransitRunnable:539 - USER GROUP Running bundle status service from last instance time = 2013-05-13T08:47Z
2013-05-13 14:18:15,532 INFO StatusTransitService$StatusTransitRunnable:539 - USER GROUP Released lock for
2013-05-13 14:18:16,431 INFO PauseTransitService:539 - USER GROUP Acquired lock for
2013-05-13 14:18:16,455 INFO PauseTransitService:539 - USER GROUP Released lock for
2013-05-13 14:18:26,624 INFO CallbackServlet:539 - USER GROUP TOKEN APP JOB ACTION callback for action
2013-05-13 14:18:26,773 INFO JavaActionExecutor:539 - USER GROUP TOKEN[] APP JOB ACTION action completed, external ID
2013-05-13 14:18:26,824 WARN JavaActionExecutor:542 - USER GROUP TOKEN[] APP JOB ACTION Launcher ERROR, reason: Main class , exit code
2013-05-13 14:18:26,922 INFO ActionEndXCommand:539 - USER GROUP TOKEN[] APP JOB ACTION ERROR is considered as FAILED for SLA
2013-05-13 14:18:26,951 INFO ActionStartXCommand:539 - USER GROUP TOKEN[] APP JOB ACTION Start action with user-retry state : userRetryCount , userRetryMax , userRetryInterval
2013-05-13 14:18:26,951 WARN ActionStartXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION Action status=DONE
2013-05-13 14:18:26,952 WARN ActionStartXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION Action updated in DB!
2013-05-13 14:18:27,012 WARN CoordActionUpdateXCommand:542 - USER GROUP TOKEN[] APP JOB ACTION E1100: Command precondition does not hold before execution, , Error Code: E1100

And when I check the job tracker log, I get the following log
2013-05-13 14:18:05,367 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Creating password for identifier: owner=root, renewer=mr token, realUser=root, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0
2013-05-13 14:18:05,387 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Creating password for identifier: owner=root, renewer=mr token, realUser=root, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0
2013-05-13 14:18:05,734 INFO org.apache.hadoop.mapred.JobInProgress: job_201305131138_0013: nMaps=1 nReduces=0 max=-1
2013-05-13 14:18:05,734 INFO org.apache.hadoop.mapred.JobTracker: Initializing job_201305131138_0013
2013-05-13 14:18:05,735 INFO org.apache.hadoop.mapred.JobInProgress: Initializing job_201305131138_0013
2013-05-13 14:18:05,742 INFO org.apache.hadoop.mapred.JobTracker: Job job_201305131138_0013 added successfully for user 'root' to queue 'default'
2013-05-13 14:18:05,743 INFO org.apache.hadoop.mapred.AuditLogger: USER=root IP=127.0.0.1 OPERATION=SUBMIT_JOB TARGET=job_201305131138_0013 RESULT=SUCCESS
2013-05-13 14:18:05,794 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Creating password for identifier: owner=root, renewer=mr token, realUser=root, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0
2013-05-13 14:18:05,928 INFO org.apache.hadoop.mapred.JobInProgress: jobToken generated and stored with users keys in /tmp/hadoop-root/mapred/system/job_201305131138_0013/jobToken
2013-05-13 14:18:05,935 INFO org.apache.hadoop.mapred.JobInProgress: Input size for job job_201305131138_0013 = 5. Number of splits = 1
2013-05-13 14:18:05,935 INFO org.apache.hadoop.mapred.JobInProgress: tip:task_201305131138_0013_m_000000 has split on node:/default-rack/localhost.localdomain
2013-05-13 14:18:05,935 INFO org.apache.hadoop.mapred.JobInProgress: job_201305131138_0013 LOCALITY_WAIT_FACTOR=1.0
2013-05-13 14:18:05,936 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201305131138_0013 initialized successfully with 1 map tasks and 0 reduce tasks.
2013-05-13 14:18:08,450 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_SETUP) 'attempt_201305131138_0013_m_000002_0' to tip task_201305131138_0013_m_000002, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:59737'
2013-05-13 14:18:14,468 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201305131138_0013_m_000002_0' has completed task_201305131138_0013_m_000002 successfully.
2013-05-13 14:18:14,473 INFO org.apache.hadoop.mapred.JobTracker: Adding task (MAP) 'attempt_201305131138_0013_m_000000_0' to tip task_201305131138_0013_m_000000, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:59737'
2013-05-13 14:18:14,475 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201305131138_0013_m_000000
2013-05-13 14:18:20,518 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201305131138_0013_m_000000_0' has completed task_201305131138_0013_m_000000 successfully.
2013-05-13 14:18:20,524 INFO org.apache.hadoop.mapred.JobTracker: Adding task (JOB_CLEANUP) 'attempt_201305131138_0013_m_000001_0' to tip task_201305131138_0013_m_000001, for tracker 'tracker_localhost.localdomain:localhost.localdomain/127.0.0.1:59737'
2013-05-13 14:18:26,545 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201305131138_0013_m_000001_0' has completed task_201305131138_0013_m_000001 successfully.
2013-05-13 14:18:26,567 INFO org.apache.hadoop.mapred.JobInProgress: Job job_201305131138_0013 has completed successfully.
2013-05-13 14:18:26,567 INFO org.apache.hadoop.mapred.JobInProgress$JobSummary: jobId=job_201305131138_0013,submitTime=1368434885696,launchTime=1368434885935,firstMapTaskLaunchTime=1368434894473,firstJobSetupTaskLaunchTime=1368434888450,firstJobCleanupTaskLaunchTime=1368434900524,finishTime=1368434906567,numMaps=1,numSlotsPerMap=1,numReduces=0,numSlotsPerReduce=1,user=root,queue=default,status=SUCCEEDED,mapSlotSeconds=13,reduceSlotsSeconds=0,clusterMapCapacity=2,clusterReduceCapacity=2
2013-05-13 14:18:26,618 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131138_0013_m_000000_0'
2013-05-13 14:18:26,624 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131138_0013_m_000001_0'
2013-05-13 14:18:26,625 INFO org.apache.hadoop.mapred.JobTracker: Removing task 'attempt_201305131138_0013_m_000002_0'
2013-05-13 14:18:26,619 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/home/user/hadoop-1.0.0/logs/history/job_201305131138_0013_1368434885696_root_oozie%3Alauncher%3AT%3Djava%3AW%3Dtest%5Fjob%3AA%3Dtest%5Fjob%3AID%3D000 to file:/home/user/hadoop-1.0.0/logs/history/done/version-1/localhost_1368425312972_/2013/05/13/000000
2013-05-13 14:18:26,637 INFO org.apache.hadoop.mapred.JobHistory: Moving file:/home/user/hadoop-1.0.0/logs/history/job_201305131138_0013_conf.xml to file:/home/user/hadoop-1.0.0/logs/history/done/version-1/localhost_1368425312972_/2013/05/13/000000
2013-05-13 14:18:26,733 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Creating password for identifier: owner=root, renewer=mr token, realUser=root, issueDate=0, maxDate=0, sequenceNumber=0, masterKeyId=0
Can you please help?
Regards,
Shouvanik

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Wed, 15 May 2013 10:15:22 GMT

Hi,
I am having a issue here.
I have installed Talend Open Studio for Big data ver 5.2.1 on windows. I have created a simple job. It consists of a simple HBASE connection. When I run the job, no error occurs.
Then I try to submit and run the job using Talend OOZIE plugin. I get the following error
Deploying job to Hadoop...
Deployment failed!
Can not access Hadoop File System with user root!
Server IPC version 7 cannot communicate with client version 4

I have checked that hadoop jars inside "C:\Users\shouvanik.haldar\Downloads\TOS_BD-r95165-V5.2.1\TOS_BD-r95165-V5.2.1\plugins\org.talend.designer.components.bigdata_5.2.1.r95165\components\tHBaseConnection" are different from the hadoop jars present in the server.
What should I do? Please help
Regards,
Shouvanik

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Wed, 15 May 2013 12:01:58 GMT

Hi Sabrina,
Can you please reply to my latest post?
Regards,
Shouvanik

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Thu, 16 May 2013 05:21:37 GMT

Hi,

I have checked that hadoop jars inside "C:\Users\shouvanik.haldar\Downloads\TOS_BD-r95165-V5.2.1\TOS_BD-r95165-V5.2.1\plugins\org.talend.designer.components.bigdata_5.2.1.r95165\components\tHBaseConnection" are different from the hadoop jars present in the server.

Sorry for the late response. I suspect the .jar are not compatible. As a workaround, you can use Talend open studio for BigData 5.3.0 in which there is a "Custom" distribution option.
See the related reference tHBaseConnection and Connecting to a custom Hadoop distribution
In addition, please check supported Hadoop version Supported+Hadoop+distribution+versions
Best regards
Sabrina

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Thu, 16 May 2013 07:22:00 GMT

Hi Sabrina,
Currently we have a constraint in that we have to stick to this version. Can you please tell me a workaround so that I can remove this error from version 5.2.2.
Please help.
Regards,
Shouvanik

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Thu, 16 May 2013 07:43:23 GMT

Hi,
What is your hadoop version? I have see the version Talend open studio for bigdata 5.2.2 r99271 and the .jar file is "hbase-0.92.1-cdh4.0.1"
Best regards
Sabrina

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Thu, 16 May 2013 08:12:54 GMT

Hi Sabrina,
I am using Talend Open Studio for Big data version 5.2.1 (TOS_BD-r95165-V5.2.1).
The hadoop version on the server(RHEL box) shows the following when I give command : hadoop version
Hadoop 2.0.0-cdh4.2.0
Subversion file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.2.0/src/hadoop-common-project/hadoop-common -r 8bce4bd28a464e0a92950c50ba01a9deb1d85686
Compiled by jenkins on Fri Feb 15 11:13:32 PST 2013
From source with checksum 3eefc211a14ac7b6e764d6ded2eeeb26
Regards,
Shouvanik

Re: Not able to schedule big-data jobs(e.g.PIG,HIVE,M/R) using OOZIE

Anonymous — Thu, 16 May 2013 08:32:09 GMT

Hi Sabrina,
I also tried the below.
Copied all hadoop jars from server location inside -> C:\Users\shouvanik.haldar\Downloads\TOS_BD-r95165-V5.2.1\TOS_BD-r95165-V5.2.1\plugins\org.talend.designer.components.bigdata_5.2.1.r95165\components\tHBaseConnection folder in addition to all the jars residing there.
Still I am getting the error as
Deploying job to Hadoop...
Deployment failed!
Can not access Hadoop File System with user root!
Server IPC version 7 cannot communicate with client version 4

Here I am testing tHBaseConnection_1 component only.

Do I also need to change the index.xml file inside
C:\Users\shouvanik.haldar\Downloads\TOS_BD-r95165-V5.2.1\TOS_BD-r95165-V5.2.1\lib\java folder?
Please help.