topic Re: Parallel job - getting the statistics in Talend Studio

Parallel job - getting the statistics

Anonymous — Wed, 15 Feb 2017 07:57:44 GMT

Hi, I have a main job when executed, runs 15 jobs in parallel. Each job has a table to write (insert or update) from sql server to mysql.
I want to get the execution time of each job and the number of rows that were affected. I want the table_name,inserted_rows, updated_rows,job_completion_time
So I run this job 4 times in a day. And the data in this log table must have information for every run
I am using TOS - 6.1.1
Any ideas on how I can achieve this would be of great help
Thanks in advance
Rathi

Re: Parallel job - getting the statistics

Anonymous — Wed, 15 Feb 2017 11:19:01 GMT

Hi Rathi,
Create a table with columns table_name,inserted_rows, updated_rows,job_completion_tim Asume the table name to be "JobTracker".
Create an autoincrement column in the JobTracker table to capture the deatils of the job with an id assume jobtrackerid.

For all the outputcomponents u would have global variable available , just click on drop down of the output component in the outline
which would be available on the bottom left.The you would find the global variable like NBLine_Inserted etc.
Just drag and drop to the required field to get the recoreds inserted or updated.

at end of each job populate the details into the table.

tfixedflowinput ---> tmysqloutput(JobTracker Table).

Regards,
Prince

Re: Parallel job - getting the statistics

Anonymous — Wed, 15 Feb 2017 11:25:02 GMT

For the execution time at start of each job create a global variable var1 (var1 = TalendDate.getCurrentDate()) to store the talend current datetime.
similarly at the end of each job apply the below logic.
var1 = TalendDate.getCurrentDate() - var1;

Re: Parallel job - getting the statistics

Anonymous — Wed, 15 Feb 2017 11:26:41 GMT

Hi Prince,
Thanks a lot for explaining.
So, should the flow be tmssqlinput --> tmap --> tmysqloutput(maintable) -- > tfixedflowinput --> tmysqloutput(jobtracker)
The same sequence will be repeated 15 times for different tables. All 15 jobs will try to insert into jobtracker, will there not be any lock issues?
Also, how do I get the execution time for each job?
Thanks
Rathi

Re: Parallel job - getting the statistics

Anonymous — Wed, 15 Feb 2017 13:38:06 GMT

for the exection time:
tjava1--->on subjob ok --->Main JOb ----->onSubjobOk--->tjava2.--->tfixedflow---->JobTracker
In tJava1:Create a variable V1 = TalendDate.getCurrentDate()
In tJava2:V1 = TalendDate.getCurrentDate()-V1
In tFixedflowinput in the column job_completion_tim privide context.V1
Regarding locks no it would nt .
please don t forget to close the JobTracker connection
example: tmysqlConnection --> tfixedflowinput--->jobtracker --->tmysqlcommit(Please check the checkbox close connection)

Re: Parallel job - getting the statistics

Anonymous — Thu, 16 Feb 2017 12:43:30 GMT

Hi,
I tried creating the job as you said, but V1 was not recognized globally, so used a tsetglobalvar.
I was reading and came across tstatscatcher,tflowmetercatcher, tlogcatcher. What method must I opt for , these components or the method you have suggested?
Thanks
Rathi