Parallel execution of Talend job

Anonymous — Mon, 30 Jun 2014 20:07:35 GMT

Hi All,
We are trying to implement generic TRest Service which in turn calls Talend job to load data into client data base by picking up file from respective client directory.
tRestRequest -> TXMLMap-> tRunJob -> TRestResponse
TRunjob - In turn picks up file from FTP, loads data into Staging table and then loads data into Database and sends report.
But our main concern is parallel execution as we deal with lot of data imports running in parallel.
We are seeing some problems when we try to invoke same service with payload information for different data loads at same time.
E.g. When I write curl command to invoke my generic service to load data into 12 tables using 12 times service call from respective client FTP at same time.
Then data load is working fine for 10 tables, but for 2 tables, files are also not getting picked up from client directory.
Four reports generated were also showing incorrect data.
Request you to please help me to handle this parallel execution issue.
Is there any better way to run job in Talend, so as to avoid such concurrency issues.
Please let me know if you need additional information on the same.
Thanks and Regards,

Re: Parallel execution of Talend job

Anonymous — Tue, 01 Jul 2014 06:38:27 GMT

Hi Sheetal,
- Whether the file are in use somewhere / or locked by some process? try moving file to other location
- Whether there is any filter issue for file names?
- Can you debug by using some print statements and tlogrow to check what is happening?
Thanks
Vaibhav

topic Parallel execution of Talend job in Talend Studio

Parallel execution of Talend job

Re: Parallel execution of Talend job