topic Re: [resolved] How to improve performance for the job? in Talend Studio

[resolved] How to improve performance for the job?

Anonymous — Tue, 16 Feb 2016 22:15:11 GMT

I have a job to load data from flat file to oracle DB. The flat file is processed many times, once per each dimension (to insert new dimensions), once to load fact. Each dimension process uses tMap to get unique new dimension and tOracleOutput to insert records. Last fact process also uses tMap to get dimension Ids and then tOracleOutput to insert new ones.
The performance is ok when there is no/little data in DB. But it downgrades sharply as more data in DB. The flat file is from 50M to 1G, with 50K records at least. Any suggestions on improving the performance?
What's the best practice for this kind of data loading to keep a good performance?

Re: [resolved] How to improve performance for the job?

Anonymous — Wed, 17 Feb 2016 03:22:53 GMT

Hi,
Have you tried to check out "Use Batch " option in toracleOutput component to activate the batch mode for data processing?
Would you mind posting your job design screenshots into forum?
Best regards
Sabrina

Re: [resolved] How to improve performance for the job?

lvsiva — Wed, 17 Feb 2016 07:31:47 GMT

Hi hjiang,
Can you try to load with tOracleoutputBulkExec as an output.
Thanks,
Siva.

Re: [resolved] How to improve performance for the job?

Anonymous — Wed, 17 Feb 2016 10:29:29 GMT

This thread should move to the Open Studio forum https://community.talend.com/t5/Archive/Sybase-Connection/td-p/56802

Re: [resolved] How to improve performance for the job?

Anonymous — Thu, 18 Feb 2016 03:31:45 GMT

Thanks Siva and Sabrina. I will test out and let you know the result later.