topic Re: Job performance in Talend Studio

Job performance

Anonymous — Fri, 19 Apr 2019 10:57:08 GMT

Hello !
I have a job that retrieves the information from tables, passes it to a tMap and stores it in another table. The problem is that I end up with 45 lines per second. The job is slow ... and I do not think it's normal.
My database is on the network and my table has 900000 lines.
Could you help me please ?

Re: Job performance

manodwhb — Fri, 19 Apr 2019 11:21:52 GMT

@Guen , You need to identify where the problem is.

As per your job you can do there things that

1) re-writing the query to improve the performance in tOracleInput

2) do the tMap operations on store on disk there you need to specify the temp location

3) use the tOracleOutputBulkExe component.

Re: Job performance

vapukov — Fri, 19 Apr 2019 11:58:40 GMT

what settings do you use for output component?

Re: Job performance

Anonymous — Fri, 19 Apr 2019 13:02:00 GMT

Hello @vapukov

Here are the simple settings and advanced settings that I use in my output component.

Re: Job performance

vapukov — Fri, 19 Apr 2019 13:04:55 GMT

Thank you,

first what you need to test - checkbox "Use Batch", test with 10000, little more but not then 20000

Re: Job performance

Anonymous — Fri, 19 Apr 2019 13:14:11 GMT

Great @vapukov ! when I checked "Use Batch" "and tested with 10000, the flow went to 330 lines per second. Should I increase it to have more lines per second?

Re: Job performance

vapukov — Fri, 19 Apr 2019 13:15:39 GMT

you can test add or reduce

total speed depends from server settings, for example insert could be faster, but commit longer

Re: Job performance

Anonymous — Fri, 19 Apr 2019 13:30:46 GMT

Thank you verry much @vapukov