topic How to improve the loading performance in Talend Studio

How to improve the loading performance

Darmesh — Tue, 10 Jul 2018 12:51:47 GMT

Help me how to improve the loading performance. i have 1 lakhs record in a table and moving into destination table.

It is loading very slow. I am using Talend open studio 6.4.1

Re: How to improve the loading performance

vapukov — Wed, 11 Jul 2018 03:33:49 GMT

performance in this case depends from target database

You can examine settings and configuration for check how them relevant for task

also You can test job with tMySQLBulkExec instead of tMySQLOutput

Re: How to improve the loading performance

Darmesh — Fri, 13 Jul 2018 11:51:45 GMT

I am doing insert/update option . it is not possible using tmysqloutputbulkexec.

So can you tel me how to speed up the loading process

Re: How to improve the loading performance

Jesperrekuh — Fri, 13 Jul 2018 12:44:00 GMT

In/Update is your problem... only a redesign of your process by doing inserts only and rejected records to update...

Or... wifi / network speed limits your throughput.

Re: How to improve the loading performance

vapukov — Fri, 13 Jul 2018 13:22:53 GMT

insert/update - always slow operations, and You can not expect any chances to improve it (yes, it possible tune database, but it will be not huge difference)

You can try redesign job - there are many variant depending from data:
- You can bulk import all into database (into separate table), and use
- or update where exists + insert where not exists
- or delete where exists + insert all

P.S. it is bad practice to use local terms on international forum if You not expect answers in French or Chinese (I can also start use Russian terms 😉 ) ... but in any case 100 000 records - it is not too big for run inside database

P.P.S. of course staging table must have indexes