topic Re: Improving load/insert process in Talend Studio

Improving load/insert process

UName1615412505 — Sat, 16 Nov 2024 00:24:05 GMT

Hello everyone,

I'm new to Talend and I wanted to know if there was any way to improve pulling and loading data from and into databases. After some research I'v found that using multiple connection for my tDBInputs and tDBCommits to "split" my load would increase performance but I'm still not sure it's the best way to go.

Also, is disabling auto-commit and using a tDBCommit after my output is it the best way to do it or maybe a tBulkInsert would be better.

If you have any other suggestions, I'm all ears.

PS: I'm using Talend Cloud and two different DB : PostgreSQL for load/insert an Oracle for loading.

Re: Improving load/insert process

Anonymous — Mon, 22 Mar 2021 04:09:55 GMT

Hello,

Generally speaking, the followings aspects could affect the job performance:

1. The volume of data, read a large of data set, the performance will degrade.

2. The structure of data, if there are so many columns on tDBInput, it will consume many memory and much time for transferring the data during the job execution.

3. The database connection, the job always runs better if the database is installed on local, if the database is on another machine, even you are on VPN, you may have the congestion and latency issues.

Could you please show us your current job design screenshots which will be helpful for us to understand your work flow well?

Best regards

Sabrina

Re: Improving load/insert process

UName1615412505 — Tue, 23 Mar 2021 07:02:27 GMT

Hello and thank you for your answer,

This was my first draft of the job (a child job)

And by adding more connection :

Also, will using a tRunJob in a child job hinder performance ?