topic Re: Large Data Transfer in Talend Studio

Large Data Transfer

thomas_2583 — Fri, 15 Nov 2024 22:57:47 GMT

Hi,

I have created a job design to move data from one database into another. This particular job is trying to transfer c.500k rows of data. I have run the job (took around 5 hours) and it looks to have run fine, however, the database (postgres) the data is being moved to, only shows around c.8k records.

Any idea why this could be, please? I am new to Talend so if there is any more information that is needed, please let me know.

Re: Large Data Transfer

gjeremy1617088143 — Tue, 03 May 2022 09:16:49 GMT

Hi could you show the configuration of your tDBOutput, basic and advance settings ?

Re: Large Data Transfer

thomas_2583 — Tue, 03 May 2022 09:22:51 GMT

I hope this is ok, I have obviously had to remove the connection details.

Re: Large Data Transfer

gjeremy1617088143 — Tue, 03 May 2022 09:44:52 GMT

Update or insert option is really poor in performance on huge data set, as you check inner join reject just use insert options, also you could raise your commit every number to 10000 for example. With that you will really lower your execution time.