Large Data Transfer

[thomas_2583]

Hi,

I have created a job design to move data from one database into another. This particular job is trying to transfer c.500k rows of data. I have run the job (took around 5 hours) and it looks to have run fine, however, the database (postgres) the data is being moved to, only shows around c.8k records.

Any idea why this could be, please? I am new to Talend so if there is any more information that is needed, please let me know.

[gjeremy1617088143]

Hi could you show the configuration of your tDBOutput, basic and advance settings ?

[thomas_2583]

I hope this is ok, I have obviously had to remove the connection details.

 

[gjeremy1617088143 · Accepted solution]

Update or insert option is really poor in performance on huge data set, as you check inner join reject just use insert options, also you could raise your commit every number to 10000 for example. With that you will really lower your execution time.