Issue with DeltaLake Output (tDBOutput)

vbadri — Sat, 16 Nov 2024 00:10:17 GMT

Hello there,

I'm having the following issue with DeltaLake Output (tDBOutput), when I select just the 'INSERT' or just the 'UPDATE' for 'Data Action', it is working great and processing inserts nice and quick (screenshot attached), processed 30k records in 70 seconds.

When I select 'UPDATE_OR_INSERT' or

'INSERT_OR_UPDATE'

, the processing speeds are awful.Any suggestions would be appreciated.

Thanks.

Re: Issue with DeltaLake Output (tDBOutput)

Anonymous — Wed, 12 May 2021 08:20:17 GMT

Hello,

By using 'Insert or update' operation(No batch size settings in it's advances settings), for each row, it will do 2 db operations (one for lookup, another one is for db insert or update).

To speed the insert or update performance, it's recommended to use tMap do a lookup with the DB data and create 2 output rows : Update , Insert,

Then select the operation 'Update' for the update row, and select the operation 'Insert' for the insert row in the Db output components.

Let us know if this solution helps.

Best regards

Sabrina

topic Issue with DeltaLake Output (tDBOutput) in Talend Studio

Issue with DeltaLake Output (tDBOutput)

Re: Issue with DeltaLake Output (tDBOutput)