tDBOutput extremely slow

[SheldonMarumoto]

I have a process that pulls from a MySQL database, compares to existing entries in a destination database, and then Inserts or Updates depending on existence.

The Updates are extremely slow, along the lines of ~2 rows/sec. This is fairly consistent regardless of batch size or commit size. Destination has an appropriate index for the key. 

When the job is modified to send Updates to a tLogRow component, output jumps to nearly 8k rows/sec.

Any ideas why this tDBOutput component is going so slow?

[Denis_Segard]

Hello
What is the Talend Studio version / Patch level used ?

Kind regards
Denis

[Dave_Simo]

Hello @SheldonMarumoto 

You can check out this discussion below where we discussed how to optimize inserts in a tDBOutput component.

https://community.qlik.com/t5/Talend-Studio/Taking-160mins-to-importing-2-48-crore-record-into-MSSQL...

Best Regards

[SheldonMarumoto]

Version 8, 20250218_0945-patch

[Denis_Segard]

Hello,

Apply at least the patch R2025-04 . A bug in R2025-02 caused performance issues .
https://help.qlik.com/talend/en-US/patch-notes/8.0/r2025-04
QAPPINT-1330 - tDB components degraded performance R2025-02

Kind regards
Denis

[SheldonMarumoto]

I have applied the latest patch, now on R2025-07v2. And it did improve performance somewhat. Still only at about 4 rows/sec, however.

[SheldonMarumoto]

Good information, but the inserts on my process are working fine, it's the updates that are slow. Is there a way to use tDBBulkExec to do updates?

[Dave_Simo]

Hello @SheldonMarumoto 

To optimize updates in Talend, generally there are 3 options that I see :

1) Either use batch size for updates

2) Either use tDBBulkExec for bulk updates (large volume)

3) Using tELT components

Best Regards

[SheldonMarumoto]

I am already using batch size. The existence or size of the batch doesn't seem to change things.

I will try to use the tdbOutputBulkExec or tELT.

[SheldonMarumoto]

tDBOutputBulkExec is giving me an error that my specified .txt file cannot be opened. "Operating system error code 3(The system cannot find the path specified.)."

Could you advise on this? Thank you very much.