topic Re: tDBOutput extremely slow in Talend Studio

tDBOutput extremely slow

SheldonMarumoto — Wed, 13 Aug 2025 22:11:47 GMT

I have a process that pulls from a MySQL database, compares to existing entries in a destination database, and then Inserts or Updates depending on existence.

The Updates are extremely slow, along the lines of ~2 rows/sec. This is fairly consistent regardless of batch size or commit size. Destination has an appropriate index for the key.

When the job is modified to send Updates to a tLogRow component, output jumps to nearly 8k rows/sec.

Any ideas why this tDBOutput component is going so slow?

Re: tDBOutput extremely slow

Denis_Segard — Thu, 14 Aug 2025 04:26:39 GMT

Hello
What is the Talend Studio version / Patch level used ?

Kind regards
Denis

Re: tDBOutput extremely slow

Dave_Simo — Thu, 14 Aug 2025 09:00:48 GMT

Hello @SheldonMarumoto

You can check out this discussion below where we discussed how to optimize inserts in a tDBOutput component.

https://community.qlik.com/t5/Talend-Studio/Taking-160mins-to-importing-2-48-crore-record-into-MSSQL-with/m-p/2522303#M147688

Best Regards

Re: tDBOutput extremely slow

SheldonMarumoto — Thu, 14 Aug 2025 15:43:56 GMT

Version 8, 20250218_0945-patch

Re: tDBOutput extremely slow

Denis_Segard — Thu, 14 Aug 2025 16:15:05 GMT

Hello,

Apply at least the patch R2025-04 . A bug in R2025-02 caused performance issues .
https://help.qlik.com/talend/en-US/patch-notes/8.0/r2025-04
QAPPINT-1330 - tDB components degraded performance R2025-02

Kind regards
Denis

Re: tDBOutput extremely slow

SheldonMarumoto — Thu, 14 Aug 2025 17:02:45 GMT

I have applied the latest patch, now on R2025-07v2. And it did improve performance somewhat. Still only at about 4 rows/sec, however.

Re: tDBOutput extremely slow

SheldonMarumoto — Thu, 14 Aug 2025 17:24:51 GMT

Good information, but the inserts on my process are working fine, it's the updates that are slow. Is there a way to use tDBBulkExec to do updates?

Re: tDBOutput extremely slow

Dave_Simo — Thu, 14 Aug 2025 18:28:29 GMT

Hello @SheldonMarumoto

To optimize updates in Talend, generally there are 3 options that I see :

1) Either use batch size for updates

2) Either use tDBBulkExec for bulk updates (large volume)

3) Using tELT components

Best Regards

Re: tDBOutput extremely slow

SheldonMarumoto — Fri, 15 Aug 2025 15:46:25 GMT

I am already using batch size. The existence or size of the batch doesn't seem to change things.

I will try to use the tdbOutputBulkExec or tELT.

Re: tDBOutput extremely slow

SheldonMarumoto — Fri, 15 Aug 2025 16:20:08 GMT

tDBOutputBulkExec is giving me an error that my specified .txt file cannot be opened. "Operating system error code 3(The system cannot find the path specified.)."

Could you advise on this? Thank you very much.

Re: tDBOutput extremely slow

Dave_Simo — Fri, 15 Aug 2025 17:05:37 GMT

Hello @SheldonMarumoto

This video does a good job of explaining how to use the tdbOutputBulkExec component for bulk insert/update.

https://www.youtube.com/watch?v=EQvAZPdHOmQ&t=12s

Best Regards

Re: tDBOutput extremely slow

SheldonMarumoto — Fri, 15 Aug 2025 17:06:24 GMT

Many thanks!

Re: tDBOutput extremely slow

SheldonMarumoto — Mon, 18 Aug 2025 18:14:18 GMT

Turns out the SQL Server Bulk Load command only works server-side. It cannot access a file on a local machine (which is where my locally-executed Talend job is storing the generated .txt file). My SQL Server instance is hosted on AWS RDS. Is it possible for a local running Talend instance to point the data file at an S3 bucket, and how would that be set up in the "File Name" section of the tdbOutputBulkExec component?

Re: tDBOutput extremely slow

Dave_Simo — Mon, 18 Aug 2025 18:35:11 GMT

Hello @SheldonMarumoto

I tested the update with a local file and it works after that it is on a Postgre database.

Normally even if your SQL Server database is hosted on AWS RDS, the job should be able to read the file where it was created.

If you have a subscription to the paid version of Talend, it would be more practical to open a ticket with Qlik/Talend support or try the update with a tELTOutput.

For information

Best Regards