topic Re: Job running really slow because of tFlowToIterate in Talend Studio

Job running really slow because of tFlowToIterate

sbarjatya — Fri, 15 Nov 2024 23:59:41 GMT

I am trying to delete rows in a target table using value from source table, using tFlowToIterate (screenshot attached). Throughput for which is really low = 0.19 rows/s. I am using cursor in tDBInput and did enable parallel execution for iterate row. But, no improvement so far. Can anyone please advise how can I improve efficiency of this simple job or if anyone can suggest any alternative approach.

Any leads would be appreciated.

Re: Job running really slow because of tFlowToIterate

Sourav_Roy — Wed, 30 Jun 2021 18:54:59 GMT

Hi,

Please increase your JVM Parameters according to your system configurations:

(JVM Settings are present in Run -> Advanced Settings -> Use specific JVM arguments.

Re: Job running really slow because of tFlowToIterate

sbarjatya — Wed, 30 Jun 2021 20:26:59 GMT

Its already running on Xms32G and Xmx64G

Re: Job running really slow because of tFlowToIterate

Anonymous — Thu, 01 Jul 2021 03:57:06 GMT

Hello,

Are you using any t<DB>connection component in your workflow?

Generally speaking, the followings aspects could affect the job performance:

1. The volume of data, read a large of data set, the performance will degrade.

2. The structure of data, if there are so many columns on tDBRow, it will consume many memory and much time for transferring the data during the job execution.

3. The database connection, the job always runs better if the database is installed on local, if the database is on another machine, even you are on VPN, you may have the congestion and latency issues.

Best regards

Sabrina

Re: Job running really slow because of tFlowToIterate

sbarjatya — Thu, 01 Jul 2021 14:49:47 GMT

Hello,

Yes, I am using t<DB>connection.

This job is trying to read only 38,000 rows.
I ran lot of jobs with much more columns in tDBRow, never got this issue.
Again, I ran more complex jobs with the same database connection residing on another machine but never got this issue.

It seems to me is its more tFlowToIterate related issue.

Re: Job running really slow because of tFlowToIterate

Anonymous — Fri, 02 Jul 2021 05:47:01 GMT

Hello,

tFlowToIterate in fact stores the (object) data in its globalMap which could need more memory to keep performance depending on the flow.

Writing all your data in one file will be much faster, than writing a file for each line.

Best regards

Sabrina

Re: Job running really slow because of tFlowToIterate

gjeremy1617088143 — Fri, 02 Jul 2021 06:32:08 GMT

Hi @Shaily Barjatya , can you share the sql request of the tDBInput and TDBRow?

Send me Love and Kudos