Job running really slow because of tFlowToIterate

[sbarjatya]

I am trying to delete rows in a target table using value from source table, using tFlowToIterate (screenshot attached). Throughput for which is really low = 0.19 rows/s. I am using cursor in tDBInput and did enable parallel execution for iterate row. But, no improvement so far. Can anyone please advise how can I improve efficiency of this simple job or if anyone can suggest any alternative approach.

Any leads would be appreciated.

[Sourav_Roy]

Hi,

Please increase your JVM Parameters according to your system configurations:

(JVM Settings are present in Run -> Advanced Settings -> Use specific JVM arguments.

[sbarjatya]

Its already running on Xms32G and Xmx64G

[Anonymous]

Hello,

Are you using any t<DB>connection component in your workflow?

Generally speaking, the followings aspects could affect the job performance:

1. The volume of data, read a large of data set, the performance will degrade.

2. The structure of data, if there are so many columns on tDBRow, it will consume many memory and much time for transferring the data during the job execution.

3. The database connection, the job always runs better if the database is installed on local, if the database is on another machine, even you are on VPN, you may have the congestion and latency issues.

Best regards

Sabrina

[sbarjatya]

Hello,

Yes, I am using t<DB>connection.

1. This job is trying to read only 38,000 rows.
2. I ran lot of jobs with much more columns in tDBRow, never got this issue.
3. Again, I ran more complex jobs with the same database connection residing on another machine but never got this issue.

It seems to me is its more tFlowToIterate related issue.

[Anonymous]

Hello,

tFlowToIterate in fact stores the (object) data in its globalMap which could need more memory to keep performance depending on the flow.

Writing all your data in one file will be much faster, than writing a file for each line.

Best regards

Sabrina

[gjeremy1617088143]

Hi @Shaily Barjatya​ , can you share the sql request of the tDBInput and TDBRow?

Send me Love and Kudos