how to speedup the process for loading data into lookup process if number of records are more than 1mn?

[Anonymous]

Hi Team,

I have a performance issue with a Talend job. Below is a screen of my Talend job

Here as I have seen that its taking around 20 min even if their is not a single record is getting inserted or update.

More than 50% of time is taking only in loading lookup records from tables. I have also tried 'store on disk' option in tmap, but still their is no improvement in performance.

Your advice or suggestions will be much appreciated and welcomed....

Thanks

Kailash

[TRF]

Hi,

What is the elapsed time when reading these tables from outside of TOS using the same SQL query?

[Anonymous]

Hi there,

If the lookup tables are all in the same database, and it's practical, then you'd be much better off doing your joins and retrieving the data in your initial query, as it'll be much more efficient.

Obviously I don't have the full picture in this case, but was troubleshooting what appears to be a job with a similar structure to your own, for someone else in a different DI tool (not Talend) and by doing the lookups up front, the job went from taking 3+ hours to around 15 minutes.

Of course, by doing this, you're going to lose a lot of the flexibility Talend offers, and massively impact the maintainability of your job for the worse.

Regards,

Chris