topic Re: Job optimization in Talend Studio

Job optimization

Tech8 — Sat, 16 Nov 2024 06:18:28 GMT

In terms of optimization what is better? Use a global variable at the start of the job and then call it in the tMaps or use several querys (tDBInput) to get the data to this tMaps. I have to use it like 8 tDBinput components only to get the same data (is a small query with 2 columns) in different parts of my job.

I supose global variable is better option but i want to know if i will some problems when calling a global var inside the tMap.

Thanks in advance

Re: Job optimization

Anonymous — Mon, 18 Mar 2019 15:59:06 GMT

Hi,

If it is a small dataset and you are using it multiple times, you can store it in global variable or hash maps.

In your use case, your current approach looks fine rather than querying database multiple times (since each query of DB has an associated resource cost).

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: Job optimization

DataTeam1 — Mon, 18 Mar 2019 16:13:58 GMT

I think you can use tHash... compnents (tHashInput)

This compnents are for creating temporary table within a job which you can use many times.

You can get data from source (db, file etc) and load it into hash(temporary) table and use it many times instead of loading data many times from source.

To see it in your palete you have to unhide them in: File => Edit project properties => Designer => Palete setings => Technical

Re: Job optimization

Tech8 — Wed, 20 Mar 2019 08:39:57 GMT

I tried both solutions and work fine for me and even looks like having several tHash components dont affect the performance of the job.

I will take the global var as the solution because im looking for use less components but as i said both solutions work.

Thanks so much