topic Re: How to use same Input multiple time in tMap in Talend Studio

How to use same Input multiple time in tMap

Anonymous — Sat, 16 Nov 2024 14:35:16 GMT

Hi,
I'd like to know if there is a way to use the same input multiple times in a tMap in JAVA
Let's consider the following example : I have a bunch of records from a FileInput each having 3 customerKey. In order to transform these records for a SQLDB, I need to make a lookup on my sql Customer Table. But since I have 3 customerKey and I can only one custmerKey by Customer Table Lookup, I currently need to have 3 Customer Lookup, each querying the BD for THE SAME DATA which has a big performance problem. It should be possible to use the same lookup to avoid duplicating requests AND data in memory.
Anyone got a solution? If it's not possible, it should definately by a feature!!
Thx,

Re: How to use same Input multiple time in tMap

amaumont — Wed, 04 Jul 2007 20:17:44 GMT

We have this solution :
- load your DB data meant to lookups in a temporary file (LOOKUP_FILE)
- load the same file LOOKUP_FILE for all your lookups
(see snapscreen)
-> with this method you should gain performance

Re: How to use same Input multiple time in tMap

Anonymous — Wed, 04 Jul 2007 20:37:11 GMT

Thanks for your answer.
Yeah I thought about that but it was so innelegant that I thought there had to be another solution!
Is this a feature for sometime to be able to use the same input for multiple lookup ? It's already all in memory, I should not have to send it in a file then read a file 3 times... It's even more annoying if we deal with a big amount of data!
Any feedback on that?
Thx a lot for your answer!

Re: How to use same Input multiple time in tMap

_AnonymousUser — Tue, 11 Sep 2007 13:52:13 GMT

If the problem is only about performance, you sould use the tETL* widget, those infact generate the intere sql statement and thant send that to the server db to execute, this surely improve performance, infact joins are execute on the server db instead of the client machine.

Re: How to use same Input multiple time in tMap

c0utta — Wed, 12 Sep 2007 00:17:01 GMT

When I'm dealing with smaller sets of data for lookups, I use the hidden components tArray and tArrayIn. Make them available in TOS by Window->Preferences...>Talend->Designer and check "Display hidden components(need restart)". As instructed, then restart TOS.
Obviously there's an array sitting behind the scenes so be careful with memory.
Cheers,
c0utta

Re: How to use same Input multiple time in tMap

Anonymous — Wed, 12 Sep 2007 10:33:20 GMT

When I'm dealing with smaller sets of data for lookups, I use the hidden components tArray and tArrayIn.

Impressive! I thought I was the only one to use such tricks 🙂 So I must be careful when I'll rename tArray into tArrayOut (to make it obvious it fills an array, while tArrayIn reads it)

Re: How to use same Input multiple time in tMap

c0utta — Thu, 13 Sep 2007 00:32:04 GMT

Now that tArray* are public ( ), can the array contents be manipulated within tPerl (or tJava for the Java guys) that doesn't violate the principles of TOS?
Being able to do such things as add/delete rows, or even create other arrays that could be used within a tArrayIn would be very beneficial.
Cheers,
c0utta

Re: How to use same Input multiple time in tMap

Anonymous — Tue, 17 Feb 2015 16:57:44 GMT

Hi,
Not sure we have a clear answer. Let me know if I understood well: the recommended design for multiple keys lookup is to use several instances of tDBInput (e.g. tMySQLInput) connected on the same tMap and to cascade the lookup flow?