How to use same Input multiple time in tMap

[Anonymous]

Hi,
I'd like to know if there is a way to use the same input multiple times in a tMap in JAVA
Let's consider the following example : I have a bunch of records from a FileInput each having 3 customerKey. In order to transform these records for a SQLDB, I need to make a lookup on my sql Customer Table. But since I have 3 customerKey and I can only one custmerKey by Customer Table Lookup, I currently need to have 3 Customer Lookup, each querying the BD for THE SAME DATA which has a big performance problem. It should be possible to use the same lookup to avoid duplicating requests AND data in memory.
Anyone got a solution? If it's not possible, it should definately by a feature!!
Thx,

[amaumont]

We have this solution :
- load your DB data meant to lookups in a temporary file (LOOKUP_FILE)
- load the same file LOOKUP_FILE for all your lookups
(see snapscreen)
-> with this method you should gain performance

[Anonymous]

Thanks for your answer.
Yeah I thought about that but it was so innelegant that I thought there had to be another solution!
Is this a feature for sometime to be able to use the same input for multiple lookup ? It's already all in memory, I should not have to send it in a file then read a file 3 times... It's even more annoying if we deal with a big amount of data!
Any feedback on that?
Thx a lot for your answer!

[_AnonymousUser]

If the problem is only about performance, you sould use the tETL* widget, those infact generate the intere sql statement and thant send that to the server db to execute, this surely improve performance, infact joins are execute on the server db instead of the client machine.

[c0utta]

When I'm dealing with smaller sets of data for lookups, I use the hidden components tArray and tArrayIn. Make them available in TOS by Window->Preferences...>Talend->Designer and check "Display hidden components(need restart)". As instructed, then restart TOS.
Obviously there's an array sitting behind the scenes so be careful with memory.
Cheers,
c0utta

[Anonymous]

When I'm dealing with smaller sets of data for lookups, I use the hidden components tArray and tArrayIn.

Impressive! I thought I was the only one to use such tricks 🙂 So I must be careful when I'll rename tArray into tArrayOut (to make it obvious it fills an array, while tArrayIn reads it)

[c0utta]

Now that tArray* are public ( ), can the array contents be manipulated within tPerl (or tJava for the Java guys) that doesn't violate the principles of TOS?
Being able to do such things as add/delete rows, or even create other arrays that could be used within a tArrayIn would be very beneficial.
Cheers,
c0utta

[Anonymous]

Hi,
Not sure we have a clear answer. Let me know if I understood well: the recommended design for multiple keys lookup is to use several instances of tDBInput (e.g. tMySQLInput) connected on the same tMap and to cascade the lookup flow?