topic Re: Lookup multiple files in Talend Studio

Lookup multiple files

Anonymous — Mon, 09 May 2016 11:56:23 GMT

Hi, As a requirement I have to join 16-20 inputs as lookup. what is the most efficient way to perform lookup ?

Re: Lookup multiple files

Anonymous — Mon, 09 May 2016 12:03:59 GMT

It's difficult to say without seeing your requirement, data structure, and volume; however, I'd say your starting point is to simply join your data using either tMap or tJoin.

Re: Lookup multiple files

Anonymous — Tue, 10 May 2016 08:48:13 GMT

Hi, requirement is something like below-
Input file- file0.dat
15 lookup files- file1.dat,file2.dat....file15.dat
lookup key is same for all lookups, say process_id
now I am fetching certain columns from each lookup file.
1) If I add input file as main flow and rest all files as lookup on a single tMap, there is a big performance hit.
2) Another way I could think is, perform these lookups on multiple tMaps instead of single tMap. (not a very good option though)
So I am looking around the other possible ways to perform this. Thanks in advance.

Re: Lookup multiple files

Anonymous — Tue, 10 May 2016 09:00:09 GMT

I still think it comes down to your data volume and also your expectation.
The simple answer is to join them in a single tMap or try tJoin.
If one or more look-up files is causing an unacceptable performance issue, then you can look for specific ways to improve this.