topic Re: Multiple tMap input fields mapped to a single lookup table instance. in Talend Studio

Multiple tMap input fields mapped to a single lookup table instance.

Anonymous — Sat, 16 Nov 2024 11:00:08 GMT

Hi,
I have a tMap with numerous fields of the same type, for example a medical diagnosis code. The medical diagnosis codes must match the valid codes in a MongoDB lookup table. My problem is that there are many such fields in the input data.
I am able to drop "Many" tMongoDbInput components in my job and do each individual lookup against a different lookup table instance. However, this presents several problems. The first obvious is that I now have a real mess with components on top of components in the job. The second is that each lookup loads a separate copy of the data. That is not such a big problem for smalll tables, but I have some fields that require very large lookup tables.
Is there any way to utilize a single lookup instance so I do not have to have bunches and bunchs of duplicated lookup tables, and also to avoid loading the lookup tabel data over and over.
Thanks so much in advance!
Charlie

Re: Multiple tMap input fields mapped to a single lookup table instance.

Anonymous — Sat, 10 Oct 2015 03:30:12 GMT

Hi
You can load the lookup table one time and cache the data in memory with tHashOutput component, read the data from memory with tHashInput when needed. eg:
tMongoInput--main--tHashOutput
|
onsubjobok
|
tFileInputDelimited--main--tMap---out1....
|
lookup
|
tHashInput

Re: Multiple tMap input fields mapped to a single lookup table instance.

Anonymous — Tue, 13 Oct 2015 22:44:14 GMT

Thanks!
I have the tHashInput and tHashOutput working nicely. That solves the multiple load problem.
The next problem is that I need to use N instances of the tHashInput. In my case that means 16 tHashInput instances and and an aditional 12 secondary loolkups/tHashInput instances. There are also 7 other data sources. This is really hitting the limits of dealing with the graphical design tool like Talend. I have looked and looked but I have not found any way to cope with that many components.
Do you know of any way to reutilize a single tHashInput component so I do not need so many components and connections in my job?
Can I write Java code to tell Talend how to cope with this?
Are there best practices regarding how to decompose Talend jobs to make this type of situation more manageable?
Thanks,
Charlie