tMap lookup by conditional query

[Anonymous]

Hi,
I will explain my need with a small example. I have a datamodel thats contains a table sources and a table items. items has a foreign constraint to sources. In my job I iterate over every source and getting information from the internet. This information will be saved to the table items setting the souce_id that is used in the current iteration.
I use the tMap inner join function for finding items, that are not already inserted. For this I join over an title column for the table items and the internet information. Every reject column is not in the table items and could be insert. The table items is huge. I need an alternative for simply loading all elements for the lookup.
Now I like to only lookup items for the current source. Beside that, I like to insert item even though the title is identical for two different sources, what will not work with my first approach.
How do I add a dynamic condition to a lookup statement for tMap?

Thanks in advanced,
Danny

[Anonymous]

A screen shot of your job would help us understand your job structure better and allow us to (possibly) provide a good solution for you.

[amaumont]

There is a new added feature called "Reload lookup" for each tMap's lookups into the releases 3.1 RC.
This feature allow the user to create global variables into tMap, then user call these variables into the lookup subjob, at each line of the main flow the lookup is reloaded.

[Anonymous]

@amaumont: I do not understand how this could help me?
So here is my job. This job is run by a master Job. On the right side you find to MySQL Input elements. Now I like to look only these elements up that belongs to the source in the current iteration of v_source_w_param.
In this job the lookup will based on all news elements. (see upon)

[Anonymous]

Hi,
I still do not know how to limit the lookup process. There is a table with 24Mio Entries which I have to lookup. There is no way, and I really want another way, to load all the data in the memory upfront.
How will it be possible to have a lookup without caching something or using a dynamic where clause?

Thanks,
Danny

[Anonymous]

Hi;
i had the same issue a while ago. Create a function on your db which has your lookup values as input and the result as output. You can call it from talend. It works like magic.
Regards
Fatih