Multiple lookups on same file in talend big data batch jobs

Anonymous — Sat, 16 Nov 2024 06:14:55 GMT

I am using Talend data fabric 7.1 enterprise edition.

I am trying to create a Big Data Batch job and using Spark framework.

I have a look up file. it has 2 columns, lookup_id, Description.

I have another large file that has fact data. It also has columns like STATUS_ID, SUB_STATUS_ID, REASON_ID, REQUEST_TYPE_ID, CATEGORY_ID.

If I were dealing with a database, i would alias my look up file, 5 times and join each column to lookup_id and get the description and my query would look like this.

select f.col1, f.col2, f.col3,

a1.description as Status,

a2.description as sub_status,

a3.description as reason_desc,

a4.description as request_type,

a5.description ass category_descr

from fact_table f

left outer join lookup_table a1 on f.STATUS_ID = a1.lookup_id

left outer join lookup_table a2 on f.SUB_STATUS_ID = a2.lookup_id

left outer join lookup_table a3 on f.REASON_ID = a3.lookup_id

left outer join lookup_table a4 on f.REQUEST_TYPE_ID = a4.lookup_id

left outer join lookup_table a5 on f.CATEGORY_ID = a5.lookup_id

but when I try to do the same thing using single tMap and 5 alias for lookup file, I get an error at tMap saying "All lookup tables must have the same expressions. This expression doesnt exist in other lookups". Talend Big Data Batch is only allowing me to use one look up per tMap component. I know it can be done easily on Standard talend job but I have functionality that needs me to use talend big data with spark. Is there any way out, any other component that will help me avoid 1 tMap per a look up column.

Re: Multiple lookups on same file in talend big data batch jobs

vapukov — Wed, 27 Mar 2019 00:09:00 GMT

Hi,

unfortunately, the text is unreadable 🙂

could you clean it and attach some pictures

based on last part of text - more than one lookup with tMap, why not?

tMap in Spark batch jobs contain some issues, but from the question is not clean - is it related or not to your problem

regards, Vlad

Re: Multiple lookups on same file in talend big data batch jobs

Anonymous — Thu, 28 Mar 2019 23:37:01 GMT

Thank you for the reply. That was my first post. Did not expect the text mess up.

I edited my post and made it more readable.

Will add some snapshots of my talend job in a couple of hours.

topic Re: Multiple lookups on same file in talend big data batch jobs in Talend Studio

Multiple lookups on same file in talend big data batch jobs

Re: Multiple lookups on same file in talend big data batch jobs

Re: Multiple lookups on same file in talend big data batch jobs