topic Re: Big Data Spark job - Inner join in Talend Studio

Big Data Spark job - Inner join

Anonymous — Sat, 16 Nov 2024 09:52:05 GMT

Hi,

I have to create big data spark job and have two dataset

Fact

Col1	Col2	Col3
101	ABC	123
102	All	234
103	XYZ	345

Dim

Col1	Col2	Col3
1001	XYZ	345
1002	WED	234
1003	WER	123

and need to join above tables based on Col2, Col3. Wherever "All" is there like in Fact table, row 2 where Col2="All", then we have to consider that field for all values. Output would be for above dataset

Col1	Col2	Col3	Col1	Col2	Col3
1001	XYZ	345	103	XYZ	345
1002	WED	234	102	All	234

How to handle this scenario? Confused....

Re: Big Data Spark job - Inner join

cterenzi — Tue, 25 Apr 2017 16:13:27 GMT

In a database query, you would probably do this with a union-- that is, a two step process and then combining the results. One lookup joining on both Col2 and Col3, and one lookup on just Col3 using only the "All" records.

Re: Big Data Spark job - Inner join

Anonymous — Tue, 25 Apr 2017 16:24:08 GMT

not sure what u r saying.. Can you explain it more?

Re: Big Data Spark job - Inner join

cterenzi — Tue, 25 Apr 2017 17:03:42 GMT

Do two lookups. One lookup on both Col2 and Col3, and the second on just Col3 but only for fact records where Col2 = "All"