topic Re: In Big data Job, look up Does it work? in Talend Studio

In Big data Job, look up Does it work?

mahesh_Dev — Sat, 16 Nov 2024 07:01:54 GMT

Hello everyone,

Look up in big data, will it works or not?

i developed the big data job which lookup from red shift.

but its giving me duplicated in red shift. how to achieve ?

waiting for your reply..

Anonymous — Mon, 17 Dec 2018 11:37:12 GMT

Hi,

Could you please refer the below link for your query?

Also could you please share the screenshot of your job flow and the tmap lookup in detail?

Warm Regards,
Nikhil Thampi

Please appreciate our members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

vapukov — Mon, 17 Dec 2018 11:52:18 GMT

HI Mahesh

batch job run in parallel and each worker make lookup independently
as a fast solution you could reduce the number of workers to 1

mahesh_Dev — Tue, 18 Dec 2018 02:44:46 GMT

@vapukov could you explain me clearly how to do this

mahesh_Dev — Tue, 18 Dec 2018 02:54:19 GMT

Big Data Spark Job:--

I am using tredshiftinput for lookup

problem is i am unable to do lookup from red shift data

its inserting the duplicates

any solution how to do this