In Big data Job, look up Does it work?

mahesh_Dev · ‎2018-12-17

Hello everyone,

Look up in big data, will it works or not?

i developed the big data job which lookup from red shift.

but its giving me duplicated in red shift. how to achieve ?

waiting for your reply..

Anonymous · ‎2018-12-17

Hi,

Could you please refer the below link for your query?

Also could you please share the screenshot of your job flow and the tmap lookup in detail?

Warm Regards,
Nikhil Thampi

Please appreciate our members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

vapukov · ‎2018-12-17

HI Mahesh

batch job run in parallel and each worker make lookup independently
as a fast solution you could reduce the number of workers to 1

mahesh_Dev · ‎2018-12-17

@vapukov could you explain me clearly how to do this

mahesh_Dev · ‎2018-12-17

Big Data Spark Job:--

I am using tredshiftinput for lookup

problem is i am unable to do lookup from red shift data

its inserting the duplicates

any solution how to do this

Capture.JPG