topic Re: Data duplication handling - scenario in Talend Studio

Data duplication handling - scenario

Anonymous — Sat, 16 Nov 2024 06:38:21 GMT

Hi,

Can I get solution for the below scenario.

I am getting two records for a single product, only difference is one column value, but I need both values to be loaded into target on different columns. Duplication removal won't help me.

It would be great if I get an idea of optimal solution, because huge number of records coming from source.

Talend version: Open Studio for Big Data 7.0.1

Target: Salesforce

source:

ProductName	ImageType	imageurl
Nokia 610	LARGE	/devices/generic-phone.png
Nokia 610	SMALL	/devices/5145.jpg

Required Data on Target: (based on 'Image Type' above)

ProductName	main_image_URL	thumbnail_image_URL
Nokia 610	/devices/generic-phone.png	/devices/5145.jpg

I tried by writing expression in tmap, but output is not as expected (below), it will create duplicate in target

ProductName	main_image	thumbnail_image
Nokia 8110	/devices/generic-phone.png
Nokia 8110		/devices/5145.jpg

Re: Data duplication handling - scenario

Anonymous — Fri, 08 Feb 2019 12:39:59 GMT

Hi,

Why don't you take it as two data sets at the bginning and then do an inner join?

Dataset one :- where ImageType ="LARGE"

ProductName	ImageType	imageurl
Nokia 610	LARGE	/devices/generic-phone.png

Dataset two:- where ImageType="SMALL"

ProductName	ImageType	imageurl
Nokia 610	SMALL	/devices/5145.jpg

Now, do inner join based on Product Name and map the values two output flow in tMap as two variables.

Mapping in the tMap

ProductName -> ProductName

imageurl(small) -> thumbnail_image_URL

imageurl(big) -> main_image_URL

This should give the desired output.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: Data duplication handling - scenario

manodwhb — Fri, 08 Feb 2019 12:50:29 GMT

@Vibin_CT ,check below job.

Re: Data duplication handling - scenario

Anonymous — Fri, 08 Feb 2019 13:06:36 GMT

Have u tried use tUniqueRow component before tMap?

I think what u need is something like this?

The desired output should be something like this

Re: Data duplication handling - scenario

Anonymous — Mon, 11 Feb 2019 09:20:00 GMT

Hi @nthampi ,

Thank you very much for your solution.

In-order to use your method, I want to load data into a intermediate database table, because the problem which I mentioned is not directly from source data, it is an intermediate data coming after doing so many transformations and I was unable to load this data into MySQL db(staging) due to MySQL table size limitation (some columns contains appended data and size is huge). So I am using thashoutput component.

I am also planning to use tfileoutputdelimited instead of thashoutput due to huge number of records and record size. Can you suggest me, which component is better to use by considering memory and performance.

Re: Data duplication handling - scenario

Anonymous — Mon, 11 Feb 2019 09:23:45 GMT

Hi,

Considering your use case, park the data as interim file using tfileinputdelimited.

Also increase the memory parameters (Xms and Xmx) of the job for better job performance.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

Re: Data duplication handling - scenario

Anonymous — Mon, 11 Feb 2019 10:35:12 GMT

Thanks @nthampi !!