topic How to Implement SCD Type 2 WIthout using Lookup with Target in Talend Studio

How to Implement SCD Type 2 WIthout using Lookup with Target

Anonymous — Thu, 11 Apr 2019 12:43:47 GMT

Hi All,

I want to implement SCD Type 2. Simply i can do with Lookup on Target table. But My Target Table is having huge records. So i think if i will perform lookup on this again performance will become very bad. Because target table is having billions of records.

Is there any way to implement SCD Type 2 without using lookup.

Regards,

Vishal Vats

+91-9650236023

Re: How to Implement SCD Type 2 WIthout using Lookup with Target

akumar2301 — Thu, 11 Apr 2019 13:18:58 GMT

My suggestion

1) create a temporary table/file with all active Keys and do lookup on this ( instead of full table) in ( tMap use "store on disk")

2) At output side of tMap , create separate flow for insert and update . use Blukcomponent instead of tDBoutput.

3) check if your table is partitioned ( I assume yes as it has billions of record). if yes work on partition by partition.

Re: How to Implement SCD Type 2 WIthout using Lookup with Target

manodwhb — Fri, 12 Apr 2019 05:52:35 GMT

@vishuoo1 ,

please find the below links.

http://www.deepinopensource.com/scd-2-implementations-in-talend/

http://talenddev.blogspot.in/2016/03/apply-scd-without-using-scd-component.html