How to Implement SCD Type 2 WIthout using Lookup with Target

[Anonymous]

Hi All,

I want to implement SCD Type 2. Simply i can do with Lookup on Target table. But My Target Table is having huge records. So i think if i will perform lookup on this again performance will become very bad. Because target table is having billions of records.

Is there any way to implement SCD Type 2 without using lookup.

Regards,

Vishal Vats

+91-9650236023

[akumar2301]

My suggestion 

1) create a temporary table/file with all active Keys and do lookup on this ( instead of full table) in ( tMap use "store on disk")

2) At output side of tMap , create separate flow for insert and update . use Blukcomponent instead of tDBoutput.

3) check if your table is partitioned ( I assume yes as it has billions of record). if yes work on partition by partition.

[manodwhb]

@vishuoo1 ,

please find the below links.

http://www.deepinopensource.com/scd-2-implementations-in-talend/

http://talenddev.blogspot.in/2016/03/apply-scd-without-using-scd-component.html