[resolved] Create SCD without using specific SCD component

[Anonymous]

Hi,
Im almost sure that I saw this solved before in some blog/forum.. but currently Im not able to find it any more.
If I remember well it was using a tMap component.
I need to to make it in this way because my DW is on Redshift and it doesn't have any specific component for it.
Any help/link is apprenticed!
Thanks!
Oscar

[janhess]

Can you be a bit more specific with your requirements.

[Anonymous]

Hi Janhess,
thanks for your answer! will try to explain, but Im looking for a generic solution that later I can apply to different scenarios.
I have a customer SCD dimension.
I need to implement the logic to insert and update depending of the new existent values in the staging table comparing it agains the dim table.
Some columns are Type 1
and others are Type 2
I just need to see some example that I can adapt later to my particular cases
I think i can use a tMap component in order achieve it, but it will be great if I can find some kind of guide that I can use to achieve this.
Thanks!
Oscar

[janhess]

Is this the sort of thing you're looking for?
https://community.talend.com/t5/Design-and-Development/update-insert-AND-delete-old-rows/td-p/100314

[Anonymous]

Hi Janhess,
thanks for you link, at the end I based my solution in this video http://www.youtube.com/watch?v=dH567IFszu4
I think its closer to what Im looking for.
Now my solution works partially good, it works well when I have only SCD1 cases, and also when I have only cases for SCD2, but the tMap seems to not be able to finish when the SCD1 and SCD2 cases needs to be processed in the same run. Also not sure if this related with redshift that may not allow to perform updates from different components.
I can provide the transformation by email if necessary, but I dont know how to attach here in the forum.

Thanks!
Oscar

[Anonymous]

Hi,
I just added a printscreean where you can see how my solution looks like.
In order to avoid Redshift locks in the updates of Type 1 and Type 2, I splited the job in two parts.
(1) In the first one I do new Inserts and Updated Type1
(2) In the second part Im making Inserts and Updates Type 2
Cheers,
Oscar

[Anonymous · Accepted solution]

hi all,
What about using SCD Talend component ?
https://help.talend.com/search/all?query=tMysqlSCD&content-lang=en
exist for other DB (Oracle,Postgres,etc)
hope it help
regards
laurent
PS : Forget !!! don't have read title well (without scd componenet) ...

[Anonymous]

Thanks Laurent for your suggestion, that will be great, but SCD component still don't exist for Redshift, which is what Im using now.
Cheers,
Oscar

[Anonymous]

sorry so ... ask R&D to add it !!
you can really ask it with talend tracker : https://jira.talendforge.org/secure/

[_AnonymousUser]

i used the postgres scd for redshift and it seem to work but the performance is terrible! would you mind sharing your target load rows/s with your "custom" talend type 2 scd for talend? Would appreciate if you can provide some screenshots oscar!