Implementing SCD and writing to Redshift

leanne_ball — Sat, 16 Nov 2024 01:36:28 GMT

Hi,

We have a need to implement SCD methodology and have the output write to Redshift using TOSBD.

We originally tried to use the

tDBSCD component, but found the performance to be very poor (~.15 rows/sec).

Then switched to using a custom tMap with four outputs (insert new, type 1 update existing, type 2 insert new, and type 2 insert existing), just like is shown here:

https://www.youtube.com/watch?v=dH567IFszu4. Performance was better there, but still not great (~9 rows/sec).

Recently heard that we might have better luck with

tJDBCSCDELT component, instead, but haven't taken a look at that one yet.

Rather than continuing to try different methods like this, curious if someone will share what components/design they're using to write to Redshift and what kind of performance you see with that method.

topic Implementing SCD and writing to Redshift in Talend Studio

Implementing SCD and writing to Redshift