topic Re: Implementing SCD2 using TOS SCD components in Talend Studio

Implementing SCD2 using TOS SCD components

sushantk19 — Sat, 16 Nov 2024 02:41:42 GMT

Hi,

I have to implement SCD2 logic(Insert/update) in my TOS code using SCD2 components. My Target DB is Amazon Redshift. Can you please let me know which TOS component needs to be used here ?Will tMysqlSCD or tOracleSCD work ?

Re: Implementing SCD2 using TOS SCD components

sushantk19 — Mon, 27 Apr 2020 17:00:11 GMT

All inputs on above query. How do I implement SCD2 in Redshift Database?

Re: Implementing SCD2 using TOS SCD components

manodwhb — Tue, 28 Apr 2020 08:16:22 GMT

@sushantk19 , I do not think that we have Redshift SCD component in Talend, you can not use MysqlSCD or tOracleSCD for Redshift Database.

Re: Implementing SCD2 using TOS SCD components

Prakhar1 — Tue, 28 Apr 2020 09:05:16 GMT

Hi,

try to use JDBC component for that.

You can configure the jdbc url for your Redshift connection.

Don't forget to give Kudos

Re: Implementing SCD2 using TOS SCD components

sushantk19 — Tue, 28 Apr 2020 10:51:13 GMT

Presently, I am trying without SCD component. But facing the below issue.Its not even Inserting data in my Target table. Please see the screenshots attached. I already have 2 country ids in my target. Source has 253 unique country ids, so as per the Insert update logic it should insert 251 and update 2 records.

Job_Design_SCD2.png
promotion_order.csv
tmap_output.png

Re: Implementing SCD2 using TOS SCD components

sushantk19 — Tue, 28 Apr 2020 17:44:47 GMT

I tried both of them :

Please see the below error:
Exception in component tDBSCD_2 (job_CORE_000_Country_test)
java.sql.SQLRecoverableException: IO Error: Got minus one from a read call

with tMySQLSCD getting communication Link error although all the DB connections are correct.

Re: Implementing SCD2 using TOS SCD components

sushantk19 — Tue, 28 Apr 2020 19:06:03 GMT

what should i give in JDBC URL? I have very limited knowledge on Java. If possible, can you share your job design?

Re: Implementing SCD2 using TOS SCD components

Prakhar1 — Wed, 29 Apr 2020 06:33:55 GMT

I have never created job like this so can't share the job design now.

The JDBC URL will look like this :

jdbc:redshift://<endpoint>:<port>/<database>

You can also go through this link to create connection to the JDBCSCD component

https://docs.aws.amazon.com/redshift/latest/mgmt/configure-jdbc-connection.html

Re: Implementing SCD2 using TOS SCD components

sushantk19 — Wed, 29 Apr 2020 12:01:50 GMT

@Prakhar @manodwhb thanks!. I used the postgresqlSCD component and worked perfectly fine for my requirement! Is there any issue with this component for implementing SCD2?

Re: Implementing SCD2 using TOS SCD components

manodwhb — Wed, 29 Apr 2020 14:20:19 GMT

@sushantk19 , you can implement scd2 with postgresqlSCD component.