topic CDC on files in Talend Studio

CDC on files

Sid3 — Tue, 03 Oct 2017 03:10:18 GMT

Hi Folks,

Anyone has idea how to acchieve the CDC between 2 files. I am looking for direct/indirect solutions without loading data to database.

As my database is Redshift and currently no CDC feature supported by talend for Redshift database.

Kindly post your thoughts

Re: CDC on files

brama — Tue, 03 Oct 2017 13:24:09 GMT

Hi sid,

Try doing a left outer join with source as primary and target as the ref link based on key cols. All the null values will be as insert. And others wil be an update. Nite ref should be a full file.

Thanks,
Ram

Re: CDC on files

Vinlogs — Tue, 03 Oct 2017 14:17:35 GMT

I agree with brama! We have redshift as output and we do the same. Thanks.

Re: CDC on files

Sid3 — Wed, 04 Oct 2017 03:58:22 GMT

Thank you guys for reply. I am aware of that approach.

What i am looking is is there any component/custom code if anyone developed for CDC comparisons between files.

Also i need to find the I,U,D,New records between both files.

Re: CDC on files

Vinlogs — Wed, 04 Oct 2017 14:21:02 GMT

I have not researched on any custom components on CDC as we were able to achieve what we were looking for (Inserts and updates) and deletes is something which we didn't needed as part of our requirements.