how to get only modified data or changed data from mysql using talend

[Anonymous]

Hi ,

Can anyone please explain how can I get only changed data.

I have read somewhere that if we apply CDC, we can get only changed or modified data but the CDC is not applicable in open source.

So, I have run one job in talend tool, output I have placed in MySQL.

if I do SCD , I will get both history and modified data.

but I want only changed data in my output.

what can I do?

please anyone help me in this...

[Anonymous]

If you store your data in a SCD you can query for only changed data. Otherwise you will have to keep a "change date" field on your table (or linked to your table's primary key) and refer to that.

[Anonymous]

please could you explain me clearly

[Anonymous · Accepted solution]

With a slowly changing dimension you should be able to easily query recently modified records. All you would need to do is keep a log (in a different table) of the last time you had run your job and then use the effective dating in the SCD with your last run date. 

Alternatively add a change_date column to your database table and update it when you make a change to a row