topic Re: Replicating databases while keeping the differences in data minimum in Talend Studio

Replicating databases while keeping the differences in data minimum

Anonymous — Tue, 07 Feb 2017 10:27:36 GMT

Hi,
My requirement is to mirror all my data in sql server to mysql. I have 200+ tables in Sql server and all these 200+ tables need to be present in mysql as well.
So, my plan is to create a job for every table, that takes data from the source to destination. I will have a column called lastmodifieddatetime in every source table and a generic table that will determine the time of the last run.
All these 200+ jobs will run in parallel (multithread) and based on the data change between last run and the lastmodifieddatetime column data(in source) is updated or inserted in the destination.
Every job will have "insert or update" set in the destination component.
The expectation is to keep the data difference as minimum as possible between the source and destination. And the entire job consisting of 200+ jobs (running in parallel) must complete within a few seconds
Is this approach that I have taken the right one? Is there a better way to do it?
Any ideas on this would be of great help
Thanks in advance
Rathi

Re: Replicating databases while keeping the differences in data minimum

Anonymous — Thu, 09 Feb 2017 06:24:51 GMT

Hi,
From your description, it seems that you are looking for how to migrate DB in talend? If so,
With talend subscription solution, you can migrate the whole database in a job in DB level. Thanks to "dynamic schema" feature.

With Talend open source, you have to move the tables one by one.
Best regards
Sabrina

Re: Replicating databases while keeping the differences in data minimum

Anonymous — Thu, 09 Feb 2017 06:28:07 GMT

Hi,
I am using Talend Open Studio. This is only thing that I can use.
And my job should understand which data changed and update/insert the data accordingly. So, I have the idea as I have told in my original post
Is this good?
Thanks