topic Re: [resolved] Data Migration between two MySQL Host with logging in Talend Studio

[resolved] Data Migration between two MySQL Host with logging

Anonymous — Sat, 16 Nov 2024 13:43:13 GMT

Hi guys.
This is my 1st Post and i'm very happy to join with you people.
Please can u give me some suggestion(that going to help to design my new job) for my scenario...
Scenario is:
i need to migrate entire DB from one MySQl source to other MySQl target DB.(Both are in diff host). Lots of tables are there to move..
At first now I'm trying to migrate only one table i.e
sourceDB.tbl_surveys_to_be_sent to targetDB.tbl_surveys_to_be_sent (" tbl_surveys_to_be_sent " is the table having about 8241723 rows)

Here i don't want to move the entire data because the source table getting update all the time so i want to migrate little by little based on some criteria. The idea is like migrating data based on two parameters that going to give start and end range of the data set . And I want to keep track of migration process... so pls tell me what are all the things want to be log for this job..

So please tell me how the design come for this job ... and What are all the things i want to take care ....
Is this following is enough
tMysqlInput_1 -----------main--------> tMap_1 -------main-------> tMysqlOutput_1

ThanX in Advance ,.... If anything pls let me know....

Re: [resolved] Data Migration between two MySQL Host with logging

Anonymous — Mon, 26 Oct 2009 03:05:09 GMT

Hello
First, just let you know, we have added a new 9223 with which we can migrate easily a whole database on Talend Integretion Suite.

Here i don't want to move the entire data because the source table getting update all the time

If you know the start and end range, you can use tFixdFlowInput to define some column with fixed value like start, end, the number of rows inserted into target table and log them into a file.
eg:
tMysqlInput_1 -----------main--------> tMap_1 -------main-------> tMysqlOutput_1
|
onSubJobok
|
tFixedFlowInput-->tFileOutputDelimited
on tFixedFlowInput, define columns:
column name: value
start: start range
end: end rage.
NB: ((Integer)globalMap.get("tMysqlOutput_1_NB_LINE_INSERTED"))
((Integer)globalMap.get("tMysqlOutput_1_NB_LINE_INSERTED")) is a global var, which counts the number of rows inserted into target table.
Best regards
shong

Re: [resolved] Data Migration between two MySQL Host with logging

Anonymous — Mon, 26 Oct 2009 07:28:18 GMT

Hi Shong,
Thanks for ur quick reply... i think its enough for now to make log.
But i don't know anything about these bugtracker stuff.... " feature 9223 " How can i get ideas abt these...

Re: [resolved] Data Migration between two MySQL Host with logging

Anonymous — Tue, 27 Oct 2009 02:28:01 GMT

Hello

But i don't know anything about these bugtracker stuff.... " feature 9223

It doesn't matter. As the feature9223 shows, this feature will be avaliable on TIS3.2.1.
Best regards
shong

Re: [resolved] Data Migration between two MySQL Host with logging

Anonymous — Tue, 27 Oct 2009 07:03:12 GMT

Thanks for the info shong.