topic Re: data from online to offline update in Talend Studio

data from online to offline update

modara — Sat, 16 Nov 2024 00:26:22 GMT

Dear Concern,

I have production server with MS SQL Server (Online) that have more than 20,00,000 data and I take these data to MySQL (Offline).

in MySQL I put Action On table: 'create table' and Action on data: 'insert' and take all 20,00,000 data from MS SQL to MySQL

as MS SQL Server is online and new data keep inserted so I need to take that new data to MySQL not the whole 20,00,000 + newly inserted

So I change MySQL Action On table: Default and Action on data: insert and Update

Then I click RUN (Job.....) it start taking whole data from MS SQL (

20,00,000 + newly inserted

) to My SQL

Not just the newly inserter data from MS SQL to MySQL and this causes server become slow performing

Please help

Re: data from online to offline update

Alaor_Silva — Mon, 15 Mar 2021 12:01:07 GMT

Hi @Not defined Not defined,

I don't know if I understood it correctly,

Do you have a primary key in your MySQL?

The Primary Key is the same as in MsSQL?

Re: data from online to offline update

modara — Tue, 16 Mar 2021 04:07:50 GMT

Hello @Alaor Silva

I don't have any primary key to production server MS SQL Server (online)

but when i take data to MY SQL (Offline) I assign primary key to MY SQL server

I configured MySQL Action On table: Default and Action on data: insert and Update

and when I click RUN it takes full data from MS SQL to MY SQL not just the only updated data

Re: data from online to offline update

modara — Sat, 20 Mar 2021 05:07:34 GMT

Dear Concern,

is there any one have the solution

I really need to know how I configure so when I click RUN it won't take whole data from production server just the update or newlly added data to MYSQL (Offline Data base)

thanks

Re: data from online to offline update

modara — Sat, 20 Mar 2021 14:36:42 GMT

@Xiaodi Shi ,

good day,

I really need help for this newly or only update data from online to offline data base. I dont want full or all data from online to offline.

Your suggestion or advice is appricable

regards,

Re: data from online to offline update

Anonymous — Mon, 22 Mar 2021 04:22:37 GMT

Hello,

If I understand your requirement very well, it seems you are looking for changed data only?

In this way, we could capture the changed data and only load these changed data into target table to achieve table sync, you can compare tables by using tMap.

The work flow should be:

Target DB-->tMap(inner join for input and set the "Catch lookup inner join" as true)-->output

Source DB-->

The output will be the changed data.

Feel free to correct me if I misunderstand your requirement.

Best regards

Sabrina

Re: data from online to offline update

modara — Mon, 22 Mar 2021 06:26:38 GMT

Dear Sabrina,

Thanks for your reply. Yes your are right. First time I want full data and next time or every time I want changed data to come from online production server database to offline database

as you advise to use tMAP then for full data I have to use without tMAP and for changed data I have to add tMAP

why cant I change the configuration in tDBOUTPUT1---- Action On table: Default and Action on data: insert and Update

by assigning a primary key

you advise is really appreciable, if you have any other solution except tMAP (inner join)

regards,

Re: data from online to offline update

modara — Mon, 22 Mar 2021 06:50:48 GMT

Dear Sabrina,

"why cant I change the configuration in tDBOUTPUT1---- Action On table: Default and Action on data: insert and Update"

I mean by changing tDBOUTPUT1 option/ configuration to ---- Action On table: Default and Action on data: insert and Update I should be able to take only changed data to offline. am I right?

regards,

Re: data from online to offline update

Anonymous — Mon, 22 Mar 2021 07:25:52 GMT

Hello,

Are you able to add the primary key in table when you are trying to insert and update from online to offline?

Best regards

Sabrina

Re: data from online to offline update

modara — Mon, 22 Mar 2021 07:36:02 GMT

Hi Sabrina,

first time I take data without primary key and then next time I

add primary key in table and then I Click RUN it takes full data again

tDBOUTPUT1---- Action On table: Default and Action on data: insert and Update

it should take only changed data for above configuration

Re: data from online to offline update

modara — Wed, 24 Mar 2021 06:13:41 GMT

Dear concern, I dont know why my talend community account is locked. If you guys not tell me the reason how would i know. I request several time to active my account but no one reply after shong replied 2-3 days back Please help to reactive my account Regards Sent from Yahoo Mail on Android On Wed, Mar 24, 2021 at 9:30 AM, MOHD OMAR DARAZ wrote: Dear Sabrina, Good day, Still I cant log in though I email to Shong to activate my account. I don't know why my account or locked Sorry to email you here. I really need to solve the issue from mssql to mysql only changed data. I just simple keep MSSQL as source data and connect it to MYSQL as destination data. I assign Primary key on both MSSQL and in MYSQL. After take full data from MSSQL to MYSQL I changed action on table: default and action on data: data type Insert and update. Here as I have no second table so I cant or no need to use tMap Please advise how Can i get only changed data from online to offline If you need more info for clarity Do not hesitate to email me. Your help is really appreciable Regards,

Re: data from online to offline update

modara — Wed, 24 Mar 2021 07:21:57 GMT

Sent from Yahoo Mail on Android On Wed, Mar 24, 2021 at 12:13 PM, Omar Daraz wrote: Dear concern, I dont know why my talend community account is locked. If you guys not tell me the reason how would i know. I request several time to active my account but no one reply after shong replied 2-3 days back Please help to reactive my account Regards Sent from Yahoo Mail on Android On Wed, Mar 24, 2021 at 9:30 AM, MOHD OMAR DARAZ wrote: Dear Sabrina, Good day, Still I cant log in though I email to Shong to activate my account. I don't know why my account or locked Sorry to email you here. I really need to solve the issue from mssql to mysql only changed data. I just simple keep MSSQL as source data and connect it to MYSQL as destination data. I assign Primary key on both MSSQL and in MYSQL. After take full data from MSSQL to MYSQL I changed action on table: default and action on data: data type Insert and update. Here as I have no second table so I cant or no need to use tMap Please advise how Can i get only changed data from online to offline If you need more info for clarity Do not hesitate to email me. Your help is really appreciable Regards,

Re: data from online to offline update

modara — Wed, 24 Mar 2021 07:23:36 GMT

Dear sabrina, To understand the issue or to solve you can remote to my pc and i can show you this talend diagram n my requirement Please inform me your suitable time Regards, Sent from Yahoo Mail on Android On Wed, Mar 24, 2021 at 1:21 PM, Omar Daraz wrote: Sent from Yahoo Mail on Android On Wed, Mar 24, 2021 at 12:13 PM, Omar Daraz wrote: Dear concern, I dont know why my talend community account is locked. If you guys not tell me the reason how would i know. I request several time to active my account but no one reply after shong replied 2-3 days back Please help to reactive my account Regards Sent from Yahoo Mail on Android On Wed, Mar 24, 2021 at 9:30 AM, MOHD OMAR DARAZ wrote: Dear Sabrina, Good day, Still I cant log in though I email to Shong to activate my account. I don't know why my account or locked Sorry to email you here. I really need to solve the issue from mssql to mysql only changed data. I just simple keep MSSQL as source data and connect it to MYSQL as destination data. I assign Primary key on both MSSQL and in MYSQL. After take full data from MSSQL to MYSQL I changed action on table: default and action on data: data type Insert and update. Here as I have no second table so I cant or no need to use tMap Please advise how Can i get only changed data from online to offline If you need more info for clarity Do not hesitate to email me. Your help is really appreciable Regards,

Re: data from online to offline update

Anonymous — Wed, 24 Mar 2021 08:01:36 GMT

Hello,

Are you using talend subscription solution? If so, you could create a support case on talend support portal so that we could give you a remote assistance through support cycle with priority.

Best regards

Sabrina

Re: data from online to offline update

modara — Wed, 24 Mar 2021 08:12:46 GMT

Dear sabrina, No i m using free version Its ok if you cant log in remote. But you help or suggesion is really apriciable in this regards to solve the issue Thanks Sent from Yahoo Mail on Android On Wed, Mar 24, 2021 at 2:01 PM, Talend wrote: @media only screen and ( _filtered_a ) {#yiv7258575727 html {}#yiv7258575727 table.yiv7258575727container {width:100% !important;}#yiv7258575727 .yiv7258575727hidden-for-mobile {display:none !important;}#yiv7258575727 .yiv7258575727callToAction, #yiv7258575727 .yiv7258575727callToAction td, #yiv7258575727 .yiv7258575727footer {font-size:12px !important;}#yiv7258575727 .yiv7258575727button .yiv7258575727text {font-size:12px !important;}#yiv7258575727 .yiv7258575727defaultIndentation {width:10 !important;}#yiv7258575727 .yiv7258575727text {font-size:12px !important;}#yiv7258575727 a.yiv7258575727button {width:96 !important;}#yiv7258575727 a.yiv7258575727buttonWide {width:118 !important;}#yiv7258575727 a.yiv7258575727buttonAuto {padding-left:1em !important;padding-right:1em !important;}#yiv7258575727 a.yiv7258575727button, #yiv7258575727 a.yiv7258575727buttonWide, #yiv7258575727 a.yiv7258575727buttonAuto {min-height:23 !important;line-height:23px !important;border-radius:4px !important;}#yiv7258575727 td.yiv7258575727buttonWrapper {width:98 !important;min-height:23 !important;}#yiv7258575727 img {display:inline !important;}#yiv7258575727 .yiv7258575727circle {border-radius:50%;}#yiv7258575727 .yiv7258575727square {border-radius:10%;}}@media only screen and ( _filtered_a ) {#yiv7258575727 table.yiv7258575727container {width:600px !important;}}@media screen { _filtered {}}#yiv7258575727 a, #yiv7258575727 a:visited {color:#015BA7;text-decoration:none;}#yiv7258575727 img {display:block;}#yiv7258575727 .yiv7258575727preheader {display:none !important;}#yiv7258575727 table {border-collapse:collapse;}Hello,Are you using talend subscription solution? If so, you could create a support case on talend support portal so that we could give you a remote assistance through support cycle with priority.Best regardsSabrina Hello, Are you using talend subscription solution? If so, you could create a support case on talend support portal so that we could give you a remote assistance through support cycle with priority. Best regards Sabrina Hello, Are you able to add the primary key in table when you are trying to insert and update from online to offline? Best regards Sabrina To understand the issue or to solve you can remote to my pc and i can show you this talend diagram n my requirement Please inform me your suitable time Regards, Sent from Yahoo Mail on Android

Re: data from online to offline update

modara — Thu, 25 Mar 2021 03:19:26 GMT

Dear Concern/ Sabrina,

sorry to bother you again. but the issue is not resolve yet. so would you be able to help me please.

regards,