topic Re: Using Talend for Data Migration Testing in Talend Studio

Using Talend for Data Migration Testing

Anonymous — Tue, 16 Sep 2014 04:29:20 GMT

Hi Team,
I want to know what are the options and components available in Talend Open Studio for Data Integration that can be used in testing of data migration.
Regards,
Asanka.

Re: Using Talend for Data Migration Testing

Anonymous — Tue, 16 Sep 2014 05:07:38 GMT

Hi,
Do you want to monitor row In/Out during your Data Migration Job executing? Would you mind giving us more description about your requirment?
Best regards
Sabrina

Re: Using Talend for Data Migration Testing

Anonymous — Tue, 16 Sep 2014 06:35:28 GMT

Hi Sabrina,
In brief I need to verify source data has been successfully migrated to the destination. This verification needs to be carried out after the migration of particular subset of data in other words after completion of migration job. This needs to be done as a part of migration test in order to verify the data migration. This migration testing should ensure that transfer of data from sources db to the target db occurs without data loss, with strict adherence to transformation rules, and in compliance with all validity checks.
Thanks and Regards,
Asanka.

Re: Using Talend for Data Migration Testing

Anonymous — Tue, 16 Sep 2014 07:13:53 GMT

Hi,
The same issue in another topic: https://community.talend.com/t5/Design-and-Development/resolved-Is-there-a-way-to-check-for-Data-Integrity-after-data/td-p/115031?
Best regards
Sabrina

Re: Using Talend for Data Migration Testing

Anonymous — Tue, 16 Sep 2014 07:24:50 GMT

Hi Sabrina,
Yes. That is the same topic what I expected. In it says that 'Talend Open Studio for Data Integration makes ETL testing easier and more effective'. Is there any standard approach or procedure to follow in ETL testing using Talend. Please share any resources if you aware of this.
Thanks and Regards,
Asanka.

Re: Using Talend for Data Migration Testing

Anonymous — Tue, 16 Sep 2014 07:34:46 GMT

Hi,
With Talend Enterprise Subscription Version, there is CDC feature(change Data Capture) which capture only the changed source data and to move it from a source to a target system(s) in real time. TalendHelpCenter:CDC architectural overview
With Talend Open Studio, you have to compare Source DB and target DB.
Best regards
Sabrina

Re: Using Talend for Data Migration Testing

Anonymous — Tue, 16 Sep 2014 07:59:50 GMT

Hi Sabrina,
Thank you very much for your quick response.
In my case it doesn't need to capture change data in real time. Data migration from source db to target db will be a one time process. What I need is to ensure source data has been successfully migrated to the target db according to the predefined mapping rules without any data losses. Is it possible use talend open studio for this purpose.
Thanks and Regards,
Asanka.

Re: Using Talend for Data Migration Testing

Anonymous — Tue, 16 Sep 2014 08:58:14 GMT

Hi,
You can compare target DB and source DB to check if there is any missing data.
For example:
TargetDBTable--main-->tmap-->--->tlogrow
SourceDBTable--lookup
In tmap, please choose inner join for your input data and set "Catch lookup inner join reject" as true. If there is no output data when executing job, no missing data during Data Migration processing.
Best regards
Sabrina

Best regards
Sabrina

Re: Using Talend for Data Migration Testing

Anonymous — Tue, 16 Sep 2014 09:27:11 GMT

Hi Sabrina,
Thank you very much for the suggestion. I'll try it and post here the result sooner. I highly appreciate your quick responses and feedback.
Thanks,
Asanka.

Re: Using Talend for Data Migration Testing

Anonymous — Tue, 16 Sep 2014 09:41:27 GMT

Hi,
Feel free to let us know if it is OK with you.
Best regards
Sabrina

Re: Using Talend for Data Migration Testing

Anonymous — Wed, 17 Sep 2014 07:35:37 GMT

Hi,
The above suggested method to identify missing records in source to target data migration works as expected. But this approach can be used if there are no any source data altering or enriching while migrating to the target db. There can be situations where we need to modify source data as it compatible with the target db schema. Those modifications will be applied to the source data while they are migrated according to the predefined transformation rules. In such situation are there any options available to verify the data migration has carried out properly according to the predefined rules?
Thanks and Regards,
Asanka.

Re: Using Talend for Data Migration Testing

Anonymous — Wed, 17 Sep 2014 09:14:37 GMT

Hi,

There can be situations where we need to modify source data as it compatible with the target db schema. Those modifications will be applied to the source data while they are migrated according to the predefined transformation rules.

For your use case, could you please set an example? Monitoring data change in DB table synchronously?
Best regards
Sabrina

Re: Using Talend for Data Migration Testing

Anonymous — Wed, 17 Sep 2014 10:15:45 GMT

Hi Sabrina,
Sorry I'm not getting your suggestion. Can you please elaborate more if you don't mind?
Thanks and Regards,
Asanka.

Re: Using Talend for Data Migration Testing

Anonymous — Wed, 17 Sep 2014 10:33:28 GMT

Hi,
For example,
the CDC functionality can capture the data changes in the source table, and then you are able to synchronize these change data to target application.
Is it what are you looking for?

Best regards
Sabrina

Re: Using Talend for Data Migration Testing

Anonymous — Mon, 22 Sep 2014 03:56:10 GMT

Hi,
Actually we are not changing data of the source tables. Data transformation will be carried out on the fly (while data is migrated to the target tables). What we need is after data is migrated to the target tables the migrated data needs to be compared with the source data while considering the transformation rules.
Thanks and Regards,
Asanka.