topic [resolved] Is there a way to check for Data Integrity after data is migrated? in Talend Studio

[resolved] Is there a way to check for Data Integrity after data is migrated?

Anonymous — Sat, 16 Nov 2024 11:30:57 GMT

Hi,
I'm working on a data migration project which involves migrating data between two different versions of oracle database schema. I'm using Talend Open Studio version 5.4.1 for this task.
When it comes to migration there can be data transformation as the old data is compatible with the new schema. I need to make sure that old data is properly migrated to the new database schema make need to verify data integrity.
Does Talend Open Studio for Data Integration provide any option to verify data integrity in data migration?
Searching through the internet I found this https://community.talend.com/t5/Design-and-Development/Data-Integrity-during-a-migration/td-p/84707 but the answer given for that query is not sufficient in my case.
Asanka.

Re: [resolved] Is there a way to check for Data Integrity after data is migrated?

Anonymous — Fri, 05 Sep 2014 08:14:01 GMT

Hi,
Are you looking for component TalendHelpCenter:tSchemaComplianceCheck?
Could you please elaborate your case with an example?

Best regards
Sabrina

Re: [resolved] Is there a way to check for Data Integrity after data is migrated?

Anonymous — Fri, 05 Sep 2014 08:34:01 GMT

Hi Sabrina,
Thank you very much for the quick response.
According to my understanding TalendHelpCenter:tSchemaComplianceCheck can be used in the pre-processing (pre migration analysis) stage to make sure that source data of a particular table is fit to the equivalent destination table.
In my scenario there can be situations when data is migrated from source to destination records may go through a data transformation process and there can be enrichment to the source data. But I need to make sure that data is migrated properly according to the mapping rules defined between source and destination tables. Hope you will get my requirement.
Asanka.

Re: [resolved] Is there a way to check for Data Integrity after data is migrated?

Anonymous — Wed, 10 Sep 2014 10:08:23 GMT

Hi
Can someone from Talend team help on this?
Thanks and Regards,
Asanka.

Re: [resolved] Is there a way to check for Data Integrity after data is migrated?

Anonymous — Wed, 10 Sep 2014 10:44:23 GMT

Hi Asanka,
I am not from talend team, but would try to give some explanation based on my understanding.
This data verification is not possible using any of the Talend component in automated way. In order to check the data, you will have to fabricate target data manually by applying all the rules implemented in the talend job on input data and then compare one to one with the target output. To do this, tMap with inner join and output reject would help to find out records which does not match to each other.
Thanks
Vaibhav

Re: [resolved] Is there a way to check for Data Integrity after data is migrated?

Anonymous — Wed, 10 Sep 2014 10:51:45 GMT

Hi Vaibhav,
Thank you very much for your reply on my query.
Your suggestion will work in my problem. I'll give a try based on your suggestion.
Thanks and Regards,
Asanka.