[resolved] Is there a way to check for Data Integrity after data is migrated?

One Star

[resolved] Is there a way to check for Data Integrity after data is migrated?

Hi,
I'm working on a data migration project which involves migrating data between two different versions of oracle database schema. I'm using Talend Open Studio version 5.4.1 for this task.
When it comes to migration there can be data transformation as the old data is compatible with the new schema. I need to make sure that old data is properly migrated to the new database schema make need to verify data integrity. 
Does Talend Open Studio for Data Integration provide any option to verify data integrity in data migration?
Searching through the internet I found this http://www.talendforge.org/forum/viewtopic.php?id=4150 but the answer given for that query is not sufficient in my case. 
Asanka.

Accepted Solutions
Four Stars

Re: [resolved] Is there a way to check for Data Integrity after data is migrated?

Hi Asanka,
I am not from talend team, but would try to give some explanation based on my understanding.
This data verification is not possible using any of the Talend component in automated way. In order to check the data, you will have to fabricate target data manually by applying all the rules implemented in the talend job on input data and then compare one to one with the target output. To do this, tMap with inner join and output reject would help to find out records which does not match to each other.
Thanks
Vaibhav

All Replies
Moderator

Re: [resolved] Is there a way to check for Data Integrity after data is migrated?

Hi,
Are you looking for component TalendHelpCenter:tSchemaComplianceCheck?
Could you please elaborate your case with an example?

Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] Is there a way to check for Data Integrity after data is migrated?

Hi Sabrina,
Thank you very much for the quick response. 
According to my understanding TalendHelpCenter:tSchemaComplianceCheck can be used in the pre-processing (pre migration analysis) stage to make sure that source data of a particular table is fit to the equivalent destination table. 
In my scenario there can be situations when data is migrated from source to destination records may go through a data transformation process and there can be enrichment to the source data. But I need to make sure that data is migrated properly according to the mapping rules defined between source and destination tables. Hope you will get my requirement. 
Asanka.
One Star

Re: [resolved] Is there a way to check for Data Integrity after data is migrated?

Hi
Can someone from Talend team help on this?
Thanks and Regards,
Asanka.
Four Stars

Re: [resolved] Is there a way to check for Data Integrity after data is migrated?

Hi Asanka,
I am not from talend team, but would try to give some explanation based on my understanding.
This data verification is not possible using any of the Talend component in automated way. In order to check the data, you will have to fabricate target data manually by applying all the rules implemented in the talend job on input data and then compare one to one with the target output. To do this, tMap with inner join and output reject would help to find out records which does not match to each other.
Thanks
Vaibhav
One Star

Re: [resolved] Is there a way to check for Data Integrity after data is migrated?

Hi Vaibhav,
Thank you very much for your reply on my query. 
Your suggestion will work in my problem. I'll give a try based on your suggestion. 
Thanks and Regards,
Asanka.