topic Re: How to Validate source data before loading into Target data? in Talend Studio

How to Validate source data before loading into Target data?

Anonymous — Tue, 07 Jan 2014 12:33:29 GMT

Hi Experts,
How to perform validation checks for source data like excel file before loading correct data into Target like SQL SERVER 2008 database.
With Regards,
Viswanath.

Re: How to Validate source data before loading into Target data?

Anonymous — Tue, 07 Jan 2014 12:44:21 GMT

Hi,
I am getting the following error while deploying to MDM server from studio. Please help me in resolving this issue. I have uploaded the FastInfoset.Jar file into lib/java folder but still getting same error
java.lang.NoClassDefFoundError: org/jvnet/fastinfoset/FastInfosetSource

Re: How to Validate source data before loading into Target data?

Anonymous — Tue, 07 Jan 2014 13:43:33 GMT

@praveens_p
please, open your own post.
regards
laurent

Re: How to Validate source data before loading into Target data?

Anonymous — Tue, 07 Jan 2014 13:45:34 GMT

@svisu87
have a look at tSchemaComplianceCheck for schema validation control :
https://help.talend.com/search/all?query=tSchemaComplianceCheck&content-lang=en
for some more data quality use Talend DQ tools.
regards
laurent

Re: How to Validate source data before loading into Target data?

Anonymous — Wed, 08 Jan 2014 13:01:41 GMT

Hi experts,
Thank you for your reply.
In my Scenario, I need to validate excel file in Source i.e. file contains 36 columns if file have more or less than 36, then data should not load into target table.
To perform this exercise, what should we do?
With Regards,
Viswanath.

Re: How to Validate source data before loading into Target data?

Anonymous — Thu, 09 Jan 2014 03:12:08 GMT

Hi,
From your description, have you checked the component TalendHelpCenter:tFileProperties
Best regards
Sabrina

Re: How to Validate source data before loading into Target data?

Anonymous — Mon, 18 Jan 2016 16:16:37 GMT

is it possible to use tschemacompliancecheck against hive table?

Re: How to Validate source data before loading into Target data?

Anonymous — Tue, 19 Jan 2016 03:40:27 GMT

Hi dooncomputer,
What's your job requirement? Could you please give us more information about your requirement? Actually, there is no limitation about the input sources.
Best regards
Sabrina

Re: How to Validate source data before loading into Target data?

Anonymous — Sun, 19 Jun 2016 07:40:32 GMT

Hi xdshi,
My scenario given below:
1. data verification between source and target databases
2. Informix is my source, oracle is my target.
3. Data is migrated from informix to oracle.
4. I need to validate the data between these database tables.
5. Send the unmatched rows to excel file
6. How do I do using talend? Need help on this.

Re: How to Validate source data before loading into Target data?

Anonymous — Mon, 20 Jun 2016 09:20:43 GMT

Hi smathan,
If we understand your requirement very well, you want to capture the changed data and only load these changed data into excel? If so, with talend open studio product, you need to compare tables to achieve your goal.
The work flow should be: Target DB-->tMap(make inner join on your input and set the "Catch lookup inner join reject" as true)-->outputexcel
Source DB-->
The output will be the changed data.
Let us know if it works.
Best regards
Sabrina

Re: How to Validate source data before loading into Target data?

Anonymous — Mon, 20 Jun 2016 14:04:56 GMT

Hi Xdshi,
Thanks for your response. Let me put my requirement in simple way.
1. I have a table data in informix (let us say t1 is the table and employee_id is the column name and the data from 1 to 10 is present)
2. This data is moved to oracle using some XXXX tool
3. Now i will have to check that in Oracle table (let us say t1_1 is the table and employee_id is the column and the data is moved 1 to 9 only but the 10 is missing in oracle)
4. This missing 10, I would like to capture in the excel.
Please do needful. thank you.

Re: How to Validate source data before loading into Target data?

Anonymous — Thu, 23 Jun 2016 16:41:04 GMT

Hi Sabrina,
I am trying to connect oracle and that was successful but when I try to retrieve schema in the retrieve schema page- page is keep on loading and i am unable to move to NEXT page. My db is huge too. Are there any simplest way or workaround to handle it? Help appreciated. thank you.

Re: How to Validate source data before loading into Target data?

Anonymous — Fri, 24 Jun 2016 10:29:38 GMT

Hi,

Thanks for your response. Let me put my requirement in simple way.
1. I have a table data in informix (let us say t1 is the table and employee_id is the column name and the data from 1 to 10 is present)
2. This data is moved to oracle using some XXXX tool
3. Now i will have to check that in Oracle table (let us say t1_1 is the table and employee_id is the column and the data is moved 1 to 9 only but the 10 is missing in oracle)
4. This missing 10, I would like to capture in the excel.

The solution provided in previous post will achieve your goal:
toracleinput-->tMap ( inner join on your input sources "primary key employee_id " and set the "Catch lookup inner join reject" as true)-->outputexcel
tinformixinput-->

Did you follow up document about:TalendHelpCenterRetrieving table schemas to retrieve schema? Is there any error message printed on console?

Best regards
Sabrina

Re: How to Validate source data before loading into Target data?

Anonymous — Wed, 29 Jun 2016 16:46:42 GMT

Hi Sabrina,
Thanks for your support. your solution is working. have a nice day.

Re: How to Validate source data before loading into Target data?

Anonymous — Wed, 29 Jun 2016 16:48:29 GMT

Hi Sabrina,
One more question, let us say i have two informix tables which needs to be joined and then one oracle table join these two tables using tmap and then output in excel. So ihave done one informix table and one oracle table before but i haven't tried two informix table and one oracle table. Please guide me.

Re: How to Validate source data before loading into Target data?

Anonymous — Thu, 30 Jun 2016 11:12:43 GMT

Hi,

One more question, let us say i have two informix tables which needs to be joined and then one oracle table join these two tables using tmap and then output in excel. So ihave done one informix table and one oracle table before but i haven't tried two informix table and one oracle table.

Could you please show us your current job design flow? What's the problem you got?
Best regards
Sabrina

Re: How to Validate source data before loading into Target data?

Anonymous — Mon, 04 Jul 2016 14:10:30 GMT

Hi Sabrina,
I have the following scenario, could you please advice.
Informix - schema1-> table1
informix - schema2 -> table2
Need to join these two tables to get columns from both using some common keys. Once this step is completed then compare the result set of the above scenario with another migrated schema (oracle db) ie, one table - which is having columns of (table1 & table2 of informix). Finally bring the rejectedrows as output into excel.

Re: How to Validate source data before loading into Target data?

Anonymous — Tue, 05 Jul 2016 10:31:03 GMT

Hi smathan,
From your description, you can use tMap to achieve it also.
The workflow should be: tOracleInput
| (join migrated schema from oracle and table1 & table2 of informix
Informix (source1)-->tMap(common key)------------------->tMap_2 set the option"catch output reject" as true in tmap)---- -->tfileoutputexcel
Informix (Source2)

Let us know if it works.
Best regards
Sabrina

Re: How to Validate source data before loading into Target data?

Anonymous — Tue, 05 Jul 2016 19:06:06 GMT

Hi Sabrina,
I was trying the same logic just myself and it worked out! Feel very happy that i did it in first time. Our productivity become more after we introduce DI tool in our department. thanks for your response as well.

Re: How to Validate source data before loading into Target data?

Anonymous — Wed, 06 Jul 2016 09:35:51 GMT

Hi smathan,
Great it works. Feel free to let us know if there is any further help we can give.
Best regards
Sabrina