What methods are you using to validate your results?

[Anonymous]

Hi everyone,

I am relatively new to Talend and am interested in hearing about what you all are doing to ensure that your results are correct.

I typically am designing jobs that pick up an excel file via ftp, manipulate them, and load them into an environment for our data science teams to use for modeling and reporting. Given that these reports and models are being used to make major business decisions, I would like to put some controls around the output to make sure all the data made it as expected. What are you all doing to validate your results?

Cheers!

Michael

[Anonymous]

Hi Michael,

    If you are looking for schema verification and separating good and bad records based on it, you can use the tschemaComplianceCheck component.

    You can take reports on top of bad records and you can even send feedback mechanisms based on it.

   Please let me know if you are looking for some other validations and we can discuss based on your specific requirements in validation.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[Anonymous]

Thanks Nickhel!

This is a good tip. But, I am more thinking along the lines of checking to make sure my numbers still sum to the same total after being manipulated. Like, if I had 40 mil in sales, I would expect it to still show that I had 40 mil on the output. 

Kind of like "if input does not match output, fail the job, send an email."

[Anonymous · Accepted solution]

Hi,

    Thanks for the reply. So you would like to reconcile the counts for both input and output. 

    You can use the NB_LINE variables for the components in this case (Please refer the screen shots below)

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[Anonymous]

That's pretty neat! I'll give it a try. Thank you!

[Anonymous]

@mcpbf 

Could you please select the answers which helped you to reach the solution for the post so that topic can be closed?

It will help the other Talend community members during their reference.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂

[Anonymous]

I was kind of hoping to get more discussion around this because I'm sure there are people out there doing all kinds of things to validate their results, but ok.