Loading only correct data from flat file to azure SQL Data warehouse and storing rejected data into the flat file

[Anonymous]

Hi, I am trying to load correct data from flat file to Azure SQL data warehouse and want to capture the rejected records into the target flat file.

In my case,only erroneous records are getting captured but correct records are not getting loaded into the azure SQL data warehouse table.

I have unchecked the "Die on error" and "Use Batch Size" options

(In schema configuration of azure SQL data warehouse table,I have put less length for Full_Name than actual. Hence, 6 records should be loaded into azure SQL DW table and 14 records should get rejected)

But in my case, correct records are not getting loaded.

[Anonymous]

Hi,

   The reason most probably is that you have not done committing of the data. Please add a tSQLDWHConnection component and select auto commit in advanced options.

Another option is to connect a tSQLDWHCOmmit component as OnSubJobOk to previous step.

Either way, once you commit the records, it will be available in database.

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved

[Anonymous]

Hi,

resolved this issue by putting the 'Commit every' value to 1 in output component used.

And it is working fine now.

[Anonymous]

Perfect!

Both approaches can resolve the issue. Could you please mark the posts which are having solutions to your original query so that it will help other Talend Community members during their reference?

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved