Error while pulling more than 200K records from Teradata in Talend
We have a requirement in our Big Data project where we need to validate around 1 Million records
We have a source SQL to retrieve data from teradata table and a target SQL to retrieve data from Hive table and then do a row/row comparison but the issue was - the Job aborts when the number of records pulled from Teradata exceeds 200K.
Is there any way to pull and validate around 1M records ?
Please refer to the attached screenshot for the error details.