how to prevent job from loading already existing records upon restart of job after failure

[Anonymous]

Hi @shong,

I have a scenario where I have job which loads data from oracle(source) to oracle(target).

Suppose if I have 10M records in source and if job fails due to some issue(network) after loading few records suppose say (2M), then how to load remaining 8M records without reloading those 2M records again?

Im using talend 6.5.1 version.

Quick reply would be appreciated

Thanks,

Kiran Kumar

[TRF]

If you have a primary key on the target table, it's not really a problem as duplicates will be rejected.

Right, you'll try to insert may 1, 2 or more millions records for nothing and it's not a good thing as soon as there is a lot of records...

If you can have a dblink between source and target databases, you can have a single query to get only records from source which don't exist in the the target database.

Else, you need to get the records from the target too and eliminate these records from the source flow using a tMap.