1) Suppose you are having source text file named "A.txt" which contains billons of records.The task is to extract all the data from source file and loads into target table named " Table D".When millions of records are inserted into your target table, in the middle of your job for the particular record/data you get error.After resolving the error, when you execute the job again I want my job to do the following :- The already inserted records should not be inserted again in the "target table D" keeping the performance of your job in mind. How can we do? Consider the cases : a) Source file may contain unique records and b) It may contains duplicate records.
1/ use a look up using tmap component with your table in order to insert only the rejected records, don't forget to use disk storage for big files.
2/ store in a file somewhere the number of rows inserted in the previous execution using a ((Integer)globalMap.get("tDBOutput_X_NB_LINE_INSERTED")) global variable
