topic Re: talend real time loading issues in Talend Studio

talend real time loading issues

psr — Sat, 16 Nov 2024 03:07:02 GMT

1) Suppose you are having source text file named "A.txt" which contains billons of records.The task is to extract all the data from source file and loads into target table named " Table D".When millions of records are inserted into your target table, in the middle of your job for the particular record/data you get error.After resolving the error, when you execute the job again I want my job to do the following :-
The already inserted records should not be inserted again in the "target table D" keeping the performance of your job in mind.
How can we do?
Consider the cases : a) Source file may contain unique records
and b) It may contains duplicate records.

Re: talend real time loading issues

Anonymous — Tue, 03 Mar 2020 21:49:58 GMT

Hi,

You have 2 possibilities in my opinion,

1/ use a look up using tmap component with your table in order to insert only the rejected records, don't forget to use disk storage for big files.

2/ store in a file somewhere the number of rows inserted in the previous execution using a ((Integer)globalMap.get("tDBOutput_X_NB_LINE_INSERTED")) global variable

Re: talend real time loading issues

fdenis — Wed, 04 Mar 2020 10:44:32 GMT

you may split on 2 time load file into a staging db then load into destination and manage error.