hi, Scenario-> I want to store Some record in File or table with ID column. Suppose, For first run Storing 10 record then, O/P will be ID ----- 1 2 3 . . 10 Now for Second run it should start from 11. for that we need to store max of last load i.e10 so it will be ID --- 1 2 3 . . 10 11 12 13 ... how to do this in Talend.
Hi
In the beginning of job, read the max ID from a static file and store it to a context variable or global variable for used later. For example:
tFileInputDelimimitecd--main--tJavaRow
on tJavaRow:
context.max_value=input_row.max_value;
In the main processing of job,
if source data are stored in database, select records only when ids bigger than max value, for example:
"select * from tableName where id>"+context.max_value
if source data are read from file or other data sources, filter the rows on tMap, and then load the result to destination. For example:
tFileInputDelimited--main--tMap--o/p component--main--tSortRow--tMap_2--main--tFleileOutDlimited_2
on tSortRow, sort all the ids with desc model.
on tMap_2: add a sequence number for each row, and filter the rows to get only the first row which is the max ID for this execution.
on tFleileOutDlimited_2: update the max ID to the static file.
Best regards
Shong
