Loading mechanism to fill the existing column in destination after adding new columns
Dear Experts,
I have an situation and thinking that how Talend will help to achieve or any common logic.
For ex:
I have a Talend job which is currently extracting data from source db and inserting into destination db.
Source:
col1,col2,col3
AB12,90,Y
Note: the source table contains 56 columns but only 3 columns what business needs at Day 1.
Destination:
col1,col2,col3
AB12,90,Y
After 15 days- business asked to add another 3 columns from source, so i have added those columns in source extraction query and also added those columns in destination table.
Source:
col1,col2,col3,col4,col5
AC1,40,Y,ET,US
After executing the job, destination looks like below with past load old data and new data.
Destination:
col1,col2,col3,col4,col5
AB12,90,Y,null,null
AC1,40,Y,ET,US
The question is, how fill the two newly added columns for pas load since it shows null,
Should i do truncate and reload ? if yes i may loss the key linkage to other table like Fact table.
Should i do update for pas load records ? if yes will talend perform well for more than 1M records whenever this situation occurs ?
