Data Archival Strategy

TC1697460415 — Fri, 15 Nov 2024 21:08:49 GMT

I have 5 jobs to load 5 records the data from csv file to Landing area i.e. mocall, mtcall, mosms,mtsms and forwardcall, then 5 different jobs to incrementally load the data in different pipelines. Staging area table contains all the unique columns from all the landing tables. Pipeline is as follows: Landing-> Staging -> fact table (using 5 dimensions) -> data mart. The first data mart I created had timestamp one below each other, The new data I want is one beside each other, both screenshots are attached. I figured out the query for new data but how do I archive the data using Talend since there is no timestamp in new data mart

This is the previous data mart

New data mart is made using the same dimenions and staging table.It looks as below:

and so on, now as and when new data come, here I have data till dec so last column is dec_non_user_flag, If january data comes, may data should get archived or atleast previous 6 months data should get archived and new columns of jan should be created. How do I achieve the same and automate same

Jobs screenshot

In tDBInput4 is the query to obtain previous data mart, should replication of same is needed for new data mart format as well for the aid in archival

topic Data Archival Strategy in Talend Studio

Data Archival Strategy