Skip to main content
Announcements
Join us at Qlik Connect for 3 magical days of learning, networking,and inspiration! REGISTER TODAY and save!
cancel
Showing results for 
Search instead for 
Did you mean: 
TC1697460415
Contributor II
Contributor II

Data Archival Strategy

I have 5 jobs to load 5 records the data from csv file to Landing area i.e. mocall, mtcall, mosms,mtsms and forwardcall, then 5 different jobs to incrementally load the data in different pipelines. Staging area table contains all the unique columns from all the landing tables. Pipeline is as follows: Landing-> Staging -> fact table (using 5 dimensions) -> data mart. The first data mart I created had timestamp one below each other, The new data I want is one beside each other, both screenshots are attached. I figured out the query for new data but how do I archive the data using Talend since there is no timestamp in new data mart

069U1000001ZzT3IAK.png069U1000001ZzXtIAK.png

This is the previous data mart

New data mart is made using the same dimenions and staging table.It looks as below:

069U1000001ZzcjIAC.png

069U1000001Zyf5IAC.png

and so on, now as and when new data come, here I have data till dec so last column is dec_non_user_flag, If january data comes, may data should get archived or atleast previous 6 months data should get archived and new columns of jan should be created. How do I achieve the same and automate same

Jobs screenshot

069U1000001a02XIAQ.png 069U1000001a05lIAA.pngIn tDBInput4 is the query to obtain previous data mart, should replication of same is needed for new data mart format as well for the aid in archival069U1000001a00vIAA.png

Labels (2)
0 Replies