Dears,
As per the new challenges that we all face it in the size of data; we need to reduce the size of the load time in any BI structures especially in QlikView in the QMC.
Usually, we have the date and time column for insert update and delete to trace the action for your data but if there is now insert update and delete column what you will do?
You will start thinking I will write a script to load the tables then I will do join by using the not exist function and this is good but if you have a transformation on the data and the size of your data is big and every day you will have more than 30 million new records this will take more time than expected.
One of the recommended way to do that and i really use it is to build a logic based on the main date that you are using and use variables to trace that and store the max date you have in your data; so you have now the max date that inserted in your data files then you will have to assign a variable for the window size let say your data will be committed in the database every day and there is no changes will happen in that so can make your days window is 2 or 3 days in case of the lake network or something happen during the reload and storing the data.
what it will happen for the 2 or the 3 days and you only need 1 day it will overlap for the existing 2 days with the same existing records if there is no updates happen and you will have the new records posted in your QVD store by using the concatenate function to get only the new records for the load window.
and it is recommended to use the Trace Function you will know what it will happen and you can print in the load window your variables that it will help you to trace the load.
