I have a table where I have designed incremental load. The table has a volume of around 10 million records. I have an insert update logic running in my incremental load.So its picking up new/modified records and concatenating them into the history qvd every 5 mins. In this way my QVD file is growing in size every hour.
Once incremental load finishes it tigers my dashboard. Now the problem is every time my dash board runs it has to read through all the days of data i.e the history data in the qvd along with the new ones that came in.Its taking a huge time which business can't afford.
Can anyone suggest me some otimization solution?
while researching I found we can do partial reload but some how I cannot releate to it since all the examples are talking about reading a new table. But what if I can use the partial reload option in my application so that it don't have to read the records it has already stored gone through?
I am using qliksense 3.2. (I think partial reload option is not native but that's a different question)
I am attaching the script log. I have issue in a certain area where I am doing a group by . I have created a calendar table to built interval for 5 min,15 min and 60 min and then join the intervals with my transaction data which falls under them .I have removed some irrelevant part from the script and also changed the business column name for confidentiality but you should get the general data flow.
Please let me know your suggestion how can I optimize. Also I am using incremental load to contact new/updated data in the transaction QVD and reading the entire qvd in my application at each run. But if any other approach is more efficient like binary load when looking at a application to handle 15-20 m records please let me know.