Skip to main content
Woohoo! Qlik Community has won “Best in Class Community” in the 2024 Khoros Kudos awards!
Announcements
Nov. 20th, Qlik Insider - Lakehouses: Driving the Future of Data & AI - PICK A SESSION
cancel
Showing results for 
Search instead for 
Did you mean: 
meimagine
Contributor
Contributor

Bulk load and then incremental load of almost 200GB data in Qlikview

Hey Guys,

We are having a very huge data set (more than 200 GB) which also needs to be incrementally updated after the bulk update. I have loaded a little bit and am already running into performance issues.(I have tried creating QVDs and direct DB loads both and I have already reduced the number of columns that are being extracted) Can some one please suggest what can be done to implement this. The stakeholders actuallly want to extract 3-4 years of data from the dashboards as well. I am pretty new to Qlikview and have not handled this much amount of data in qlik dashboards before. Any suggestions on what process I can try?

Are there options to archive the data and only load it on the dashboard when someone is trying to extract a specific data set? But even if I keep the minimum the data set for a year would also be pretty huge.(20 million rows a month) Thanks.

Labels (1)
22 Replies
marcus_sommer

In general it's not possible because Qlik isn't row-level oriented like the classical sql-databases else it's column-oriented. This means data couldn't be simply append like you would do it manually as an example by copying a log-record from one log-file and appending at the bottom of another log-file and saving this task. Qlik organized the data within system-tables (one for each field which stored only distinct values) and data-tables which just contain a binary pointer to the system-table. 

For each adding/removing from data these system-tables and the pointer must be re-created. There is an exception by using partial loads but it has some restrictions in regard to the uniqueness of the values. Depending on your data and requirements this may be a problem or not. Beside this I doubt that you would significantly save load-times with such an approach. Nevertheless you may try it:

Scripting/ScriptPrefixes/Add.htm 

Therefore I think my lastly made suggestion of reloading the dashboard binary and adding then the new data with an optimized qvd-load will be the fastest way to refresh your dashboard. By a well-built data-architecure + datamodel and an appropriate Server it should work quite well for your mentioned parameter of 200 GB of data (are this really 200 GB in Qlik and by optimizing the data-structures ?) and a refresh time-frame of 15 minutes.

- Marcus

Brett_Bleess
Former Employee
Former Employee

You have received a lot of responses, and we would appreciate it if you would close out the thread if those responses have gotten you the information you required by using the Accept as Solution button on those posts that helped the most.  If you did something different, you can post that and then use the button on that post to mark things. This gives the posters credit for the help they provide and lets the other Community Members know what helped.  If you have further questions, please leave an update post.

You may want to check out the following area of the Community too for further ideas and here are a couple of links related to best practices too:

https://community.qlik.com/t5/Qlik-Design-Blog/bg-p/qlik-design-blog

https://help.qlik.com/en-US/qlikview/April2020/Subsystems/Client/Content/QV_QlikView/application-per...

https://help.qlik.com/en-US/qlikview/April2020/Subsystems/Client/Content/QV_QlikView/LoadData/best-p...

Regards,
Brett

To help users find verified answers, please do not forget to use the "Accept as Solution" button on any post(s) that helped you resolve your problem or question.
I now work a compressed schedule, Tuesday, Wednesday and Thursday, so those will be the days I will reply to any follow-up posts.
meimagine
Contributor
Contributor
Author

Hi Brett,

I am still working on using these solutions. Once I finish those and get an answer, I'll update the thread. Thanks.