Qlik Community

New to Qlik Sense

If you’re new to Qlik Sense, start with this Discussion Board and get up-to-speed quickly.

Announcements
Tell us which business and trade publications you read most regularly: RESPOND NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Any way to use large amount of data without importing/loading that data in local pc ?

Hello Community ,

I need to ask something important that is there any way to use a data without loading/importing that data .. Let's say we have a data around 90 gb and we want to make a report or dashboard using this data in qliksense but such an amount of data if load will take a lot of time and memory in pc .. anyother way to use  data without loading/importing it ?

Thanks in advance

6 Replies
marcus_sommer
MVP & Luminary
MVP & Luminary

If you want to display these data and/or to calculate anything with them you will need to load the data. But it doesn't mean that you mandatory need to load all data every time else you could use incremental approaches and/or to slice the data down to the really needed records and fields and/or to consolidate them to the needed granularity.

Also 90 gb of raw-data doesn't mean that a qlik application will also need 90 gb. Depending on the data-structure and especially the amounts of distinct field-values it could be much lesser - often you need only 10% - 20% of the raw-data size.

- Marcus

rittermd
Master
Master

You must load data into the app for it to process anything.

As Marcus said you can do incremental loads so that you are not loading the total 90GB each time.  You also would want to load your raw data into a QVD file and then use that QVD as your data source.  This is a very efficient file structure for Qlik.

Do as much data manipulation as you can when creating the QVD so that you don't need to do it in the app.  There are things that you can do to limit the amount of data and indexing required.

A clean data model is always important.

Qlik Sense can handle 90GB of data if you follow best practices.

Not applicable
Author

I have to make report with this complete data set and make an associations in order to  make a report .. i have to load the entire data set ?

tomasz_tru
Specialist
Specialist

compressed 90GB and incremental load would give you effective solution; but if you insist on not loading the data, there's still Direct Discovery: Accessing large data sets with Direct Discovery ‒ Qlik Sense

rittermd
Master
Master

If you need all of that data for the app then you need to load it all. 

Comment out the fields that you don't need.  Use a where clause if you can limit what is loaded. 

Again, just follow best practices.

MatthiasAnders
Employee
Employee

OnDemand App Generation might also on option for your use case.