Qlik Community

New to Qlik Sense

Discussion board where members can get started with Qlik Sense.

Not applicable

Any way to use large amount of data without importing/loading that data in local pc ?

Hello Community ,

I need to ask something important that is there any way to use a data without loading/importing that data .. Let's say we have a data around 90 gb and we want to make a report or dashboard using this data in qliksense but such an amount of data if load will take a lot of time and memory in pc .. anyother way to use  data without loading/importing it ?

Thanks in advance

6 Replies

Re: Any way to use large amount of data without importing/loading that data in local pc ?

If you want to display these data and/or to calculate anything with them you will need to load the data. But it doesn't mean that you mandatory need to load all data every time else you could use incremental approaches and/or to slice the data down to the really needed records and fields and/or to consolidate them to the needed granularity.

Also 90 gb of raw-data doesn't mean that a qlik application will also need 90 gb. Depending on the data-structure and especially the amounts of distinct field-values it could be much lesser - often you need only 10% - 20% of the raw-data size.

- Marcus

rittermd
Honored Contributor

Re: Any way to use large amount of data without importing/loading that data in local pc ?

You must load data into the app for it to process anything.

As Marcus said you can do incremental loads so that you are not loading the total 90GB each time.  You also would want to load your raw data into a QVD file and then use that QVD as your data source.  This is a very efficient file structure for Qlik.

Do as much data manipulation as you can when creating the QVD so that you don't need to do it in the app.  There are things that you can do to limit the amount of data and indexing required.

A clean data model is always important.

Qlik Sense can handle 90GB of data if you follow best practices.

Not applicable

Re: Any way to use large amount of data without importing/loading that data in local pc ?

I have to make report with this complete data set and make an associations in order to  make a report .. i have to load the entire data set ?

tomasz_tru
Valued Contributor

Re: Any way to use large amount of data without importing/loading that data in local pc ?

compressed 90GB and incremental load would give you effective solution; but if you insist on not loading the data, there's still Direct Discovery: Accessing large data sets with Direct Discovery ‒ Qlik Sense

rittermd
Honored Contributor

Re: Any way to use large amount of data without importing/loading that data in local pc ?

If you need all of that data for the app then you need to load it all. 

Comment out the fields that you don't need.  Use a where clause if you can limit what is loaded. 

Again, just follow best practices.

Employee
Employee

Re: Any way to use large amount of data without importing/loading that data in local pc ?

OnDemand App Generation might also on option for your use case.

Community Browser