Qlik Community

Technology Partners Ecosystem Discussions

Qlik Technology Partners Ecosystem

Highlighted
orlando-b
New Contributor

Cannot load data from Impala table with 1 billion rows: Qlik Sense Deskop

1. What is the maximum number of rows Qlik Sense Desktop can handle (with 8GB of RAM)?

     I am able to select my table but I can not load it. When I load the data, it seems my network connection is at 50% for multiple minutes before I can start, also I don't want to wait for it to finish, more than a minute is unacceptable to get started.

 

My table has 1 billion rows.

 

2. If this number of rows is too much, is it possible to work with 'samples' when design the app and then click a 'generate' button or something to compute all the views on all the data?

Tags (3)
Labels (1)
3 Replies
MVP & Luminary
MVP & Luminary

Re: Cannot load data from Impala table with 1 billion rows: Qlik Sense Deskop

I think 8 GB will be to small for 1 billion records even if the table is rather small and there aren't many distinct values.

Beside this a database will usually need a lot of time to deliver such amount of data and of course the network between the database and your machine might be another bottleneck.

I suggest that you slice down the data on a monthly and/or category-level and storing them into qvd's and implement an incremental approach with them.

The most topics of developing an application could be done with a reduced set of data - but I doubt that you will be ever happy with it by 8 GB of RAM if you then load some more data ...

- Marcus

orlando-b
New Contributor

Re: Cannot load data from Impala table with 1 billion rows: Qlik Sense Deskop

If I'd move my data to Hadoop and connect through Impala, would I be able to leave the data in place and work with let's say the first 10000 rows e.g. in Qlik desktop and when I'm ready trigger to execute a computation on all the views. But in such a way that the (SQL) operations are being pushed down to the cluster rather than the data being copied to my desktop machine? Or is this maybe an advanced feature only possible with the server edition of Qlik Sense?

MVP & Luminary
MVP & Luminary

Re: Cannot load data from Impala table with 1 billion rows: Qlik Sense Deskop

Using incremental load-approaches with optimized qvd-loadings especially in server-environments with local ssd-storages is very fast and AFAIK most of any alternatives are a lot slower.

I think without a Sense server-environment you could end this task here. With it you could just pull a smaller slice of the data-set maybe 1 - 10 millions of records and developing your applications.

- Marcus

Community Browser