topic Re: Loading a large data set causing performance issues in QlikView

Loading a large data set causing performance issues

Thu, 10 Jul 2014 12:22:43 GMT

I'm trying to load a fairly large dataset (I've loaded larger with no issues).

After I load I place all the datapoints in a scatter plot with the following expressions:

Long = sum( round (256*pow(2,($(var_zoom)-1)))+( Long *((256*pow(2,$(var_zoom)))/360)))

Lat= sum(((256*pow(2,($(var_zoom)-1)))+((0.5*log((1+(sin((Lat)*pi()/180)))/(1-(sin((Lat)*pi()/180)))))*((-256*pow(2,$(var_zoom)))/(2*pi())))))

(i'm doing this: Step-by-Step Google Map API v3 integration)

This works GREAT with my first test load of about 50k rows, but once I hit the 1.5 million rows it just spins and spins (trying to load the scatter plot) and stays like that until i give up a few minutes later.

Am I missing something from a performance standpoint, what could be wrong!?

PS: My QVW is not doing ANYTHING other than that scatter plot with the Google API Integration)

Re: Loading a large data set causing performance issues

marcus_sommer — Thu, 10 Jul 2014 13:24:16 GMT

If there aren't any errors (synthetic keys, circular references) within the data-structure (tableviewer) you should look to the taskmanager and/or working sets if there are enough RAM available. Further it could be helpful to select a smaller data-ranges before you activate this object.

- Marcus

Re: Loading a large data set causing performance issues

Thu, 10 Jul 2014 13:26:47 GMT

I barely have enough RAM left, the QVW is taking about 6gigs of ram when loading the full dataset.

I did select a small range, and when I did it worked, it just breaks with the full range.

Re: Loading a large data set causing performance issues

marcus_sommer — Thu, 10 Jul 2014 14:17:13 GMT

Objects have also a RAM limit - it could be somewhere changed on user-level but it's not recommended. Perhaps you have somewhere in your data errors and/or null-values - check for these and if there any null/errors wrapped your expressions(parts) within alt(Expression, AlternativeValue) or use if-loops.

- Marcus