Significant lag issues with large dataset

[#1]

Hi all,

I am a QV beginner, and I am having issues setting up a dashboard using my large dataset. I have about three years' worth of survey-type data, with about 3-4 million observations per year. (Don't know if it helps, but I load from a local csv file, where each line is a unique respondent, with info on age, income, # kids, etc. A data reload usually takes ~10 min, but I haven't had any issues here.)

I don't know if it's a computer issue (I am running QV desktop, with 8GB RAM and an Intel quad-core processor) or data structure, but it takes forever even to do basic calculations. For example, doing sums and counts with the "straight table" chart (with no other graphs, tables on the active sheet) takes 30+ seconds of processing time. Making any selection changes leads to another 30 sec lag, etc., etc.

The lag is OK for now but not OK for demonstrations/end user use.

Does anyone have any suggestions for how I can speed QV up? Thanks,

Andrew

[#2]

Hi.

Could you put a small .qvw file to look the script and data model?

Alessandro Furtado

[#3]

Sure, I've uploaded a sample file, with 5k obs. Obviously, a lot smaller than the real deal.

I think I face the longest lags with variable income tables...

[#4]

Hi.

I'm not sure, but try to change the dimension just to know if aggr is too slow or not. Just to try. Maybe you can use valueloop() or class().

Also I read that count(distinct ) is too slow.  Maybe you can in the script put 1 or 0 in a flag and than use sum(flag).

Sorry if I dont' realy help you.

Alessandro Furtado

[jerrysvensson]

Using aggr in a calculated dimension is extremely bad for performance.

Try to create a dimension in the load script instead.

[#6]

OK, so 'aggr' is not so good. The reason I use aggr is so the user can customize income groups (incomes $0-$5k, $5k-$15k, $15-50k, etc.). Are customized dimensions still doable with load script?

Thanks.