Running 4 millions rows in qvw. The data schema is 2 big flat tables.

[#1]

Hi All,

I have the same question.

I'm on Intel E7-L8867 @ 2.13Ghz, 64Gb RAM. NUMA Disabled.

Running 4 millions rows in qvw.

When opened, 8GB of RAM is consumed.

The data schema is 2 big flat tables, one for dimension one for facts. with sequential numeric as key to link them.

I'm not so worried about RAM as we don't have very big qvw.

But, it's more towards processing time.

When passing from one chart to another, i can see the hour glass running.

i have tried to optimise the formula and i dont see any other more efficient way to do it.

So, the question now is, do i upgrade my CPU?

QT told me that i'm on low energy model, does switching to non-energy helps a lot?

if yes, how much it will be improved?

I also took a look at server grade CPU, the next "better" is  E7-8870 @ 2.4Ghz, not much compared to the one that i'm already using. should i change to this one?

Could you guys give some comments on what should i do?

Thanks.

[#2]

Is there a reason not to join two tables into one at the load stage (in the script)? Thus you will probably have less CPU load during runtime.

[hps]

Hi Qlik Ranger,

If we isolate your hardware in this question it is an Intel E7-8800 chipset architecture which I would advise against.

The E7-8800 is particularly aimed at eight socket(CPU) architecture that introduced unfavourable inter CPU communication over QPI links. Much as I explained in my post below.

If you are only using a single CPU this should be less of a concern for now, but will be terrible once you add more CPUs to the server. If you are already on multiple CPUs changing the chipset is highly likely to be beneficial.

The alternatives are either a recommended server based on E5-2600 or E7-4800.

As you mention the upgrade to E7-8870 will not give that much of an increase.

Switching to non low power would be ideal and recommended, but it's impossible to state how much the improvement will be. I'd expect a bigger improvement from the change of chipset to E5-2600 or E7-4800.

This was only a brief comment on the hardware.

Two other massive factors come into play as well: Usage pattern (how many users, what do they select etc..) and Application design (Expressions, data model etc..)

Regards,

Hampus