topic Re: please help build a new server - HP DL380 or DL580 in Scalability

please help build a new server - HP DL380 or DL580

Anonymous — Mon, 21 Jul 2014 13:44:55 GMT

Hi guys,

I am looking for a second opinion as we are already working with our vendors and Qlik reps. We just purchased a new server and I found out that it performs terribly with QV even though it is 4 times more expensive and powerful. We are going to return this server to our vendor (HP) and get something different hence I am asking for an opinion on an ideal and fastest for QV HP server.

Right now my company uses a single server to run all QV services, including QVS, QDS and AccessPoint. It is very fast but we are running out of RAM and it is not expandable so we are in the process of getting a second server for QV which will only run AccessPoint and QVS.

Our old server:

2 socket HP DL380P GEN8

Intel Xeon CPU E5-2690 2.90GHz

256Gb of RAM

Our new server (which we are going to return):

4 socket HP DL580 Gen8

Intel Xeon CPU E7-4870 v2 2.30GHz

1Tb of RAM

I was super excited about DL580 and when it arrived I started running some tests. We have a fairly complex dashboard with 90+ million rows and a lot of extensive calculations and I found out that our old server was 2-5 times faster on the most of the tabs. We tuned the server using QV tech. paper specifically made sure that max. energy profile is used, hyper threading is off and NUMA is off. Turbo Boost is on.

It helped a little bit especially then we disabled NUMA – I am confused why though because according to QV, NUMA is not a problem on 11.2 but apparently it is. So after we disabled NUMA, things improved maybe by 10-20% but it was still a way slower than our old server.

We reached to HP and one of their senior architects explained that DL580 has to do more work with RAM because of the way RAM is shared between processors and the clock speed is 1333 vs. 1666 on DL380.

At that point I downloaded and ran 4 different benchmarking tools (MAXXMEM2,NOVABENCH, PassMark and SiSoft Sandra) and all of them showed 2-6 times difference in RAM tests – DL580 was slower again!

Our goals for a new server:

1) Should be faster than the old one
2) Should be expandable to 1Tb of RAM at least
3) Should support at least 50 concurrent users
4) Should be made by HP – rack server HP Proliant family.
5) Our budget is 50k
6) The server will run ONLY QVS and AccessPoint.

I apologize for a long introduction to my question but now I am puzzled what HP server we should pick since DL580 with 4 processors clearly does not meet our needs.

I am not a hardware expert and was relying on our hardware people and Qlik but apparently the configuration they picked did not meet our goals. They are working again to revise the config but I wanted to get a second opinion from a forum and you.

I am thinking now to either get the fastest E5 or E7 and this time only two sockets to minimize memory hops. Also I wanted to see if we can use higher clocked RAM (1887?)

It is going to be a rack server from HP Proliant family – you can actually build it online here

http://www8.hp.com/us/en/products/proliant-servers/index.html?facet=ProLiant-DL-Rack

Any suggestions are highly appreciated!

Re: please help build a new server - HP DL380 or DL580

Tue, 22 Jul 2014 07:27:35 GMT

Hi Borys,

We are also using the same set of servers and we are also facing the same issue with HP Proliant GL580 gen7. One thing which improved the server response is by reducing the number of CPU's used by the Qlikview server from 64 CPU's to just 32 CPU's.

The perfromance has improved a bit. You can try that in your system and provide your feedback.

Thanks,

Sagar

Re: please help build a new server - HP DL380 or DL580

beunderf — Tue, 22 Jul 2014 07:43:23 GMT

We had the same problem with a different server. Someone gave me the following explanation, which could be your problem as well, I hope it helps ( the problem is going from two to four sockets):

-------

Frank,

We ran into this as well on our QV publisher box, but first let me say that as a standard we disable NUMA (enable node-interleaving) on all our QlikView hardware (physical and virtual) and have measured a 35-45% performance boost by simply making that change. For the QVS, we see impacts of NUMA mostly on chart calculations and rendering with AJAX clients (plus all the CPUs tend to stay at max 100% for a few wall-clock seconds if NUMA is on, even for one chart calculation). If you search the community on the scalabilty lab sections you will find several hardware setting recommendations that basically say disable NUMA, disable hyperthreading unless you are on a 2-way Intel E5-xxxx architechture, enable hardware prefetch, and maybe one or two I'm forgetting here.

I'm guessing your server has 4 physical chips (slots) with 8 cores each for a total of 32 cores, set out probably like this on the board:

1 2

3 4

I can't give you the true "nuts and bolts" technical explanation, but I will try to paraphrase the QlikView Scalabilty Lab's recommendation to us. Basically, a 4x architechture presents issues for QlikView because of the way Qlikview splits processes into multiple cores on multiple chip slots and the inefficiencies in managing resources between the cores on chips 1 and 4 and 2 and 3. So if a single chart, etc. is using cores on chips 1 and 4 OR 2 and 3 to calculate, there's some efficiency issues in splitting those among the actual "slots"; if using cores from 1 and 2 OR 3 and 4, those inefficiencies are not present, so a 2x architechture would not have this issue. When the QlikTech scalability lab and I modified our CPU Affinity for publisher as per the instructions below, we measured about a 50% performance improvement. Our QVS machines were already all 2X so I've not experienced it on those but I expect it would be similar.

The QVS CPU affinity can be changed in the management console, but for publisher, you have to change the CPU affinity through a settings file

ProgramData\QlikTech\DistributionService\Configuration.xml

The row: <QVBConfig MaxSimultaneousQVBs="4" MaxSimultaneousReaderQVBs="20" CPUAffinity="11111111111111111111111111111111" CPUPriority="Low" />

If you have 32 logical cores as in my case you would see 32 ones to indicate that all are active. If you want to use only the last 16 then to the following:

1. Stop QDS service

2. Replace the first 16 ones with zeros

3. This would give the following setting: .." CPUAffinity="00000000000000001111111111111111" ..

4. Start the QDS service

You are now only using half of the possible cores

Re: please help build a new server - HP DL380 or DL580

Tue, 22 Jul 2014 08:02:34 GMT

We were able to improve the QV publisher performance by updating the wondows 2008 server bios as well. check out below details, hopefully that might also help.

What --> Upgrade System BIOS

Faulty BIOS --> P65 07/01/2013

Fix --> Apply SP63928.exe to upgrade to P65 10/01/2013

More Info --> P Support Center (http://h20566.www2.hp.com/portal/site/hpsc/template.PAGE/public/psi/swdDetails/?javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken&javax.portlet.prp_bd9b6997fbc7fc515f4cf4626f5c8d01=wsrp-navigationalState%3Didx%253D2%257CswItem%253DMTX_63d3fca65f874a63a0adebc3bc%257CswEnvOID%253D54%257CitemLocale%253D%257CswLang%253D%257Cmode%253D4%257Caction%253DdriverDocument&javax.portlet.tpst=bd9b6997fbc7fc515f4cf4626f5c8d01&sp4ts.oid=4142792&ac.admitted=1406015906171.876444892.199480143)

Re: please help build a new server - HP DL380 or DL580

Tue, 22 Jul 2014 08:03:14 GMT

We were able to improve the QV publisher performance by updating the wondows 2008 server bios as well. check out below details, hopefully that might also help.

What --> Upgrade System BIOS

Faulty BIOS --> P65 07/01/2013

Fix --> Apply SP63928.exe to upgrade to P65 10/01/2013

Re: please help build a new server - HP DL380 or DL580

Tue, 22 Jul 2014 08:54:51 GMT

Hi,

"QVBConfig MaxSimultaneousQVBs" this is only for reloads and will not improve the QVS performance 😕

/olli

Re: please help build a new server - HP DL380 or DL580

Anonymous — Tue, 22 Jul 2014 13:01:23 GMT

I am glad we are not the only ones facing this issue - thanks for all your replies and I feel better now

I already reviewed and applied settings recommended as per Scalability Lab document. Disabling NUMA helped a little bit and I noticed maybe 10-20% improvement. As for other settings like disabling hyperthreading and disabling prefetch - I do not think it actually did anything.

The BIOS is fresh - our server is literally straight out of factory. Even CPUs are from the most recent batch so everything is fresh

I did play with CPUAffinity for AccessPoint and disabled last half of CPU cores and rebooted after that - it actually did not help at all. Our HP architect (super knowledgeable guy and he works for HP) recommended to remove 2 CPUs - but for me it does not make sense because each processor cost us like 6k.

I think we are moving towards exchanging the server on a 2 socket one and probably getting the fastest Xeon E7 v2

. The problem there though that most 2 socket configurations can only access 512GB of RAM or 768GB at most (but slower DIMMs).

But all in all it is a big disappointment for us so far - we spent 4x times more money on this server than our our old one cost us and it is literally x4 times slower than the old one which is ironic but sad matter.

I wonder now if something can be improved in QV engine itself by Qlik - 4 and 8 socket machines are pretty cheap these days and we will see soon even more powerful servers so I believe it is something that Qlik needs to look at and address in future.

Re: please help build a new server - HP DL380 or DL580

Tue, 22 Jul 2014 18:54:26 GMT

Sagar,

by reducing from 64 to 32 did you mean setting up the processor affinity to every other core?

So in effect you would disable hyper-threading.

Re: please help build a new server - HP DL380 or DL580

Tue, 22 Jul 2014 19:01:02 GMT

Frank Beunder wrote:

...

The QVS CPU affinity can be changed in the management console,

Frank,

what pattern have you checked out on the managment console? something like 000000000000011111111111111 or 101010101010101?

Re: please help build a new server - HP DL380 or DL580

Wed, 23 Jul 2014 08:29:58 GMT

Yes KlickiBunti,

I got a document from internet on best practices configuring HP Proliant DL580 gen8 server, based on that hyperthreading causes issue when number of CPU's are more than 32. To overcome this we reduced the CPU Affinity from 64 CPU's to 32 CPU's/

http://h20195.www2.hp.com/V2/GetPDF.aspx%2F4AA5-1110ENW.pdf

CPU Affinity set is like 1111111111111111111111111111111100000000000000000000000000000000

Re: please help build a new server - HP DL380 or DL580

Anonymous — Fri, 25 Jul 2014 13:22:36 GMT

I tried that too but things got much slower in my case though. HP recommended to physically disable 2 processors in BIOS or pull them but this will also disable half of RAM we purchased.

Re: please help build a new server - HP DL380 or DL580

Fri, 25 Jul 2014 15:58:42 GMT

Borys, do you mind posting your data schema here? May be a few expressions too.

There are many things that can be done to improve the performance of a Qlikivew app. May be the collective mind will be able to suggest a few things that might help in your case.

90 million rows is minuscule 🙂 For instance I have over 7 billion rows (but not that many users).

Re: please help build a new server - HP DL380 or DL580

Anonymous — Fri, 25 Jul 2014 17:31:31 GMT

Hi KlickiBunti, thanks for your offer but we have many dashboards and all of them work much slower. The one I was testing with 90MM rows is the most complicated though and it took 2 years to develop it. A lot of financial calculations and metrics with a Health care twist so I cannot possible explain all of it. Please be assured though most of the best practices are already in place.

the problem is not the number of rows - I have 1 billion rows app which is blazing fast but complexity of calculations is really low. If all you do is straight sums - it is one thing, if you have to do nested aggrs and dynamic set expressions - totally different. We already spent a good deal of time optimizing it but it is super complex and our users require raw level of details which does not help of course.

Re: please help build a new server - HP DL380 or DL580

Mon, 28 Jul 2014 09:54:33 GMT

A few thoughts (and I may have made some implicit assumptions - so correct me where I am wrong) :

The 4-socket architecture should only be slower if it has to access the memory in an inefficient manner.

So I wonder why would the memory architecture play a role if your dataset is rather small row-wise? They should all fit nicely into one NUMA node (256GB), shoudn't they. You want to have a 1TB of RAM - how come 90million rows amount to such a big dataset?

If these are rather the user caches that are not managed correctly - how about testing that? How about creating 10 documents with the same content, but partitioning your user base to access them in cohorts.

Now if 90 million rows are big because they contain many columns - may be you could split it into a star schema with multiple tables? So the columns that are used for calculation are separated from text fields etc. So the calculations will not have to deal with those columns thus reducing inefficient data transfer.

Another thing is you might want to check is whether you could exchange the data size for the calculation complexity. May be you could pre-calculate a few things or create specialized tables. In other words how about denormalizing your dataset to speed up calculations. If you only have 90 Million rows - extend it into a dataset that has 9 Billion rows but has more or less anything you need to calculate your stuff in a more straightforward way.

This might be totally irrelevant to your situation but I just want to throw that at you. Just for the sake of hearing your reaction if nothing else.

Re: please help build a new server - HP DL380 or DL580

Anonymous — Mon, 28 Jul 2014 13:32:08 GMT

sorry if I was not clear, but we have more than one app and the app I am testing does not take 1Tb of RAM. That app takes about 4-5Gb of space compressed and 20Gb uncompressed in RAM. I also tested a few others apps - they all were very slow.

I understand about app optimization but at this point I am just trying to make the new server work at least with the same performance. That app I am testing is very complex and we already spent a lot of time optimizing it using different techniques. I already preload everything I can but a lot of metrics are not additive and cannot be calculated in a script unfortunately.

Re: please help build a new server - HP DL380 or DL580

Thu, 28 Aug 2014 11:28:15 GMT

Hi Scalability Team,

We have done the tweaks which are mentioned in above threads. We also raised support request 00330211 - Qlikview compatibilty issue with HP DL580 Server. Team says that they cannot help on this scalability team could help to resolve the performance with HP DL580 servers.

Could somebody suggest what should be the next steps taken or the server is incompatible with Qlikview.

Thanks,

Sagar

Re: please help build a new server - HP DL380 or DL580

Anonymous — Thu, 28 Aug 2014 12:19:29 GMT

Hi Sagar,

I worked with QV on this and went through their benchmarks. They analyzed results and came back with the bunch of charts and confirmed that our DL580 performs similarly to other DL580s they tested.

They told us that DL380 is a much faster server but not that capable if you have more than 100s of _concurrent_ users and this there DL580 will shine. They told us DL380 is like a Ferrari and DL580 is like a school bus - first can only ride two people and very fast, while school bus can ride 100 people but slow.

I really wish we knew this before we've made the purchase especially because Qlik was involved in this deal and architect from Qlik suggested this server to us since we asked specifically for the best hardware configuration. They also knew the complexity of our app and our business case.

So now Qlik suggested to get DL380 but the money was spent and looks like we are stuck with super powerful 4 times more expensive server which is 4 times slowed than DL380.

Another interesting thing that if your app does not have complex calculations like ours, the difference would not be so huge but still it is unlikely based on these tests that 580 can outperform 380 strictly looking at performance measures.

I am going to mark my answer as correct since the above was confirmed by Qlik and I really hope it will help someone not to make the same mistake.

Re: please help build a new server - HP DL380 or DL580

Mon, 01 Sep 2014 10:11:12 GMT

Thanks Borys for your quick response.

Re: please help build a new server - HP DL380 or DL580

Anonymous — Wed, 26 Nov 2014 17:08:24 GMT

i thought i would post an update...HP was kind enough to take DL580 back and we will be getting newly released Gen9 HP DL380 which has more RAM bandwidth and faster CPUs (v3 line). I will update you guys again once we get it and test it, but so far specs for gen9 looks promising - especially new HP smartMemory which is supposedly runs at 2133 MHz (but at that speed looks like RAM is maxed at 512Gb).

Re: please help build a new server - HP DL380 or DL580

julian_rodriguez — Sun, 15 Feb 2015 16:37:13 GMT

Hello Borys,

I have read your post and we have a similar problem with a cluster with 5 DL560 gen8 HP servers. We notice that these server are slower than another DL360 gen7.

How was your change?... The performance were improved with the Gen9 HP DL380?... we are considering seriously this option, but it will be great to hear your experience before to make a decision.

Thanks in advance...