please help build a new server - HP DL380 or DL580

[Anonymous]

Hi guys,

I am looking for a second opinion as we are already working with our vendors and Qlik reps. We just purchased a new server and I found out that it performs terribly with QV even though it is 4 times more expensive and powerful. We are going to return this server to our vendor (HP) and get something different hence I am asking for an opinion on an ideal and fastest for QV HP server.

Right now my company uses a single server to run all QV services, including QVS, QDS and AccessPoint. It is very fast but we are running out of RAM and it is not expandable so we are in the process of getting a second server for QV which will only run AccessPoint and QVS.

Our old server:

2 socket HP DL380P GEN8

Intel Xeon CPU E5-2690 2.90GHz

256Gb of RAM

Our new server (which we are going to return):

4 socket HP DL580 Gen8

Intel Xeon CPU E7-4870 v2 2.30GHz

1Tb of RAM

I was super excited about DL580 and when it arrived I started running some tests. We have a fairly complex dashboard with 90+ million rows and a lot of extensive calculations and I found out that our old server was 2-5 times faster on the most of the tabs. We tuned the server using QV tech. paper specifically made sure that max. energy profile is used, hyper threading is off and NUMA is off. Turbo Boost is on.

It helped a little bit especially then we disabled NUMA – I am confused why though because according to QV, NUMA is not a problem on 11.2 but apparently it is. So after we disabled NUMA, things improved maybe by 10-20% but it was still a way slower than our old server.

We reached to HP and one of their senior architects explained that DL580 has to do more work with RAM because of the way RAM is shared between processors and the clock speed is 1333 vs. 1666 on DL380.

At that point I downloaded and ran 4 different benchmarking tools (MAXXMEM2,NOVABENCH, PassMark and SiSoft Sandra) and all of them showed 2-6 times difference in RAM tests – DL580 was slower again!

Our goals for a new server:

1. 1) Should be faster than the old one
2. 2) Should be expandable to 1Tb of RAM at least
3. 3) Should support at least 50 concurrent users
4. 4) Should be made by HP – rack server HP Proliant family.
5. 5) Our budget is 50k
6. 6) The server will run ONLY QVS and AccessPoint.

I apologize for a long introduction to my question but now I am puzzled what HP server we should pick since DL580 with 4 processors clearly does not meet our needs.

I am not a hardware expert and was relying on our hardware people and Qlik but apparently the configuration they picked did not meet our goals. They are working again to revise the config but I wanted to get a second opinion from a forum and you.

I am thinking now to either get the fastest E5 or E7 and this time only two sockets to minimize memory hops. Also I wanted to see if we can use higher clocked RAM (1887?)

It is going to be a rack server from HP Proliant family – you can actually build it online here

http://www8.hp.com/us/en/products/proliant-servers/index.html?facet=ProLiant-DL-Rack

Any suggestions are highly appreciated!

[Anonymous]

Hi Julian,

we just got the new server and finished testing it. We bought HP DL380 gen 9 - we were waiting for gen 9 to come out because of DDR4 and newer E5.

NEW:

2 CPU 18 Core E5-2699 v3 2.3GHz

768GB DDR4 LRDRAM 2333Mhz

(operates at 1600Mhz)

Out of the box, the new server was 5-10% slower than our old one - here is the spec for it

OLD:

2 CPU 8 Core E5-2690 2.90GHz

256GB DDR3 DRAM 1888Mhz

(operates at 1600Mhz)

After some tweaks and benchmarks, I settled down on this configuration in BIOS:

hyperthreading is off, Enable HW prefetch, Enable NUMA, virtualization is off, Power set to High Performance, Turbo Boost is on

I used our the most complex and resource intensive dashboard which uses set-analysis, aggr functions and such.

After these tweaks, the new server performs about the same as the old one. I have to say that the new server cost us 2.5 times more than the old one.

I also downloaded and ran some benchmarking tools - GeekBench, Passmark and SiSoftware Sandra. While they are not designed to test servers, I could get the idea of the performance relative to our old server.

Even though CPU clock speed on our new server is 2.3Ghz, it beat our old server in all CPU benchmarks. RAM unfortunately was much slower in tests - I am attaching my results if you are curious to look at this.

We also tried to pull out 256Gb worth of DIMMs so mb operates at 2333Mhz clock speed, but it did not make any difference surprisingly for QlikView - it did make some different in software benchmarks though.

To sum up, while we have now more RAM (768 vs 256) and 4 more cores, I do not think it was worth the price difference.

HP people are telling us now that QlikView can go through certification process with HP DL380 server - part of this process would be to make sure that QlikView engine uses all the latest innovations and advanced command sets supported now by E5. I will let our QlikView rep know that but unless Qlik is willing to invest time and money in this process, I do not have much hope here.

[julian_rodriguez]

Hello Borys

First of all, thank you for your detailed answer.

I'm surprised (sadly I should say) with your experience. I was hoping that the new server were better than the old one.

I have to ask, because I have not it clear: Which server have best performance between these two: DL580 gen8 or DL380 gen9?

Or definitly the old one is better than the newer DL580 or DL380?

HP people told you when they will be giving the certification process results?

Thank so much.

[Anonymous]

I have corrected my post as our old server had 8-core not 16 core CPUs - my bad.

Based on my testing our old DL380 gen7 is the fastest at this point, closely followed by DL380 gen9. DL580 gen8 was way too slow for us which can be explained by QPI thing between 4 processors so I would not go that one.

We just had a call with HP architects and looks like they will be willing to loan a couple more processors to us for test - they say it could be something because of high number of cores we have on gen9 and if we downgrade core-wise, RAM should work much faster. But it is a speculation that needs to be verified - if we ever get other processors to test out, I will post an update.

As far as certification, it is something that Qlik should initiate and work with HP - nothing you or me can do here.

[Anonymous · Accepted solution]

Our final configuration

HP DL380 Gen9

2 CPU 8 Core E5-2667 v3 3.2 GHz

512Gb of DDR4 LRDRAM 2333Mhz

(you could go up to 768Gb but it will force it to work at 1600Mhz)

BIOS settings:

     hyperthreading is off, Enable HW prefetch, Enable NUMA, virtualization is off, Power set to High Performance, Turbo Boost is on

Read below for more details.

We just got E5-2667v3 CPU (8 core 3.2 GHz) and I just finished testing it. We also purchased 768Gb DDR4 of RAM but at that amount it can only work at 1600MHz clock speed and I wanted to test its native clock speed at 2333Mhz.

Boy that made a difference! While I saw 15-20% improvement in performance for this new CPU / 768Gb of RAM, once we removed some DIMMs to get down to 512Gb (so RAM could work at 2333Mhz), I saw 30-35% performance improvement compared to our old server and for some sheets as high as 60-65%! The last one was on one of our most popular Summary dashboard pages that has over 75 metrics calculated at once. On our old server it would take 15-20 seconds to calculate that page and now it takes 5-10 seconds, quite an improvement.

So my takeaway from this long journey is this:

0) Hardware upgrade should be the last resort - use best practices when you build your apps! spend good amount of time on your data model. If it looks like spider web or spaghetti, redo it. Same with expressions - bad expression can even crash your server no matter how powerful it is. In our case, we had a very decent data model and an aggregated version of our most popular dashboard but it was not enough and our user base was growing rapidly the upgrade was justified.

1) do not trust the hardware specs in the case of QlikView - more expensive / faster hardware, does not mean it will make your QlikView dashboards faster. 4 CPU DL580 server was 4 times slower for us and 3 times more expensive! Luckily we were able to return it to vendor. Pick your most used dashboard with a lot of calculations and test, test, test

2) the more RAM you install, the slower it will be - get high performance RAM (DDR4) and make sure your RAM operates at the highest clock speed it can support

3) higher clocked CPUs matter! if you want the speed, get one with less cores, but higher clock. IF you want capacity to handle more users, get the one with more cores, but it will be slower.

4) also keep in mind that CPUs with large number of cores (above 8), will have some overhead to use RAM so will be slower (but will handle more users concurrently)

5) play/test BIOS settings, I ended up with the following:

     hyperthreading is off, Enable HW prefetch, Enable NUMA, virtualization is off, Power set to High Performance, Turbo Boost is on

6) 4 CPU beasts is NOT a good choice - they are slow with QlikView. Go with 2 CPU servers.

Hope it will help someone! Took us 9 months and I am finally happy with our choice. I always knew that hardware should be picked considering the software it will run and in QlikView case it was very evident!

[julian_rodriguez]

Hello Boris,

I'm glad to read that you have solved your performance issue.

We are tunning the Qlikview applications, spliting one big model on 5 models (one for each bussiness area) and doing Publisher reductions trough some regions or divisions.

At the end, we have improve the performance, with the same hardware.

So, I would add to your summary, that it's important to analyze if the Qlikvew Application is designed as best as possible, or if you can improve it.

Thanks for sharing your experience, and I'll take note about your advises.

Best regards

Julian

[Anonymous]

thanks, Julian, yes I forgot to mention that we exhausted our options improving the model - actually I even built an aggregated version of our dashboard and used chained document feature. Another problem, we have 25 dashboards now and 20 developers so it was getting crowded on our old server. I will add a note to my post though about data model / calculations - personally I normally spend 60% of my time on data model which always pays off when you deal with volume above 10MM.

[Troy1]

What are your thoughts on below cpu in comparison to what you have chosen, have you had chance to test any of them. what are your thoughts on pros and cons.

E5-2667 compare to

E5-2697 v3  - 14 cores

E5-2690 v3 - 12 cores

E5-2687 v3 - 10 cores

http://ark.intel.com/compare/83361,81909,81713,81059

• Does # of cores reduce performance?
• Is it better to have more cores if you want to balance performance and concurrent users.
• Does base speed matter or do you look at max turbo speed.
• Does it matter if some of these CPU may not be in tested list of Qlik Scalability Center. is it better to chose from their list?
• can 512gb Ram be distributed with these CPU in optimal fashion to provide max performance?

[Anonymous]

if you want a nice balance between speed (performance) and capacity (number of concurrent users), go with higher clocked (base clock not turbo) but less cores CPU. I am pretty happy E5-2667v3 and you can read my story above for all the troubles we had with more expensive but slower servers.

We did work with Qlik architect from Scalability center and he was aware of our situation and recommended 4P DL580 which turned out a really bad choice for us so be VERY cautious of what they tell you - they are good guys but in the end of the day it is you and your company that matters and only you know what's good for you, so do you own research, especially if you upgrading your existing server. In our case our new 4P machines turned out to be x4 times slower!

I also spent a number of hours talking with a hardware architect from HP - a super knowledgeable guy. He told me right away that 4P machine is not a good choice if you talk speed no matter what and 2P is the only good choice.

When he said that the more cores you get, the more work machine does to handle RAM intensive apps so it becomes slower and there is also direct correlation between # of cores and clock speed - more cores less the clock speed. But at the same time server can handle more concurrent requests.

So ultimately it is your choice. If you are not constrained by $$, i would go with faster 2P machine (meaning less cores and higher CPU clock) and in future you can always get another one and set up a cluster if you ever grow that far.

[Anonymous]

as far as RAM configuration, you want the fastest clock speed you can get which means lesser total amount. You can get an idea by using "hp memory configurator" - just google it. Go with less RAM but the fastest configuration.

[Troy1]

Thanks Boris

How big is your dashboard (size MB, GB) and how many dashboard are running at same time + how many concurrent users have you tested on your E5-2667 machine.

You had 512gb configued at highest speed. Qlik suggest that 384gb is most optimal, do you think 512gb in your case is most optimal configuation.