Troubleshooting Qlik Sense Upgrades

Q: We have upgraded Qlik Sense from June 2020 Patch 4 to May 2021 Patch 5 , we have installed bundled version, PostgreSQL version 9.6. How to create separate DB node from bundled version and how to upgrade postgreSQL from 9.6 to 12.x version?

A: So if you want to migrate from bundle to standalone you could follow this article: How to Configure Qlik Sense to use a stand alone PostgreSQL.

If you want to stay on a bundled postgres and run version 12.x here is an article that could help you: Qlik Sense May 2021 Upgrade bundled PostgreSQL to 12 5 version.

As always, my recommendation is to test this in a test environment before applying the process in Prod.

Q: Can we have more than one windows version in the Qlik Sense cluster nodes?

A: It should be technically possible but not recommended and perhaps not even supported. It is important to keep your cluster as identical as possible as there could be feature/settings difference between Operating System version causing inconsistent behavior.

Q: I tried an Update, but it hangs durring removal of older patch.

A: It is possible to manually remove the patch and then update to the newer patch. Sometimes is happens that for some reason the installer is not able to perfom this operation successfully.

Q: Do the streams and applications carry over to the new version?

A: Yes, they do along with all your settings.

Q: Hello, is there anything specific when upgrading a multi-node environment (in terms of installation and troubleshooting)?

A: In a multi node environment it is recommended to stop all nodes and start by upgrading the central node. If you are using the Qlik Sense failover feature, make sure which node is having the role of central node before stopping the services. Also, when you stop your nodes, always stop the rim nodes first and then the central node at the end.

Q: We are multi-node and multi-environment. Whenever upgrading, each node takes a fair amount of time, is this normal?

A: Yes, and this is most probably because of the dispatcher service which covers multiple subservices.
In addition to that it is possible that a 3rd party software such as an antivirus is scanning each and every upgrade operation like copy files which as a result slows down the whole process.

Q: I get the error qlik notification service already exist. Support says its under investagation. Would upgrade to the a newer version fix this?

A: I wouldn’t recommend upgrading a non-working environment except if advised otherwise by the support team. This is a risk to make things worst.

Q: Is this the same process for QlikView Server?

A: While there are some similarities especially when it comes to the steps prior to the upgrade such as testing, system requirements and backup. However the actual process is different. There are articles and help documentations covering this such as: Upgrading and Migrating Best Practices for QlikView Services

Transcript:

Hello everyone and welcome to the September edition of Support Techspert Thursdays. I’m Troy Raney and I’ll be your host for today's session. Today's presentation is Troubleshooting Qlik Sense Upgrades with Bastien Laugiero. Bastien, why don't you tell us a little bit about yourself?

Hi everybody. So, my name is Bastien Laugiero. I’ve been working at Qlik for almost six years now focusing primarily on Qlik Sense on-promise; and I recently moved to a new role as a program delivery manager focusing on our SaaS platform.

Okay, and what are we going to be talking about today?

So, today we're going to talk about Qlik Sense upgrade and troubleshooting approach. The first thing I would like to talk about is the requirement and preparation for a successful upgrade. Then we're going to go through a couple of demos: one that shows an upgrade failure and another one that shows a platform down after upgrade. So, basically a successful upgrade, but a platform that won't come up. And there's gonna be some takeaway at the end of the sessions, some articles being shared and so on.

Okay. So, when it comes upgrading Qlik Sense, what constitutes a successful upgrade?

There is never any guarantee that an upgrade is going to be successful, right? We are talking about software here. So, there are however key factors. It’s the amount of preparation we put prior to the upgrade. That's basically going to guarantee at least a recovery in a working state.

And what kind of preparation would you consider required for an upgrade?

A very simple step and often missed is to read the documentation, right? So, you can look at upgrading Qlik Sense and you will end up here. So, here the whole process is being described. Something that you really not want to miss is to look at the system requirements, the precautions, backup and so on. But everything is detailed on this page here.

Okay. Help is a great place to start. Where else can admins look for guidance?

Take a look at our Knowledge Base and Release Notes. So, this can be found in our Community. If you go under Community.com, you've got Support and Release Notes here. You can also search in our Knowledge Base. For instance: “Upgrade to Qlik Sense August 2021” and see if there is any results.

How do customers really know what version of Qlik Sense they should be upgrading to?

That's a good question. So, if you go to Qlik.com, if you go under Support and Download; you're going to end up in our Download site. Select your product “Qlik Sense” and you will have all the versions available.

So, the latest should be on top there?

Exactly. And that's what we see here on the screen, right? August 2021, and we've got the initial reason the patch one available.

And where can you find what's in the release, like features or fixes you might want?

For that, you can go to Help.Qlik.com, under “Qlik Sense on Windows” you're going to find this “What's New in Qlik Sense,” and you can select the version that you are targeting to upgrade, right? We're gonna talk about August 2021, and you've got a list of different features and so on.

Yeah, a lot of new features there.

The last thing I always recommend to do is: to look once again at the Release Notes, because the reason it contains the defect fixes that are available as well as new features, settings that needs to be enabled to enable potential features. So this one is very very rich in terms of information.

Okay, and if someone like with this version that's the latest, it already has a patch. Should we upgrade automatically to the latest patch? Is that possible?

So, no. We're going to take a quick example here. Let's say that: you want to upgrade from Qlik Sense May to Qlik Sense August patch 1. You cannot upgrade directly to August patch 1. You will need to first upgrade to the initial release, and then to the patch.

Okay.

So, once you are on the initial release, so let's say August 2021, you don't need to upgrade to patch 1, patch 2, patch 3, patch 4. You can go directly to the latest patch.

But you need to start with the initial release first?

Always, yes.

Now, I know that in IT in general, but especially when upgrading Qlik Sense, the most important step is backing up. What specifically should be backed up so that Qlik Sense can be restored if anything unexpected happens?

The first one will be the product backup, so Qlik Sense backup. You can go into our Help site. And here you've got the process to actually backup your Repository Database, important files. That's the first thing you need to do. We also provide some Qlik Fix video on how to do that step-by-step as well to follow here. The second thing that you can do is to take a server backup or virtual machine snapshots. This is often easier and faster to restore from this type of backup.

What kind of failures can occur when upgrading?

There are two type of failures in my opinion. The first one will be: so you get a pop-up that shows that your upgrade failed, and we're going to see that in the next demo. The second one is the installer says that you've upgraded successfully to target release; however, when you start up your services, you can't access the Hub, you can't access the QMC. So, the platform seems to be down even though the Windows services are up and running.

Okay, well, let's jump to the demo. What did you want to show us first?

So, here we have this virtual machine running Qlik Sense November 2020, and we're gonna basically upgrade it to Qlik Sense May 2021. As you can see here, I’m not going to jump to February in between. I’m going to go directly to May. I’ve been taking all the precautions that I was talking about before. I’ve checked my system requirement. I took my backup. Now I’m just gonna run the upgrade. Always run your installer as an Administrator.

And you've downloaded it locally and now you're just running it?

Exactly. Alright so, here we're gonna click on Yes, because we want to run the installer. Click on Upgrade.

I like that the installer recognizes the pre-installed version, that's cool.

Accept the terms. Service account information. And here, enter the Repository Database password for this demo. I’m going to just put the wrong password. Just a quick note on that: I’m able to make this upgrade fail, because I have stopped all the services. Normally, if the services are up when I’m actually running the upgrade, there will be a check of the super user password. And if it's wrong, the installer will tell you that. We want to install the latest bundles. We accept the terms for the bundles. And we're going to start the upgrade process. So, it's going to fail at the Qlik Sense Repository Database. Here, we've got our upgrade failure. So, in most cases, you're going to see actually a meaningful error message here that you can actually search then into our Knowledge Base. And we can just search for “Password Validation Failure.” We can just filter on “article only.” And here, you will end up with different articles related to that error message.

What if the error message wasn't so helpful or clear is that one? Was that a link to the log files from the upgrade?

Exactly. So, this is a link to the installation or upgrade log file. So, this is important, because if you have to reach out to a Support to help you troubleshooting this issue, they will often ask you for the installation or upgrade logs. So, these are the logs that we are expecting to receive. So, if you click on that link, you're gonna end up on this folder here. So, this was the logs that were created during my initial installation on November 2020. As you can see, it just gives us the release name here. So, there are a lot of different logs, and the reason is because Qlik Sense has multiple services to upgrade. And each service will be represented by a specific log file. So, this is the order it went through. So, the first one as you can see it's PostgreSQL, so Qlik Sense Repository Database, and then it will continue here. If it fails at any time, let's say it fails at the Scheduler; so, it's not going to go through the other services. However, you will see a rollback log file on top of each of the ones that were already upgraded. So, you're going to see this these five here, plus an additional five with a dash roll back at the end. So, what you - the log that you want to open is actually the latest one that don't that, don't have this rollback…

Reference.

Exactly. If we go back to PostgreSQL. So, there was no rollback here, because it just failed at the password validation. So, it fails right at the start, so there was no rollback to perform. But this is basically the latest log that we've got from the installer. So, this is the one we're going to look at, right?

So, when you're looking at a log file what do you usually search for to zero in on what is important?

Yeah so, you can search for “Value 3.” So, “Value 3” it's an error code that will trigger a rollback basically. The other thing that you can search that is always associated with that return Value 3 is “Error Code 1603” here. If you start to read a little bit above that error, you can find back our Password Validation Failed error message. However, you may have other error messages here that you can search in our Knowledge Base.

Okay so, if it fails, you're not sure what the message means, you could theoretically copy that message that you find here in the log file, search in a Community, and hopefully find more information? But this is definitely a log file that Support would need if they if you require some additional assistance?

Exactly.

Okay.

My advice here is that: when you send logs to Support, don't just select the latest log. Take all the logs that are associated to your upgrade basically.

And it's pretty easy to see which ones they are, because in this in this case, we're upgrading to a different version and it's designated there in the first part of the log which version wrapped it into.

Correct.

Now quick question: since this upgrade failed, is the November 2020 still active and functioning on this system?

Yes. In fact, we can take a look at the services here.

So, it didn't really affect the current installation.

No, it didn't. The installer is built in a way that if there is any failure, it will roll back to the previous state. However, it may happen that this rollback is not happening correctly, and therefore this precaution of taking a backup becomes critical.

Yeah, I totally understand working in IT that everything starts with a backup. Okay, you had a second scenario you wanted to run us through. Tell us about that?

All right. So, we're gonna look at scenario where the upgrade is actually successful, but you are not able to start your platform. My services are started; however, when I try to launch the QMC, I’m just getting a 404 error. So, well, the first thing I want to do since I am on the Qlik Sense Server here; the first thing I want to test of course is if I’m facing any DNS or network issues, I’m going to try localhost. And I’m going to basically end up with the same problem, which means that my Qlik Sense platform is down here. The troubleshooting approach is a little bit different, right? Because we we're not going to look at the upgrade or installation logs, because those who are successful. What we want to look at is the Qlik Sense service logs.

Okay, taking a look at the services, would you help explain what all the services do exactly?

So, we've got the Qlik Sense Repository Database, which contains basically the entire configuration of your platform. So, this one is the first one that needs to function basically for the other services to pick up the configuration. The next ones are the Repository Service and Service Dispatcher. And then you've got the remaining ones which are the Engine, Printing, Proxy Scheduler that all depend on the Repository Service to be started to function properly. When you start the services here in Windows, there is an entire startup phase that's happening in the background we're going to look at through the logs.

So, since the Repository Database starts first, is that where you would begin looking in the logs?

Exactly. So, there is a much easier way for this one at least to to see if this one works fine than looking at logs.

Okay.

We want to connect to the database, right? This is a Postgres database running in the background. One thing that we can do is to actually try to access the database. We usually do that with pgAdmin. There is an article actually explaining how to install it and configure it. It's called “Installing and Configuring pgAdmin4.” So, you can find that in our Community Knowledge Base.

And that is a free utility, right? That people can download?

Exactly. So, what we want to do is to basically try to connect to this Repository Database. So, I’ve already have a pre… connection that is already created, but the setup is very simple, right? So, the host is “localhost” because my service is installed on this machine. Got Postgres, and I will put my super user password during the setup. If we try to connect here, I’m gonna do “Connect Server,” it's gonna ask me for my super user password. And here, I can see that I can access my databases. I can open them, the connection here is working fine. So, I can assume that this service is actually functioning properly.

Since the database is running fine, what's the next step?

Will be to look at the Qlik Sense Repository Service, right? For that, we need to look at the Qlik Sense logs.

And where do we find those?

So, the Qlik Sense logs are - you can find them under C: > ProgramData > Qlik > Sense > Log. Here you've got all the logs for each services.

So, each service has its own log folder?

Exactly. Got the Repository folder, and we're going to look at Trace and System. So, if we look at this log, just scroll to the bottom here. We can see an error. .

Ah, right there.

What we are seeing here, if we look at the couple of lines above is: it's trying to connect to the QSR Database. It's waiting 300 second for a response, which is five minutes. So, we can assume that this service is not able to actually connect to the QSR Database; and that's most likely the root cause of my issue. So, once again, this is just a troubleshooting approach. You may have different error messages in there. The idea is just to take this error message, search for it in our Knowledge Base and find some potential root causes and solution.

Okay.

In this particular case, so, I just modified manually the connection string the Qlik Sense Repository Service so that it will fail to connect to my database. I can quickly show that. So, if you go under C: > ProgramFiles > Qlik > Sense, you've got something called Repository > Util > QlikSenseUtil.

Okay, so what is this utility?

So, this utility it's basically providing several information: such as your connection string, your service cluster setup, and more. We usually use it for service clustering connection string editor. So, if you start to read your connection string. So, this is my QSR connection string here. What I’ve done is simply modify this port so that it won't be able to reach my QSR Database, because the default port is 4432 and not 5432. If I start to save this, and exit; I should now be able to actually start my services.

Now you're restarting because you actually changed that connection string, right?

Exactly. So, I need to re-trigger a connection attempt.

Okay, and once that's finished, you should be able to connect to the QMC?

Then yes, I should be able to access my QMC if I’m not facing any other error of course. There should be a line in this log that says “Startup Phase Completed.” So, you can just search for that, and there I can find it. So, that means my Repository started correctly.

If that started up fine, but we still weren't able to access the QMC or Hub, what would be next?

The next service that I would look at is my Proxy Service, because my Proxy Service is the one that delivers the web interface.

Okay, so, that actually hosts the QMC?

Exactly. I can take a Qlik look at my Proxy logs. So, same thing: Proxy > Trace, and then we've got System Proxy.

And I noticed before there are a couple warnings there.

Yes. So, that's because of the startup phase, right? it tries to connect to my Repository, but since my Repository was not up and running; so, it failed, but it's retrying, right, until it can. Then you can see the last line on this log is “Proxy is Running.” So, that means this service started up fine.

Great.

It's always important to look at the entire context, and not just stop at the first error or warning, because it might be actually a false alarm, right? So, if we now try to access our QMC, and our services are working, and I can connect to my environment.

Great. And would you say those are two of the most common issues that people face when they're upgrading: the upgrade fails or the upgrade succeeds, but the services don't work correctly?

Yes. Exactly. Now, the root cause can vary from case to case.

Sure.

From version to version as well, because our R&D team is working very hard on trying to fix those potential use cases, but there might be new ones coming in other release, right?

Just as a reminder, we will be sharing the links to everything that's been discussed today along with the recording. Bastien, are there any other resources you'd like to highlight?

So, I just wanted to highlight a couple of things. Here are a few links around our System Requirements for Qlik Sense, how to back-up your Qlik Sense site, a link to the Release Notes and the upgrade process. And finally, I’m highlighting here a couple of articles that will help you troubleshooting your Qlik Sense installation or upgrade if any failure occurs. So, it's kind of a summary of what we just discussed today.

Okay, great! Thanks. Okay, now it's time for Q&A. Everyone, go ahead and enter your questions in the Q&A panel on the left side of your On24 console. Bastien, which question would you like to address first?

All right. So, the first question is: how to prevent Qlik Sense Enterprise on Windows from rewriting config files such as C: > ProgramFiles > QlikSenseproxyproxy.exe.config, which as for example: the keeper live timing parameter.

So, unfortunately this is not possible. So, when we do a major version upgrade, these files are being replaced or overwritten with the new ones corresponding to the updated version. My best advice here is to actually keep track of these changes you are making in these config files, to then update them post upgrade. Do not replace these configuration files from a previous version to a newer one, because they could be different, like new settings for example or deprecated settings. So, it's very important to keep the original file and just change the setting that you had configured before the upgrade.

That's good to know. Next question?

Do all nodes in a separated environment needs to be disconnected from the Repository before proceeding with the upgrade?

So, in a multi-environment, it's recommended to stop all nodes and start by upgrading the central node. So, if you are using a Qlik Sense failover feature, that's a Qlik tip here: make sure which node is having the role of central node before stopping the sources. It's very possible that actually a rim node became a central. And this one you will either need to move back the central node role to the original central. You will need to consider this stream. Know that the central node and basically be the first one to upgrade. Also, when you stop your nodes, always stop the rim node first, and then the central node at the end. When it comes to the rim node, there is not really any - it's not really important in which order you are going to stop your rims. But the central should be the last in any case.

That makes sense. Next question?

Please include any tips or suggestions you may have regarding certificate changes that occur during a Sense upgrade.

So, there are normally no changes to the certificate during an upgrade. They're not being touched. It has happened in the past for some specific releases that the certificate needed to be renewed as a post upgrade activity, but this is always documented in the Release Notes; so, with step-by-step instructions as well. So, finally the certificate is also part of the backup that you take before upgrading, so if sometimes it happens that something can go wrong; and in this case, you will have this backup to cover you. But in general, there is no change on the certificate that's happening in Qlik Sense.

Okay, next question?

So, we had a lot of problem in the past with broken certificate that then needed to be recreated reinstalled, so that's why we postpone upgrade unfortunately.

So, this issue - it's often environmental, and can be caused due to many different reasons such as: a security policy applied on the server, or third-party software such as antivirus blocking or blocking the files basically or corrupting them. So, as mentioned before, my best advice here is to make sure that you are backing up the certificate before upgrading. Which means that if they get corrupted post upgrade or at any time, you can safely restore them without any problem.

That's interesting that it's not, that happens people, but it's not actually Qlik Sense that's affecting it. It's just being affected by the environment.

Correct, and that's the thing with Qlik Sense. It runs on Windows and is very dependent on the overall environment that it's installed on.

Okay, next question.

Upgrading multi-node Qlik Sense from November 2019 to February 2020 was unsuccessful. Some users can log on, some cannot. How to check the root cause in the log file?

So, it's very difficult to say to say right now without any clear symptoms and clear message. So, what I will do is: based on your question, it sounds like the upgrade was successful from the installer (which was what we discussed previously in the demo), but now you are you are facing some post upgrading issue, right? In that case, I would just suggest to focus first on the symptoms that the users are experiencing. For instance: is there any specific error message or specific behavior that they experience when it faced with the problem and start from here. Alright. You can search this behavior or error messages in our Knowledge Base. That should give you a good point to start, here.

And if it has to do with logon issues, what log files or what service log file would you recommend looking at?

So, if that's pure logon, I will first check the Windows event log. So, that depends also which authentication method you are using, right? Let's say that you are using Windows authentication in this case. The first thing to know is that Qlik Sense does not take care of the authentication. So, if you use for instance Windows, you will need to check the Windows event log security at the time the user had the failure to see if there is an error messages there. If you are using single sign-on, you probably need to check your in the single second side as well.

Very good, thanks. Next question?

All right so please cover upgrades related to Postgres Repository sitting on node outside of Qlik.

So, the process is no different, right? Qlik Sense will automatically detect where the database is and will proceed with the upgrade. However, note that Qlik Sense will not upgrade your Postgres. You are responsible to upgrade it if you wish, making always sure that the version you choose is actually listed under the Qlik Sense system requirements for the version that you are running of course.

Okay.

If moving from one version to another and you are currently several release out of date, do you need to install each releases in turn?

The answer is no. You do not need to upgrade to each release in between. However, if you want to install a patch, you need to first install the initial release associated to that patch. So, let's just take a quick example: you are running Qlik Sense February 2018 which is a very old version, and you want to upgrade to Qlik Sense May 2021. You can do that. You don't need to upgrade to each major version in between. However, if you want to upgrade to Qlik Sense May 2021 patch 4, you will first need to upgrade to Qlik Sense May 2021 initial release; and then apply the patch. And here the for the patch, same applies. You don't need to patch to apply the patch one by one. You can go directly to patch 4.

Thanks for clarifying that. So, next question.

Where can we find the latest patch?

So, this is something we quickly saw in the presentation before. You can find it in our download page which is accessible from Qlik.com in the Support area of that website, right?

Do I need to apply all patches or just the latest?

This is kind of related to the question we just answered before. So, no you don't need to apply all patches one by one. A patch is cumulative. So, which means that you can simply install the latest one and all fixes from the previous patch will also be included.

The next question is: we got error 1068 “The dependency service or group failed to start” with the Repository Service. What does that mean?

In my experience, so, this error message happens when you try to start one of the Qlik Sense Services in the service console, the Windows service console; and this basically it's very possible that the service account does not have the right privileges to start the services. I know that there are some articles that use about this error here. So, we'll just search directly for the error message in our Knowledge Base, and you should find information on what caused this and what is the potential solution.

Good.

What right does the QS service account require? Does it also need a license?

So in terms of Windows privileges, it's recommended that the Qlik Sense service account is part of the local administrator group on the Qlik Sense server. However, if for security reason this is not an option; so, it's very often that your IT department will not allow you that sort of privileges. There is a way documented on Help.Qlik.com to run the Qlik Sense services without admin privileges. When it comes to the license, the Qlik Sense service account does not need the license. The license is really about accessing the Hub, opening and creating application, so you don't need to have a license to access the QMC and do administrative tasks there.

That's good to know. Okay, we're getting near the end. Next question?

So, the next question is: usually I test future new releases of Qlik Sense server with the desktop version. Since there is no desktop version anymore, how do you guys test new versions before upgrading your whole cluster?

That's an interesting question. So, and here we are really talking about how deeply you should test your environment before upgrading. So, my recommendation here is to always have a Qlik Sense server test environment, because Qlik Sense desktop is very good to test your application and how they will behave with the updated version. However, something that you also need to consider testing is the server part of the software. Potentially the overall upgrade process as well, so that you are familiar with all the steps that needs to be taken when it's time to upgrade your production. So, testing application it's of course needed, right? But you also have the server side, the server parts of your upgrade that also need to be tested, in my opinion.

Is that like a license they could get from their account manager to have a test license?

Yeah, this is - you can reach out to your company to see what your options are, but yeah definitely having a test license is possible for a given period of time here.

Okay, we have time for one last question.

So, is the upgrade similar for QlikView?

Interesting question. So, while there are some similarities, especially when it comes to the steps prior to the upgrade such as the testing the system requirements, the backup and so on. The actual process is actually different. We are talking about two different products that work different way, right? So, there are there are definitely articles and help documentation covering this. So, you can search in Help.Qlik.com for the upgrade process, and you can also look at our Knowledge Base for upgrading and migrating best practices for QlikView.

Great.

I hope this was useful. Thank you all for attending and have fun with your upgrade!

Thank you everyone! We hope you enjoyed this session. And thank you to Bastien for presenting. We appreciate getting experts like Bastien to share with us. Here's our legal disclaimer. And thank you once again. Have a great rest of your day.

Troubleshooting Qlik Sense Upgrades

Troubleshooting Qlik Sense Upgrades

Environment

Administration

Configuration

Deployment

Download Page

How To

Training