Search or browse our knowledge base to find answers to your questions ranging from account questions to troubleshooting error messages. The content is curated and updated by our global Support team
Featured Content
-
Optimizing Performance for Qlik Sense Enterprise
This Techspert Talks session addresses: Understanding Back-end Infrastructure Measure Monitor Troubleshoot Environment Qlik Sense Enterprise on Wi... Show MoreThis Techspert Talks session addresses:
- Understanding Back-end Infrastructure
- Measure
- Monitor
- Troubleshoot
Environment
- Qlik Sense Enterprise on Windows
00:00 - Intro
01:22 - Multi-Node Architecture Overview
04:10 - Common Performance Bottlenecks
05:38 - Using iPerf to measure connectivity
09:58 - Performance Monitor Article
10:30 - Setting up Performance Monitor
12:17 - Using Relog to visualize Performance
13:33 - Quick look at Grafana
14:45 - Qlik Scalability Tools
15:23 - Setting up a new scenario
18:26 - Look QSST Analyzer App
19:21 - Optimizing the Repository Service
21:38 - Adjusting the Page File
22:08 - The Sense Admin Playbook
23:10 - Optimizing PostgreSQL
24:29 - Log File Analyzer
27:06 - Summary
27:40 - Q&A: How to evaluate an application?
28:30 - Q&A: How to fix engine performance?
29:25 - Q&A: What about PostgreSQL 9.6 EOL?
30:07 - Q&A: Troubleshooting performance on Azure
31:22 - Q&A: Which nodes consume the most resources?
31:57 - Q&A: How to avoid working set breaches on engine nodes?
34:03 - Q&A: What do QRS log messages mean?
35:45 - Q&A: What about QlikView performance?
36:22 - Closing
Resources:
Qlik Help – Deployment examples
Using Windows Performance Monitor
PostgreSQL Fine Tuning starting point
Qlik Sense Shared Storage – Options and Requirements
Qlik Help – Performance and Scalability
Q&A:
Q: Recently I'm facing Qlik Sense proxy servers RAM overload, although there are 4 nodes and each node it is 16 CPUs and 256G. We have done App optimazation, like delete duplicate app, remove old data, remove unused field...but RAM status still not good, what is next to fix the performace issue? Apply more nodes?
A: Depends on what you mean by “RAM status still not good”. Qlik Data Analytics software will allocate and use memory within the limits established and does not release this memory unless the Low Memory Limit has been reached and cache needs cleaning. If RAM consumption remains high but no other effects, your system is working as expected.
Q: Similar to other database, do you think we need to perform finetuning, cleaning up bad records within PostgresQL , e.g.: once per year?
A: Periodic cleanup, especially in a rapidly changing environment, is certainly recommended. A good starting point: set your Deleted Entity Log table cleanup settings to appropriate values, and avoid clean-up tasks kicking in before user morning rampup.
Q: Does QliKView Server perform similarly to Qlik Sense?
A: It uses the same QIX Engine for data processing. There may be performance differences to the extent that QVW Documents and QVF Apps are completely different concepts.
Q: Is there a simple way (better than restarting QS services)to clean the cache, because chache around 90 % slows down QS?
A: It’s not quite as simple. Qlik Data Analytics software (and by extent, your users) benefits from keeping data cached as long as possible. This way, users consume pre-calculated results from memory instead of computing the same results over and over. Active cache clearing is detrimental to performance. High RAM usage is entirely normal, based Memory Limits defined in QMC. You should not expect Qlik Sense (or QlikView) to manage memory like regular software. If work stops, this does not mean memory consumption will go down, we expect to receive and serve more requests so we keep as much cached as possible. Long winded, but I hope this sets better expectations when considering “bad performance” without the full technical context.
Q: How do we know when CPU hits 100% what the culprit is, for example too many concurrent user loading apps/datasets or mutliple apps qvds reloading? can we see that anywhere?
A: We will provide links to the Log Analysis app I demoed during the webinar, this is a great place to start. Set Repository Performance logs to DEBUG for the QRS performance part, start analysing service resource usage trends and get to know your user patterns.
Q: Can there be repository connectivity issues with too many nodes?
A: You can only grow an environment so far before hitting physical limits to communication. As a best practice, with every new node added, a review of QRS Connection Pools and DB connectivity should be reviewed and increased where necessary. The most usual problem here is: you have added more nodes than connections are allowed to DB or Repository Services. This will almost guarantee communication issues.
Q: Does qlik scalability tools measure browser rendering time as well or just works on API layer?
A: Excellent question, it only evaluates at the API call/response level. For results that include browser-side rendering, other tools are required (LoadRunner, complex to set up, expert help needed).
Transcript:
Hello everyone and welcome to the November edition of Techspert Talks. I’m Troy Raney and I’ll be your host for today's session. Today's presentation is Optimizing Performance for Qlik Sense Enterprise with Mario Petre. Mario why don't you tell us a little bit about yourself?
Hi everyone; good to be here with everybody once again. My name is Mario Petre. I’m a Principal Technical Engineer in the Signature Support Team. I’ve been with Qlik over six years now and since the beginning, I’ve focused on Qlik Sense Enterprise backend services, architecture and performance from the very inception of the product. So, there's a lot of historical knowledge that I want to share with you and hopefully it's an interesting springboard to talk about performance.
Great! Today we're going to be talking about how a Qlik Sense site looks from an architectural perspective; what are things that should be measured when talking about performance; what to monitor after going live; how to troubleshoot and we'll certainly highlight plenty of resources and where to find more details at the end of the session. So Mario, we're talking about performance for Qlik Sense Enterprise on Windows; but ultimately, it's software on a machine.
That's right.
So, first we need to understand what Qlik Sense services are and what type of resources they use. Can you show us an overview from what a multi-node deployment looks like?
Sure. We can take a look at how a large Enterprise environment should be set up.
And I see all the services have been split out onto different nodes. Would you run through the acronyms quickly for us?
Yep. On a consumer node this is where your users come into the Hub. They will come in via the Qlik Proxy Service and consume applications via the Qlik Engine Service, that ultimately connects to the central node and everything else via the Qlik Repository Service.
Okay.
The green box is your front-end services. This is what end users tap into to consume data, but what facilitates that in the background is always the Repository Service.
And what's the difference between the consumer nodes on the top and the bottom?
These two nodes have a Proxy Service that balances against their own engines as well as other engines; while the consumer nodes at the bottom are only there for crunching data.
Okay.
And then we can take a look at the backend side of things. Resources are used to the extent that you're doing reloads, you will have an engine there as well as the primary role for the central node, active and failover which is: the Repository Service to coordinate communication between all the rest of the services. You can also have a separate node for development work. And ultimately we also expect the size of an environment to have a dedicated storage solution and a dedicated central Repository Database host either locally managed or in one of the cloud providers like AWS RDS for example.
Between the front-end and back-end services where's the majority of resource consumption, and what resources do they consume?
Most of the resource allocation here is going to go to the Engine Service; and that will consume CPU and RAM to the extent that it's allocated to the machine. And that is done at the QMC level where you set your Working Set Limits. But in the case of the top nodes, the Proxy Service also has a compute cost as it is managing session connectivity between the end user's browser and the Engine Service on that particular server. And the Repository Service is constantly checking the authorization and permissions. So, ultimately front-end servers make use of both front-end and back-end resources. But you also need to think about connectivity. There is the data streaming from storage to the node where it will be consumed and then loading from that into memory. And these are three different groups of resources: you have compute; you have memory, and you have network connectivity. And all three have to be well suited for the task for this environment to work well.
And we're talking about speed and performance like, how fast is a fast network? How can we even measure that?
So, we would start for any Enterprise environment, we would start at a 10 Gb network speed and ultimately, we expect response time of 4 MS between any node and the storage back end.
Okay. So, what are some common bottlenecks and issues that might arise?
All right. So, let's take a look at some at some examples. The Repository Service failing to communicate with rim nodes, with local services. I would immediately try to verify that the Repository Service connection pool and network connectivity is stable and connect. Let's say apps load very very slow for the first time. This is where network speed really comes into play. Another example: the QMC or the Hub takes a very very long time to load. And for that, we would have to look into the communication between the Repository Service and the Database, because that's where we store all of this metadata that we will try to calculate your permissions based on.
And could that also be related to the rules that people have set up and the number of users accessing?
Absolutely. You can hurt user experience by writing complex rules.
What about lag in the app itself?
This is now being consumed by the Engine Service on the consumer node. So, I would immediately try to evaluate resource consumption on that node, primarily CPU. Another great example for is high Page File usage. We prefer memory for working with applications. So, as soon as we try to cache and pull those results again from disk, performance we'll be suffering. And ultimately, the direct connectivity. How good and stable is the network between the end users machine and the Qlik Sense infrastructure? The symptom will be on the end user side, but the root cause is almost always (I mean 99.9% of the time) will be down to some effect in the environment.
So, to get an understanding of how well the machine works and establish that baseline, what can we use?
One simple way to measure this (CPU, RAM, disk network) is this neat little tool called iPerf.
Okay. And what are we looking at here?
This is my central node.
Okay. And iPerf will measure what exactly?
How fast data transfer is between this central node and a client machine or another server.
And where can people find iPerf?
Great question. iPerf.fr
And it's a free utility, right?
Absolutely.
So, I see you've already got it downloaded there.
Right. You will have to download this package, both on the server and the client machine that you want to test between. We'll run this “As Admin.” We call out the command; we specify that we want it to start in “server mode.” This will be listening for connection attempts.
Okay.
We can define the port. I will use the default one. Those ports can be found in Qlik Help.
Okay.
The format for the output in megabyte; and the interval for refresh 5 seconds is perfectly fine. And then, we want as much output as possible.
Okay.
First, we need to run this. There we go. It started listening. Now, I’m going to switch to my client machine.
So, iPerf is now listening on the server machine and you're moving over to the client machine to run iPerf from there?
Right. Now, we've opened a PowerShell window into iPerf on the client machine. Then we call the iPerf command. This time, we're going to tell it to launch in “Client Mode.” We need to specify an IP address for it to connect to.
And that's the IP address of the server machine?
Right. Again, the port; the format so that every output is exactly the same. And here, we want to update every second.
Okay.
And this is a super cool option: if we use the bytes flag, we can specify the size of the data payload. I’m going to go with a 1 Gb file (1024 Mb). You can also define parallel connections. I want 5 for now.
So, that's like 5 different users or parallel streams of activity of 1 Gb each between the server machine and this client machine?
Right. So, we actually want to measure how fast can we acquire data from the Qlik Sense server onto this client machine. We need to reverse the test. So, we can just run this now and see how fast it performs.
Okay. And did the server machine react the same way?
You can see that it produced output on the listening screen. This is where we started. And then it received and it's displaying its own statistics. And if you want to automate this, so that you have a spot check of throughput capacity between these servers, we need to use the log file option. And then we give it a path. So, I’m gonna say call this “iperf_serverside…” And launch it. And now, no output is produced.
Okay.
So, we can switch back to the client machine.
Okay. So, you're performing the exact same test again, just storing everything in a log file.
The test finished.
Okay. So, that can help you compare between what's being sent to what's being received, and see?
Absolutely. You can definitely have results presented in a way that is easy to compare across machines and across time. And initial results gave us a throughput per file of around 43.6, 46, thereabouts megabytes per second.
So, what about for an end user who's experiencing issues? Can you use iPerf to test the connectivity from a user machine on a different network?
Yep. So, in in the background we will have our server; it's running and waiting for connections. And let's run this connection now from the client machine. We will make sure that the IP address is correct; default port; the output format in megabytes; we want it refreshed every second; and we are transferring 1 Gb; and 5 parallel streams in reverse order. Meaning: we are copying from the server to the client machine. And let's run it.
Just seeing those numbers, they seem to be smaller than what we're seeing from the other machine.
Right. Indeed. I have some stuff in between to force it to talk a little slower. But this is one quick way to identify a spotty connection. This is where a baseline becomes gold; being able to demonstrate that your platform is experiencing a problem. And to quantify and to specify what that problem is going to reduce the time that you spend on outages and make you more effective as an admin.
Okay. That was network. How can admins monitor all the other performance aspects of a deployment? What tools are available and what metrics should they be measuring?
Right. That's a great question. The very basic is just Performance Monitor from Windows.
Okay.
The great thing about that is that we provide templates that also include metrics from our services.
Can you walk us through how to set up the Performance Monitor using one of those templates?Sure thing. So, we're going to switch over first to the central node. So, the first thing that I want to do is create a folder where all of these logs will be stored.
Okay. So, that's a shared folder, good.
And this article is a great place to start. So, we can just download this attachment
So, now it's time to set up a Performance Monitor proper. We need to set up a new Data Collector Set.
Giving it a name.
And create from template. Browse for it, and finish.
Okay. So it’s got the template. That's our new one Qlik Sense Node Monitor, right?
Yep. You'll have multiple servers all writing to the same location. The first thing is to define the name of each individual collector; and you do that here. And you can also provide subdirectory for these connectors, and I suggest to have one per node name. I will call this Central Node.
Everything that comes from this node, yeah.
Correct. You can also select a schedule for when to start these. We have an article on how to make sure that Data Collectors are started when Windows starts. And then a stop condition.
Now, setting up monitors like this; could this actually impact performance negatively?
There is always an overhead to collecting and saving these metrics to a file. But the overhead is negligible.
Okay.
I am happy with how this is defined. Now, this static collector on one of the nodes is already set up. There is an option here that's called Data Manager. What's important here to define is to set a Minimum Free Disk. We could go with 10 Gb, for example; and you can also define a Resource Policy. The important bit is Minimum Free Disk. We want to Delete the Oldest (not the largest) in the Data Collector itself. We should change that directory and make sure that it points to our central location instead of locally; and we'll have to do this for every single node where we set this up.
Okay. So, that's that shared location?
Yep.
And you run the Data Collector there. And it creates a CSV file with all those performance counters. Cool.
So, here we have it now. If we just take a very quick look inside, we'll see a whole bunch of metrics. And if you want to visualize these really really quick, I can show you a quick tip that wasn't on the agenda but since we're here: on Windows, there is a built-in tool called Relog that is specifically designed for reformatting Performance Monitor counters. So, we can use Relog; we'll give it the name of this file; the format will be Binary; the output will be the same, but we'll rename it to BLG; and let's run it.
And now it created a copy in Binary format. Cool thing about this Troy is that: you can just double click on it.
It's already formatted to be a little more readable. Wow! Check that out.
There we go. Another quick tip: since we're here, first thing to do is: select everything and Scale; just to make sure that you're not missing any of the metrics. And this is also a great way to illustrate which service counters and system counters we collect. As you can see, there's quite a few here.
Okay. So, that Performance Monitor is, it's set up; it's running; we can see how it looks; and that is going to run all the time or just when we manually trigger it?
You can definitely configure it to run all the time, and that would be my advice. Its value is really realized as a baseline.
Yeah. Exactly. That was pretty cool seeing how that worked, using all the built-in utilities. And that Relog formatting for the Process Monitor was new to me. Are there any other tools you like to highlight?
Yeah. So, Performance Monitor is built-in. For larger Enterprises that may already be monitoring resources in a centralized way, there's no reason why you shouldn't expect to include the Sense resources into that live monitoring. And this could be done via different solutions out there. A few come to mind like: Grafana, Datadog, Butler SOS, for example from one of our own Qlik luminaries.
Can we take a quick look at Grafana? I’ve heard of that but never seen it.
Sure thing. This is my host monitor sheet. It's nowhere built to a corporate standard, but you can see here I’m looking at resources for the physical host where these VMs are running as well as the domain controller, and the main server where we've been running our CPU tests. And the great part about this is I have historical data as far back I believe as 90 days.
So, this is a cool tool that lets you like take a look at the performance and zoom-in and find the processes that might be causing some peaks or anything you want to investigate?
Right. Exactly. At least come up with a with a narrow time frame for you to look into the other tools and again narrow down the window of your investigation.
Yeah, that could be really helpful. Now I wanted to move on to the Qlik Sense Scalability Tools. Are those available on Qlik community?
That's right. Let me show you where to find them. You can see that we support all current versions including some of the older ones. You will have to go through and download the package and the applications used for analysis afterwards. There is a link over here. So, once the package is downloaded, you will get an installer. And the other cool thing about Scalability Tools is that you can use it to pre-warm the cache on certain applications since Qlik Sense Enterprise doesn't support application pre-loading.
Oh, cool. So, you can throttle up applications into memory like in QlikView. Can we take a look at it?
Yes, absolutely. This is the first thing that you'll see. We'll have to create a new connection. So, I’ll open a simple one that I’ve defined here and we can take a look at what's required just to establish a quick connection to your Qlik Sense site.
Okay, but basically the scenario that you're setting up will simulate activity on a Qlik Sense site to test its performance?
Exactly. You'll need to define your server hostname. This can be any of your proxy nodes in the environment. The virtual proxy prefix. I’ve defined it as Header and authentication method is going to be WebSocket.
Okay.
And then, if we want to look at how virtual users are going to be injected into the system, scroll over here to the user section. Just for this simple test, I’ve set it up for User List where you can define a static list of users like so: User Directory and UserName.
Okay. So, it's going to be taking a look at those 2 users you already predefined and their activity?
Exactly. We need to test the connection to make sure that we can connect to the system. Connection Successful. And then we can proceed with the scenario. This is very simple but let me show you how I got this far. So, the very first thing that we should do is to Open an App.
So, you're dragging away items?
Yep. I’m removing actions from this list. Let's try to change the sheet. A very simple action. And now we have four sheets, and we'll go ahead and select one of them.
Okay, so far, we have Opening the App and immediately changing to a sheet?
Yep. That's right. This will trigger actions in sequence exactly how you define them. It will not take into consideration things like Think Time. I will just define a static weight of 15 seconds, and then you can make selections.
But this is an amazing tool for being able to kind of stress test your system.
It's very very useful and it also provides a huge amount of detail within the results that it produces. One other quick tip: while defining your scenario, use easy to read labels, so that you can identify these in the Results Application. Let's assume that the scenario is defined. We will go ahead and add one last action and that is: to close, to Disconnect the app. We'll call this “OpenApp.” We'll call this “SheetChange.” Make sure you Save. The connection we've tested; we've defined our list of users. First, let's run the scenario. There is one more step to define and that is: to configure an Executor that will use this scenario file to launch a workload against our system. Create a New Sequence.
This is just where all these settings you're defining here are saved?
Correct. This is simply a mapping between the execution job that you're defining and which script scenario should be used. We'll go ahead and grab that. Save it again; and now we can start it. And now in the background if we were to monitor the Qlik Sense environment, we would see some amount of load coming in. We see that we had some kind of issue here: empty ObjectID. Apparently I left something in the script editor; but yeah, you kind of get the idea.
So, all this performance information would then be loaded into an app that is part of the package downloaded from Qlik community. How does that look?
So, here you will see each individual result set, and you can look at multiple-exerciser runs in the single application. Unfortunately, we don't have more than one here to showcase that, but you would see multiple-colored lines. There is metrics for a little bit of everything: your session ramp, your throughput by minute, you can change these.
CPU, RAM. This is great.
Exactly. CPU and RAM. These are these are not connected. We don't have those logs, but you would have them for a setup run on your system. These come from Performance Monitor as well, so you could just use those logs provided that the right template is in place. We see Response Time Distribution by Action, and these are the ones that I’ve asked you to change and name so that they're easy to understand.
Once your deployment is large enough to need to be multi-node and the default settings are no longer the best ones for you, what needs to be adjusted with a Repository Service to keep it from choking or to improve its performance?
That's a great question Troy. So, the first thing that we should take a look at is how the Repository communicates with the backend Database and vice versa. The connection pool for the Repository is always based on core count on the machine. And the best rule of thumb that we have to date is to take your core count on that machine, multiply it by 5, and that will be the max connection pool for the Repository Service for that node.
Can you show us where that connection pool setting can be changed?
Yes. So, we will go ahead and take a look. Here we are on the central node of my environment. You'll have to find your Qlik installation folder. We'll navigate to the Repository folder, Util, QlikSenseUtil, and we'll have to launch this “As Admin.”
Okay.
We'll have to come to the Connection String Editor. Make sure that the path matches. We just have to click on Read so that we get the contents of these files. And the setting that we are about to change is this one.
Okay. So, the maximum number of connections that the Repository can make?
Yes. And this is (again) for each node going towards the Repository Database.
Okay.
Again, this should be a factor of CPU cores multiplied by 5. If 90 is higher than that result, leave 90 in place. Never decrease it.
Okay, that's a good tip.
Right. I change this to 120. I have to Save. What I like to do here is: clear the screen and hit Read again; just to make sure that the changes have been persisted in the file.
Okay.
Once that's done, we can close this. We can restart the environment. We can get out of here.
So, there you adjusted the setting of how many connections this node can make to the QSR. Then assuming we do the same on all nodes, where do we adjust the total number of connections the Repository itself can receive?
That should be a sum of all of the connection strings from all of your nodes plus 110 extra for the central node. By default, here is where you can find that config file: Repository, PostgreSQL, and we'll have to open this one, PostgreSQL. Towards the end of the file…
Just going all the way to the bottom.
Here we have my Max Connections is 300.
Okay. One other setting you mentioned was the Page File and something to be considered. How would we make changes or adjust that setting?
Right. So, this is a Windows level setting that's found in Advanced System Settings; Advanced tab; Performance; and then again Advanced; and here we have Virtual Memory.
Okay.
We have to hit Change. We'll have to leave it at System Managed or understand exactly which values we are choosing and why. If you're not sure, the default should always be System Managed.
Now, I want to know what resources are available for Qlik Sense admins; specifically, what is the Admin Playbook?
It's a great starting place for understanding what duties and responsibilities one should be thinking about when administering a Qlik Sense site.
So, these are a bunch of tools built by Qlik to help analyze your deployment in different ways. I see weekly, monthly, quarterly, yearly, and a lot of different things are available there.
Yeah. So, we can take a look at Task Analysis, for example. The first time you run it, it's going to take about 20 minutes; thereafter about 10. The benefits: it shows you really in depth how to get to the data and then how to tweak the system to work better based on what you have.
Yeah, that's great.
Right? So, not only we put the tools in your hands, but also how to build these tools as you can here. See here, we have instructions on how to come up with these objects from scratch. An absolute must-read for every system admin out there.
Mario, we've talked about optimizing the Qlik Sense Repository Service, but not about Postgres? Do larger Enterprise level deployments affect its performance?
Sure. The thing about Postgres is again: we have to configure it by default for compatibility and not performance. So, it's another component that has to be targeted for optimization.
The detail there that anything over 1 Gb from Postgres might get paged - that sounds like it could certainly impact performance.
Right, because the buffer setting that we have by default is set to 1 Gb; and that means only 1 Gb of physical memory will be allocated to Postgres work. Now, we're talking about the large environment 500 to maybe 5,000 apps. We're talking 1000s of users with about 1000 of them peak concurrency per hour.
So, can we increase that Shared Buffer setting?
Absolutely. And in fact, I want to direct you to a really good article on performance optimization for PostgreSQL. And when we talk about fine-tuning, this article is where I’d like to get started. We talk about certain important factors like the Shared Buffers. So, this is what we define to 1 Gb by default. Their recommendation is to start with 1/4 of physical memory in your system. 1 Gb is definitely not one quarter of the machines out there. So, it needs tweaking.
And again these are settings to be changed on the machine that's hosting the Repository Database, right?
That's correct. That's correct.
Now, is there an app that you're aware of that would be good to kind of look at all these logs and analyze what's going on with the performance?
Absolutely. This is an application that was developed to better understand all of the transactions happening in a particular environment. It reads the log files collected with the Log Collector either via the tool or the QMC itself.
Okay.
It's not built for active monitoring, but rather to enhance troubleshooting.
Sure. So, basically it's good for looking at a short period of time to help troubleshooting?
Right. The Repository itself communicates over APIs between all the nodes and keeps track of all of the activities in the system; and these translate to API calls. If we want to focus on Repository API calls, we can start by looking at transactions.
Okay.
So, this will give us detail about cost. For example, per REST call or API call, we can see which endpoints take the most, duration per user, and this gives you an opportunity to start at a very high level and slowly drill in both in message types and timeframe. Another sheet is the Threads Endpoints and Users; and here you have performance information about how many worker-threads the Repository Service is able to start, what is the Repository CPU consumption, so you can easily identify one. For example, here just by discount, we can see that the preview privileges call for objects is called…
Yeah, a lot.
Over half a million times, right? And represents 73% of the CPU compute cost.
Wow, nice insights.
And then if we look here at the bottom, we can start evaluating time-based patterns and select specific time frames and go into greater detail.
So, I’m assuming this can also show resource consumption as well?
Right. CPU, memory in gigabytes and memory in percent. One neat trick is: to go to the QMC, look at how you've defined your Working Set Limits, and then pre-define reference lines in this chart. So, that it's easier to visualize when those thresholds are close to being reached or breached. And you do that by the add-ons reference lines, and you can define them like this.
That's just to sort of set that to match what's in the QMC?
Exactly.
Makes a powerful visualization. So, you can really map it.
Absolutely. And you can always drill down into specific points in time we can go and check the log details Engine Focus sheet; and this will allow us to browse over time, select things like errors and warnings alone, and then we will have all of the messages that are coming from the log files and what their sources.
Yeah. That's great to have it all kind of collected here in one app, that's great.
Indeed.
To summarize things, we've talked about to understand system performance, a baseline needs to be established. That involves setting up some monitoring. There are lots of options and tools available to do that; and it's really about understanding how the system performs so the measurement and comparisons are possible if things don't perform as expected.
And to begin to optimize as well.
Okay, great. Well now, it's time for Q&A. Please submit your questions through the Q&A panel on the left side of your On24 console. Mario, which question would you like to address first?
We have some great questions already. So, let's see - first one is: how can we evaluate our existing Qlik Sense applications?
This is not something that I’ve covered today, but it's a great question. We have an application on community called App Metadata Analyzer. You can import this into your system and use it to understand the memory footprint of applications and objects within those applications and how they scale inside your system. It will very quickly illustrate if you are shipping applications with extremely large data files (for example) that are almost never used. You can use that as a baseline for both optimizing local applications and also in your efforts to migrating to SaaS, if you feel like you don't want to bother with all of this Performance Monitoring and optimization, you can always choose to use our services and we'll take care of that for you.
Okay, next question.
So, the next question: worker schedulers errors and engine performance. How to fix?
I think I would definitely point you back to this Log Analysis application. Load that time frame where you think something bad happened, and see what kind of insights you can you can get by playing with the data, by exploring the data. And then narrow that search down if you find a specific pattern that seems like the product is misbehaving. Talk to Qlik support. We'll evaluate that with you and determine whether this is a defect or not or if it's just a quirk of how your system is set up. But that Sense Log Analysis app is a great place to start. And going back to the sheet that I showed: Repository and Engine metrics are all collected there. And these come from the performance logs that we already produce from Qlik Sense. You don't need to load any additional performance counters to get those details.
Okay.
All right. So, there is a question here about Postgres 9.6 and the fact that it's soon coming end of life. And I think this is a great moment to talk about this. Qlik Sense client-managed or Qlik Sense Enterprise for Windows supports Postgres 12.5 for new installations since the May release. If you have an existing installation, 9.6 will continue to be used; but there is an article on community on how to in-place upgrade that to 12.5 as a standalone component. So, you don't have to continue using 9.5 if your IT policy is complaining about the fact that it's soon coming to the end of life. As we say, we are aware of this fact; and in fact, we are shipping a new version as of the May 2021 release.
Oh, great.
So, here's an interesting question. If we have Qlik Sense in Azure on a virtual machine, why is the performance so sluggish? How do you fine-tune it? I guess first we need to understand what would you mean by sluggish? But the first thing that I want to point to is: different instance types. So, virtual machines in virtual private cloud providers are optimized for different workloads. And the same is true for AWS, Azure and Google Cloud platform. You will have virtual machines that are optimized for storage; ones that are optimized for compute tasks or application analytics; some that are optimized for memory. Make sure that you've chosen the right instance type and the right level of provisioned iOps for this application. If you feel that your performance is sluggish, start increasing those resources. Go one tier up and reevaluate until you find a an instance type that works for you. If you wish to have these results (let's say beforehand), you will have to consider using the Scalability Tools together with some of your applications against different instance types in Azure to determine which ones work best.
Just to kind of follow up on that question, if we're looking at that multi-node example from Qlik help, what nodes would you consider would require more resources?
Worker nodes in general. And those would be front and back-end.
So, a worker node is something with an engine, right?
Exactly. Something with an engine. It can either be front-facing together with a proxy to serve content, or back-end together with a scheduler a service to perform reload tasks. These will consume all the resources available on a given machine.
Okay.
And this is how the Qlik Sense engine is developed to work. And these resources are almost never released unless there is a reason for it, because us keeping those results cached is what makes the product fast.
Okay.
Oh, here's a great one about avoiding working set breaches on engine nodes. Question says: do you have any tips for avoiding the max memory threshold from the QIX engine? We didn't really cover this this aspect, but as you know the engine allows you to configure memory limits both for the lower and higher memory limit. Understanding how these work; I want to point you back to that QIXs engine white paper. The system will perform certain actions when these thresholds are reached. The first prompt that I have for you in this situation is: understand if these limits are far away from your physical memory limit. By default, Qlik Sense (I believe) uses 70 / 90 as the low and high working sets on a machine. With a lot of RAM, let's say 256 - half a terabyte of RAM, if you leave that low working set limit to 70 percent, that means that by default 30 of your physical RAM will not be used by Qlik Sense. So. always keep in mind that these percentages are based on physical amount of RAM available on the machine, and as soon as you deploy large machines (large: I’m talking 128 Gb and up) you have to redefine these parameters. Raise them up so that you utilize almost all of the resources available on the machine ,and you should be able to visualize that very very easily in the Log Analysis App by going to Engine Load sheet and inserting those reference lines based on where your current working sets are. Of course, the only way really to avoid a working set limit issue is to make sure that you have enough resources. And the system is configured to utilize those resources, so even if you still get them after raising the limit and allowing the - allowing the product to use as much RAM as it can without of course interfering with Windows operations (which is why you should never set these to like 99, 98, 99). Windows needs RAM to operate by itself, and if we let Qlik Sense to take all of it, it will break things. If you've done that and you're still having performance issues, that means you need more resources.
Yeah. It makes sense.
Oh, so here is another interesting question about understanding what certain Qlik Repository Service (QRS) log messages say. There is a question here that says: try to meet the recommendation of network and persistence the network latency should be less than 4 MS, but consistently in our logs we are seeing the QRS security management retrieved privileges in so many milliseconds. Could this be a Repository Service issue or where would you suggest we investigate first? This is an info level message that you are reporting. And it's simply telling you how long it took for the Repository Service to compute the result for that request. That doesn't mean that this is how long it took to talk to the Database and back, or how long it took for the request to reach from client to the server; only how long it took for the Repository Service to look up the metadata look up the security rules and then return a result based on that. And I would say this coming back in 384 milliseconds is rather quick. It depends on how you've defined these security rules. If these security rules are super simple and you are still getting slow responses, we would definitely have to look at resource consumption. But if you want to know how these calls affect resource consumption on the Repository and Postgres side, go back to that Log Analysis App. Raise your Repository performance logs in the QMC to Debug levels so that you get all of the performance information of how long each call took to execute. And try to establish some patterns. See if you have calls that take longer to execute than others; and where are those coming from any specific apps, any specific users? All of these answers come from drilling down into the data via that app that I demoed.
Okay Mario, we have time for one last question.
Right. And I think this is an excellent one to end. We talked a whole bunch here about Qlik Sense, but all of this also applies to QlikView environments. We are always looking at taking a step back and considering all of the resources that are playing in the ecosystem, not just the product itself. And the question asks: is QlikView Server performance similar to how it handles resources Qlik Sense? The answer is: yes. The engine is exactly the same in both products. If you read that white paper, you will understand how it works in both QlikView and Qlik Sense. And the things that you should do to prepare for performance and optimization are exactly the same in both products. Excellent question.
Great. Well, thank you very much Mario!
Oh, it's been my pleasure Troy. That was it for me today. Thank you all for participating. Thank you all for showing up. Thank you Troy for helping me through this very very complicated topic. It's been a blast as always. And to our customers and partners, looking forward to seeing your questions and deeper dives into logs and performance on community.
Okay, great! Thank you everyone! We hope you enjoyed this session. Thank you to Mario for presenting. We appreciate getting experts like Mario to share with us. Here's our legal disclaimer and thank you once again. Have a great rest of your day.
-
Migrating to Qlik Cloud from Client-Managed Qlik Sense or QlikView
Qlik Cloud is a modern analytics and data platform built on the same software engine as QlikView and Qlik Sense Client-Managed and adds significant va... Show MoreQlik Cloud is a modern analytics and data platform built on the same software engine as QlikView and Qlik Sense Client-Managed and adds significant value to empower everyone in an organization to make better decisions daily. Qlik Cloud allows you to use one common platform for all users – executives, decision-makers, and analysts.
Migrating to Qlik Cloud can help your organization:
- Expand your analytics possibilities.
- Reduce infrastructure costs and the total cost of ownership of your Qlik solutions.
- Automate entitlement management and upgrades.
Migrating from Client-Managed Qlik Sense
This site provides you the tools to monitor, manage, and execute a migration from Client-Managed Qlik Sense to Qlik Cloud.
No two client-managed Qlik Sense Enterprise deployments are the same. And no two migrations will be the same. The processes, procedures, and instructions in this section shouldn’t be considered a cookbook. Rather, they’re meant to guide you.
The Qlik Cloud Migration Center provides a general approach to migration along with sequencing, strategy, and best practice recommendations. It also includes tools such as a Qlik Sense app, scripts, and worksheets to aid in planning elements of the migration.
If your organization has a complex deployment with custom tooling, or sophisticated or complicated data integration pipelines, consider contacting your Qlik Customer Support representative.Migrating from QlikView
This site provides comparisons of QlikView and Qlik Cloud, as well as best practices on how to move content, including information about migration assessments and QlikView document conversions.
Recent Documents
-
Nprinting On-Demand request retention time. How long are On-Demand reports kept ...
How many days are On-Demand request coming from Qlik Sense or QlikView kept in memory? Where are they saved? Is it possible to change this? Environmen... Show MoreHow many days are On-Demand request coming from Qlik Sense or QlikView kept in memory? Where are they saved?
Is it possible to change this?
Environment:The days NPrinting keeps an On-Demand request in memory depends on a parameter that can be added in the scheduler.config file.
Resolution:
The NPrinting On-Demand requests (both QlikView and Qlik Sense) are kept in memory in the "C:\ProgramData\NPrinting\ondemand\responses" folder for some days and then automatically removed:
- completed requests remain for 30 days
- aborted requests and failed requests are kept in memory for 6 days
It is possible to change these values by adding an entry in the "scheduler.config" file that is saved in "C:\Program Files\NPrintingServer\NPrinting\Scheduler":
- To change the days to keep for completed requests, add this tag and specify the number of days:
<add key="completed-days-to-keep" value="30" /> - To change the days to keep for aborted requests, add this tag and specify the number of days:
<add key="aborted-days-to-keep" value="6" />
!Note: NPrinting Installations, system folders and files are limited to the "c drive" where the installation is performed by default. This point remains documented in each Qlik NPrinting Product Release notes.
For details visit: Qlik NPrinting Known issues and limitations. File and folder redirection to different local or network locations is not currently available.
-
How to configure Qlik Sense to use a dedicated PostgreSQL database
Or: How to move Qlik Sense database to a standalone PostgreSQL server? The Qlik Sense Repository Database (QSR) can be moved to a dedicated standalone... Show MoreOr: How to move Qlik Sense database to a standalone PostgreSQL server?
The Qlik Sense Repository Database (QSR) can be moved to a dedicated standalone PostgreSQL instance not hosted on the same machine as other Qlik Sense Services. It is also possible to move an already existing QSR from a local Sense install, to a dedicated PostgresSQL database.
Environment
- Qlik Sense Enterprise on Windows , September 2017 and up
Set up a postgresSQL database
Instructions for this might vary slightly. Please refer to the Qlik Online Help for details. We recommend watching this video.
Qlik Sense Enterprise server has a set of five databases. For more details visit Installing and configuring PostgreSQL on help.qlik.com.
1. The Qlik Sense repository database (QSR):
The QSR is the primary database in your Qlik Sense deployment.
2. The Qlik Sense services database (SenseServices):
The SenseServices database contains schemas for each of the Qlik Sense services and allows growth independently of the Qlik Sense Repository Database, while still sharing the same PostgreSQL instance and login role.
3. The Qlik Sense message queue database (QSMQ):
The QSMQ database provides a light-weight method of passing messages internally between services in Qlik Sense Enterprise. The NOTIFY and LISTEN functionality in PostgreSQL allows services to be notified about new messages that have been written to the messaging table.
4. The Qlik Sense logging database (QLogs):
The QLogs database centralizes logging by collecting log messages from all Qlik Sense nodes in your deployment and stores them in a PostgreSQL database.
5. The Qlik Sense license service database (Licenses):
The Licenses database is used to sync the assignments between the Qlik license backend and the local installation.
Simplified checklist of steps:
- Download a valid version of postgresSQL. For Sense June 2017 and up, 9.6.x is required, from November 2019 version 11.5.x as well. From May 2021 version 12.5 can be used. Review the system requirements of your version before proceeding.
- Install based on instructions found on the Qlik Online Help for Qlik Sense; select the version you are running on then select from Menu , Deploy - Qlik Sense installation - Installing and configuring PostgreSQL.
Examples:Moving the Qlik Sense Share Folders as well?
Follow How to change the shared path in Qlik Sense
Backup
Depending on the version, you need to backup different databases, on the current Qlik Sense central node hosting the Qlik Sense Repository Database service, follow the backup instructions to obtain a database copy of each Qlik Sense Database Present .
Example for February 2021 you need to backup the databases reported in section "Installing and configuring PostgreSQL"; referring to the Help site these are :
QSR , QSMQ , SenseServices , Licenses , if configured QLogsRestore
On the dedicated remote postgresSQL host, perform a restore of the backed up Databases of previous point.
Changing the Connection String to QSR and QSMQ Database
- On the Qlik Sense central node, all services but the Qlik Sense Repository Service are shut down.
- Open QlikSenseUtil. By default, this will be installed in: C:\Program Files\Qlik\Sense\Repository\Util\QlikSenseUtil\QlikSenseUtil.exe
- Open the Connection String Editor and Read the current configuration, decrypting the current data.
- Replace all mentions of the current database hostname. In our example, the new database resides on 172.16.16.103. Instead of an IP you can also use the FQDN of the db server.
- You need to repeat this step for the sections <add name="QSMQ" and <add name="postgres"
- Hit Save Value in config file encrypted, committing any changes.
Changing the Connection String to SenseServices Databases.
Execute steps in Moved Postgress database to new host, but some qliksense is still accessing the SenseServices old database; check if different version of article are present for different Qlik Sense version .
Changing the Connection String to Licenses Database ( April 2019 and up )
Starting from April 2019 new License Service implementation is using this database.
- Open an elevated Powershell and change directory to C:\Program Files\Qlik\Sense\Licenses
- Run the script .\Configure-Service.ps1 and provide:
- new Hostname or IP
- database port default is 4432
- database user qliksenserepository
- databasepassword is the qliksenserepository password used during configuration of the new database.
- example:
- cd 'C:\Program Files\Qlik\Sense\Licenses'
- .\Configure-Service.ps1 172.16.16.102 4432 qliksenserepository Password123!
It will look like this:
Windows PowerShell
Copyright (C) 2014 Microsoft Corporation. All rights reserved.
PS C:\Windows\system32> cd \
PS C:\> cd 'C:\Program Files\Qlik\Sense\Licenses'
PS C:\Program Files\Qlik\Sense\Licenses> .\Configure-Service.ps1 172.16.16.102 4432 qliksenserepository Password123!
Licenses configuration started.
WARNING: Skipping the database initialization. No superuser or password specified.
Reading the settings file.
Saving the modified settings.
Licenses configuration successful.
PS C:\Program Files\Qlik\Sense\Licenses>
repeat this for:C:\Program Files\Qlik\Sense\NotifierService\install\Configure-Service.ps1
C:\Program Files\Qlik\Sense\AppDistributionService\Configure-Service.ps1
C:\Program Files\Qlik\Sense\HybridDeploymentService\Configure-Service.ps1
C:\Program Files\Qlik\Sense\MobilityRegistrarService\install\Configure-Service.ps1
C:\Program Files\Qlik\Sense\PrecedentsService\install\Configure-Service.ps1
C:\Program Files\Qlik\Sense\NLAppSearch\install\Configure-Service.ps1there might come new services or disappear with new releases, please check in the C:\Program Files\Qlik\Sense\ folder for "Configure-Service.ps1" to find additional one in your version of Qlik Sense.
Update the Qlik Logging Service Connection String
If the QLogs needs to be update you could check on help.qlik.com the section "Updating the connection string parameters" providing new hostname or ip.
Remove Database Service dependency and Postgres Server ( to avoid issues with future Upgrade of Environment)
Qlik Sense Repository service dependency on the Database service needs to be removed and Database needs to be removed.
In an elevated command prompt, execute the following on prompt:
c:\sc config "QlikSenseRepositoryService" depend= "" c:\sc config "QlikLoggingService" depend= ""
This should return:
[SC] ChangeServiceConfig SUCCESS
Restart the Qlik Sense Repository Service.
Check that Qlik Sense Repository Database doesn't start (confirms [SC] ChangeServiceConfig SUCCESS ).
Remove the (local) Qlik Sense Repository Database
Find the most recent PostgreSQL.msi in C:\ProgramData\Package Cache\ and remove it. This removes the service but no old local database data. This is needed to prevent issues on future upgrades of Environment.
You can find the precise folder with this command depending on the Postgresql Version you used installing the inital release.9.6:
c:\cd ProgramData\Package Cache c:\ProgramData\Package Cache\dir /s PostgreSQL.msi The folder containing PostgreSQL.msi will be revealed. Right click on the PostgreSQL.msi file and select uninstall from the menu.
12.5:
c:\cd ProgramData\Package Cache c:\ProgramData\Package Cache\dir /s PostgreSQL125.msi The folder containing PostgreSQL.msi will be revealed. Right click on the PostgreSQL.msi file and select uninstall from the menu.
Finalize
- Restart the Qlik Sense Repository Service and verify that Qlik Sense Repository Database Service doesn't exist when followed the previous steps.
- Start all remaining Sense Services
- Progress on the startup sequence (including the connection to the new database) can be tracked in the _System_Repository log files located in C:\ProgramData\Qlik\Sense\Log\Repository\Trace
- Verify functionality.
-
In Multi-node scenario you need to repeat steps "Changing the Connection Strings to Databases" on all Rim nodes (QlikSenseUtil for Repository.exe.config and Configure-Service.ps1 for the micro services), and if required configure "Update the Qlik Logging Service Connection String" (note that is EOL with Qlik Sense Enterprise on Windows February 2022) .
-
This article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
-
Upgrading Qlik Sense Repository Database from PostgreSQL 9.6 to 12.5 - New tool ...
This Support article covers functionality available in the new Qlik PostgreSQL Installer (QPI), which allows you to upgrade an existing Qlik Sense Re... Show MoreThis Support article covers functionality available in the new Qlik PostgreSQL Installer (QPI), which allows you to upgrade an existing Qlik Sense Repository Database (QRD) from PostgreSQL 9.6 to PostgreSQL 12.5, or install a brand new QRD based on PostgreSQL 12.5.
PostgreSQL 9.6 has reached End of Life Reference: PostgreSQL Versioning Policy
>> Qlik PostgreSQL Installer - Download Link <<
Environment
- New Qlik Sense Repository Database (QRD) setups for, compatible with Qlik Sense Enterprise Client Managed since November 2019 - Reference: Qlik Sense's compatibility with PostgreSQL for the... - Qlik Community - 1716510
- Existing QRD setups running Qlik Sense embedded PostgreSQL 9.6 (Qlik Sense February 2022 and higher)
Supported Scenarios
- Upgrading an existing QRD from PostgreSQL 9.6 to 12.5. Upgrading earlier versions of QRD is not supported via this method. Instead, upgrade to a version of Qlik Sense that supports 9.6 to automatically migrate through this minor version - List of supported PostgreSQL and early Qlik Sense versions
- Installing a new QRD cluster on a new server, for a new environment or to migrate an existing Database to a separate host.
- Qlik Sense Enterprise on Windows February 2022 and above are supported versions for QPI. Upgrade to a supported version for QPI first and then use QPI to upgrade PostgreSQL.
Considerations
- This installer will not detect a standalone PostgreSQL database installed by any other means, other than through an official Qlik installer package
- The installer itself does not provide a rollback feature, this is described below
- The user who runs the installer has sufficient permissions to execute it and install services
- The user who runs the installer has sufficient permissions to create folders and files in the backup destination
- The backup destination has sufficient free disk space to dump the existing database
- The backup destination is not a network path or virtual storage folder. It is recommended the backup is stored on the main drive
- There will be downtime during this operation, please plan accordingly
Installing a new Qlik Sense Repository Database using PostgreSQL 12.5
- All services on rim nodes must be stopped
- Run the Qlik PostgreSQL Installer
- Click on "Install"
- Accept the Qlik Customer Agreement
- Set your "Local database settings" and click "Next". You will use these details to connect other nodes to the same cluster
- Set your "Database superuser password" and click "Next"
- Review your settings and click "Install", then click "Finish"
- Restore your existing QRD backup OR
- Start installing Qlik Sense Enterprise Client Managed, connecting to an existing cluster
A new service will now be present in your Services management console:
If you are migrating your existing databases to a new host, please remember to reconfigure your nodes to connect to the correct host. How to configure Qlik Sense to use a dedicated Pos... - Qlik Community - 1791775
The tool will display information on the actions being performed. Once installation is finished, you can close the installer.
The PostgreSQL package can be removed via the Control Panel in case you need to uninstall the database at one point.
Upgrading an existing QRD to PostgreSQL 12.5 - Central Node
It is recommended that you first upgrade Qlik Sense to a version that supports QPI, starting with the February 2022 release. More information on Qlik Sense and PostgreSQL compatibility.
Installations other than in C:\Program Files\Qlik\Sense are not supported at the moment.
In this scenario, the new QRD will be using the same port as the old service, and rim nodes do not need any further reconfiguration.
- Stop all services on rim nodes
- On your Central Node, stop all services but Qlik Sense Repository Database
- Run the Qlik PostgreSQL Installer, an existing Database will be detected
- Click on "Upgrade"
- Accept the Qlik Customer Agreement
- Provide your existing "Database superuser password" and click "Next"
- Set your desired backup location and click "Next"
- Click "Upgrade", then exit by clicking "Finish"
- Start all other services on Central Node
- Start all services on all other nodes
Final steps
At this point, it is important to validate that all services and nodes are operating as expected. If it seems like some data is missing from the QMC or elsewhere, you may revert back to the old database instance. To do this, stop the newly installed "postgresql-x64-12" service and set its startup type to Manual. Then start the original "Qlik Sense Repository Database" service, along with all other services.
Once the upgrade and all validation is complete, the old Qlik Sense Repository Database service should be removed on the server.
Perform the following steps to delete the Qlik Sense Repository Database Service
c:\cd ProgramData\Package Cache
c:\ProgramData\Package Cache\dir /s PostgreSQL.msi
The folder containing PostgreSQL.msi will be revealed.
Right click on the PostgreSQL.msi file and select uninstall from the menu.This version is fully compatible with the official PostgreSQL installers from https://www.enterprisedb.com/downloads/postgres-postgresql-downloads
If you need to upgrade to a higher release than Qlik provides you can install the same major PostgreSQL release 12.x or up to version 13.
Troubleshooting
If the installation crashes, server reboots unexpectedly during this process or there is a power outage, the new database may not be in a serviceable state. Installation / upgrade logs are available in your temporary files location, for example:
C:\Users\Username\AppData\Local\Temp\2
A backup of the original database contents is available in your chosen location, or by default in:
C:\ProgramData\Qlik\Sense\Repository\PostgreSQL\backup\9.6
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support. The video in this article was recorded in a earlier version of QPI, some screens might differ a little bit.
>> Qlik PostgreSQL Installer - Download Link <<
Related Content
Techspert Talks - Upgrading Qlik Sense Repository Service
Backup and Restore Qlik Sense Enterprise documentation
Migrating Like a Boss
Optimizing Performance for Qlik Sense Enterprise
Qlik Sense Enterprise on Windows: How To Upgrade Standalone PostgreSQL
How-to reset forgotten PostgreSQL password in Qlik Sense
How to configure Qlik Sense to use a dedicated PostgreSQL database
Troubleshooting Qlik Sense Upgrades -
Access Issues and unusual behavior with SAML authentication after upgrade to Nov...
After upgrading to Qlik Sense Enterprise November 2021 or a later version, users accessing with SAML authentication may experience issues accessing Ql... Show MoreAfter upgrading to Qlik Sense Enterprise November 2021 or a later version, users accessing with SAML authentication may experience issues accessing Qlik Sense, lose their rights and permissions, or experience other unusual behavior.
However, performing a UDC sync and connecting with Windows Authentication works without issue.
Environment
Qlik Sense Enterprise on Windows November 2021 and later releases
Cause
Prior to the November 2021 version of Qlik Sense, the attributes from SAML authentication were not persisted in Qlik Sense and were only used and seen in the logs during session setup.
Starting with November 2021 a product enhancement was made to persist. This was done so that custom rules could be written to more easily take advantage of the SAML attributes.
The problem arises from the fact that many existing SAML virtual proxies map SAML attributes to existing attributes used by the UDC, such as the Group attribute.This means that when a user accesses Qlik Sense using SAML the Group attribute from the UDC will be overwritten with the SAML attribute. So if these two attributes are not identical any custom security rules written based on the attributes from UDC will no longer apply to the user trying to access Qlik Sense.
Resolution
There are several resolutions possible, but it is up to each customer to decide which is the best fit for them, primarily based on how much they are concerned with utilizing the attributes delivered by SAML.
1. Remove The Attribute Mapping From The Virtual Proxy
By simply removing the attribute mapping(s) from the virtual proxy settings you prevent Qlik Sense from overwriting the existing values pulled in by the UDC by not pulling in any values from the UDC.
2. Revert The Behavior And Disable Persistence
This is done by setting following flag to false in C:\Program Files\Qlik\Sense\CapabilityService\capabilities.json, followed by restart of services.
{"contentHash":"2ae4a99c9f17ab76e1eeb27bc4211874","originalClassName":"FeatureToggle","flag":"SessionDataPersistanceEnabled","enabled":false}
3. Modify Existing Custom Rules To Accept Both Formats
This option is fairly straightforward in concept, without modifying the virtual proxy or any configs this allows both formats to work if there is a discrepancy between what the UDC and SAML are providing.
For most organizations, this is the most convoluted and labor-intensive solution on this list, but it is a valid option and included in the interest of providing all the options possible.
4. Change The Naming Of The Attribute Mapping In The Virtual Proxy Settings
Instead of mapping to Group (to continue using the scenario from above), you can change the attribute name the values are mapped to, such as GroupSAML for example.
Since this now maps to a different attribute it will not overwrite the values brought in by UDC. This allows existing rules to work and can be used if you'd like to shift your custom rules to utilizing SAML attributes while maintaining a UDC synch to pull in usersOptions 1 & 2 prevent SAML from persisting data and are perfectly fine if you do not plan to use the SAML attributes to determine access and rights.
However, if you want to utilize the SAML attributes in your custom rules and use them to dictate access and rights then options 3 & 4 are viable. -
Qlik Replicate connection error to R4Z SqlState: 08001 NativeError: -30081 Messa...
Qlik Replicate connecting to DB2 z/OS fails with a Communication Link failure function detecting the error: "selectForRecvTimeout". Protocol specific... Show MoreQlik Replicate connecting to DB2 z/OS fails with a Communication Link failure function detecting the error:
"selectForRecvTimeout". Protocol specific error code(s): "115", "*", "*". SQLSTATE=08001 SQL30081N.
Environment
- Qlik Replicate all versions
Resolution
This error is an indication there is a connection issue from the Qlik Replicate Server to the z/OS environment as a general error. When a Task starts it can start up to six (6) threads connecting to the Source R4Z environment which are connections to the DDF DB2 environment.
On z/OS run the below system command:
-DIS DDF Detail and check the output below as these are the Max Connections and Active Connections
DSNL090I DT=A CONDBAT= 400 MDBAT= 200
DSNL092I ADBAT= 45 QUEDBAT= 203 INADBAT= 0 CONQUED= 0CONDBAT and MDBAT: These refer to two parameters, CONDBAT and MAXDBAT, in the Db2 subsystem's ZPARM module (think of the ZPARMs as Db2 for z/OS configuration parameters). CONDBAT specifies the maximum number of connections that DDF-using applications can have with the Db2 subsystem. MAXDBAT is the maximum number of those connections that can be active at one time
Cause
CONDBAT and MDBAT: These refer to two parameters, CONDBAT and MAXDBAT, in the Db2 subsystem's ZPARM module (think of the ZPARMs as Db2 for z/OS configuration parameters) where set to low for the concurrent Qlik Replicate Tasks running against this Db2 environment.
Related Content
https://www.ibm.com/docs/en/db2-for-zos/11?topic=commands-display-ddf-db2
-
An error occurred / Failed to load connection error message in Qlik Sense - Serv...
***Changes must be made to all Qlik servers that will not be provided with internet access.***This is mainly due to the Qlik DataMarket reaching out ... Show More
***Changes must be made to all Qlik servers that will not be provided with internet access.***
This is mainly due to the Qlik DataMarket reaching out to the internet although it may not even be used. Undoing or not agreeing to the Qlik DataMarket Terms of Service does not fully disable certain Qlik DataMarket functionalities.NOTE: Qlik DataMarket has been discontinued. Qlik Sense Enterprise on Windows versions February 2021 and newer no longer included it. For updated information see Qlik DataMarket will be retired
For servers not connected to the internet, they may be prompted with a pop-up error when browsing in the Hub or in the Data Load Editor with the following errors:
- The Hub has continuous bubbles while loading forever
- The Hub is stock on the initial loading bar
- Failed To Load Connection - Qlik Sense 3.0.x - Feb2018
- Failed To Load Connection - Qlik Sense 2.2
- An Unknown Error Occurred - Qlik Sense 2.1.1 and 2.0.x
Browser Debug tools may display the following error in the console:
Data Market: entry-point-building.js??123145:62
API could not be reached.
Cause:
The Qlik Sense DataMarket connector executable is cryptography signed for authenticity verification, and the .NET Framework's verification procedure when launching an executable includes checking OCSP and Certificate Revocation List information, which means fetching it online if the system doesn't have a fresh cached copy locally.
Qlik DataMarket connector will try to connect Internet to initialize and verify even without using it.
By making the changes in 3.x and June 2017 newer, Qlik Sense will not wait for a response on the verification step, which is the cause of the Failed To Load Connection and related error messages. It will also fix monitoring app issues and problems where the Data Load Editor loads forever. Note: it is highly suggested that all Qlik servers have internet access, however this will help to avoid some issues caused by it.
Resolution:
Option 1 will persist through upgrades, whereas Option 2 would have to be reapplied after every Sense upgrade.
Option 1:
1. Stop all Qlik Sense services on all nodes
2. Edit C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config
3a. If there is an empty <runtime/> tag, change it to<runtime> <generatePublisherEvidence enabled="false"/> </runtime>3b. If there is no <runtime/> tag, add the below anywhere above the closing </configuration> tag.<runtime> <generatePublisherEvidence enabled="false"/> </runtime>
3.c If the <runtime> section has existing content, add generatePublisherEvidence.<runtime> <some other key="value"/> <generatePublisherEvidence enabled="false"/> </runtime>
Example:
4. Save machine.config
5. Repeat on all nodes (if applicable)
6. Start services on all nodesOption 2:
1. Stop all services on all nodes
2. Open C:\Program Files\Common Files\Qlik\Custom Data\QvRestConnector\QvRestConnector.exe.config in an admin level Notepad window
3a. If there is an empty <runtime/> tag, change it to<runtime> <generatePublisherEvidence enabled="false"/> </runtime>3b. If there is no <runtime/> tag, add the below anywhere above the closing </configuration> tag.<runtime> <generatePublisherEvidence enabled="false"/> </runtime>
3c. If the <runtime> section has existing content, add generatePublisherEvidence.<runtime> <some other key="value"/> <generatePublisherEvidence enabled="false"/> </runtime>
4. Repeat on all nodes and repeat for all .config files- C:\Program Files\Common Files\Qlik\Custom Data\QvOdbcConnectorPackage\QvOdbcConnectorPackage.exe.config
- C:\Program Files\Common Files\Qlik\Custom Data\QvDataMarketConnector\QvDataMarketConnector.exe.config
- C:\Program Files\Common Files\Qlik\Custom Data\QvSalesforceConnector\QvSalesforceConnector.exe.config
- C:\Program Files\Common Files\Qlik\Custom Data\QvEssbaseConnector
- C:\Program Files\Common Files\Qlik\Custom Data\QvRestConnector
- C:\Program Files\Common Files\Qlik\Custom Data\QvWebStorageProviderConnectorPackage
5. Start all services on all nodes
6. Validate
It may be necessary to include the key immediately before the runtime closing tag if there are many values in the runtime section.
For Qlik Versions 2.x.x [mid-2016 and older]... this will fully disable Qlik Sense DataMarket:
Remove OR rename the file on all nodes
C:\Program Files\Common Files\Qlik\Custom Data\QvDataMarketConnector
NOTE: This folder is likely to be re-created upon upgrade.
-
Error: Connector connect error: Error starting QvConnect process: Instance 'QvOd...
Qlik Sense connector (from the ODBC connector package like for example Oracle, MS SQL, Hive, Redshift, etc...) fails randomly with the following error... Show MoreQlik Sense connector (from the ODBC connector package like for example Oracle, MS SQL, Hive, Redshift, etc...) fails randomly with the following error message which can be found in the script and the Qlik ODBC connector package log:
Error: Connector connect error: Error starting QvConnect process: Instance 'QvOdbcConnectorPackage#1' does not exist in the specified Category.
Environment
- Qlik ODBC Connector Package May 2021 or lower
- Qlik Sense Enterprise on Windows May 2021 or lower
Resolution
The issue is related to a software defect and has been solved in the Qlik Sense version August 2021.
Upgrade your Qlik Sense version to any version after Qlik Sense August 2021. The ODBC connector package will be automatically upgraded as well.
Workaround:
Besides doing an upgrade to Qlik Sense August 2021, there are the following workarounds available:
- Check if the affected connector has a connector reading strategy "engine" if yes, then try to change it to "connector", like for example " <add key="redshift-reading-strategy" value="connector" />" (this can be done in the config file " QvOdbcConnectorPackage.exe.config" under "C:\Program Files\Common Files\Qlik\Custom Data\QvOdbcConnectorPackage"
- Try to add "<add key="QlikEngineLocation" value="C:\Program Files\Qlik\Sense\Engine\QvConnect64.exe" />" in the ODBC connector package config file
Fix Version:
Qlik Sense August 2021
Cause
Product Defect ID: QB-3803
-
Qlik Sense SaaS - Azure AD groups limitation per user
In larger organizations, the number of groups a user is a member of may exceed the limit that Azure Active Directory will add to a token. 150 groups f... Show MoreIn larger organizations, the number of groups a user is a member of may exceed the limit that Azure Active Directory will add to a token. 150 groups for a SAML token, and 200 for a JWT.
Environment
Solution
Qlik Cloud does not read the groups into the ID token returned by Azure, but instead will fetch the groups for the user from the MS Graph API (a maximum of 1000 groups is fetched).
The groups are fetched from the following endpoint and nested groups are not supported.
https://docs.microsoft.com/en-us/graph/api/user-list-memberof
However, please note that if sending only specific groups to Qlik Sense is needed (and not all groups assigned to the user in Azure), then you will have to use the IdP type "ADFS" to read attributes from the ID token and use roles in Azure instead of groups to pass only the wanted groups.
https://joonasw.net/view/using-groups-vs-using-app-roles-in-azure-ad-apps
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
Related Content
https://docs.microsoft.com/en-us/azure/active-directory/hybrid/how-to-connect-fed-group-claims
https://joonasw.net/view/using-groups-vs-using-app-roles-in-azure-ad-apps
-
Qlik SAAS download chart images or data fails when accessing through VPN
When accessing the Qlik Cloud tenant through VPN, all vpn users are unable to download Qlik Cloud chart images and data with errors: Something went w... Show MoreWhen accessing the Qlik Cloud tenant through VPN, all vpn users are unable to download Qlik Cloud chart images and data with errors:
- Something went wrong or
- something went wrong with
Environment
Resolution
- Check with your cloud security provider and ask them to insert your Qlik Cloud tenant address into its access whitelist for your environment
Cause
- Cloud related security platform such as zscalar.net blocks download of Qlik chart images and data
Related Content
Internal Investigation ID
- QB-14441
-
Disable RC4 stream cipher in QlikView May 2022 (12.70) SR1 or higher
Is it possible to disable the non-secure RC4 cipher in QlikView? Environment QlikView 12.70 (May 2022) SR1 Resolution QlikView 12.70 SR1 introd... Show MoreIs it possible to disable the non-secure RC4 cipher in QlikView?
Environment
- QlikView 12.70 (May 2022) SR1
Resolution
QlikView 12.70 SR1 introduces the possibility of disabling RC4.
This will require applying a change to the QlikView Management Service and QlikView WebServer’s config settings and the QlikView Server's settings.ini.
This will also require applying a change to every client that needs to use the Open In Server (QlikView Desktop) functionality and for every client that uses the IE-Plugin on Microsoft Edge (Internet Explorer is no longer supported).
Note: Clients using only AJAX (browser) or only opening local files (QlikView Desktop) will not require a setting change.
LIMITATION: disabling RC4 is currently not supported if QlikView Desktop or the IE-Plugin are running on Windows 2022.
Server-Side changes:
- On all nodes, stop the QlikView WebServer (or Setting Service) and QlikView Management Service
- Take a backup copy of the QVManagementService.exe.config (under C:\Program Files\QlikView\Management Service) and the QVWebServer.exe.config (under C:\Program Files\QlikView\Server\Web Server) config files
- Using a text editor with administrator permissions, edit the two above-mentioned config files
- Change ”true” in the line <add key="EnableRC4" value="true" /> --> to ”false”
- Save the files and close
- Restart the services
Further server-side changes (only needed for "Open in Server" and "IE-plugin"):
- On all nodes, stop the QlikView Server (QVS.exe) service
- Find the servers’ settings.ini files and take a backup copy
- Edit the files
- Change the line SessionAlgorithm=Legacy to SessionAlgorithm=CALG_AES_128
- Save the files and close
- Restart the QVS.exe service
Client-side changes:
(to be applied to each client using the IE-Plugin or the QlikView Desktop client's "Open in Server" functionality)
- For each client, find the settings.ini file and take a backup copy
- Edit the file
- Change the line SessionAlgorithm=Legacy to SessionAlgorithm=CALG_AES_128
- Save the file and close
- Relaunch the client or browser
IMPORTANT: when using the IE-plugin or "Open In Server" the server and the clients will need to run on the same SessionAlgorithm, either Legacy (RC4 enabled) or CALG_AES_128 (RC4 disabled). Using different algorithms will break the connectivity between a client and a server. AJAX is not affected by this and does not require client-side changes. Changes are also not required if a Desktop client is only used to open local files (no connection to QVS).
-
Qlik Sense May 2022: failing to load a field type NCLOB or CLOB when connecting...
The following error or similar could be generated when loading data of type NCLOB or CLOB from an Oracle Data Source. ERROR [HY000] [Qlik][Oracl... Show MoreThe following error or similar could be generated when loading data of type NCLOB or CLOB from an Oracle Data Source.
ERROR [HY000] [Qlik][OracleOCI] (3000) Oracle Caller Interface: ORA-00942:the table or view does not existThe issue was found after upgrading from a previous Qlik Sense version (example QS February 2021 SR4; QS May 2021 SR2) to QS May 2022 SR1 and SR2
Environment
- Qlik Sense Enterprise May 2022
Resolution
- Defect QB-10789 was reported to R&D
- We noticed that only fields of data type NCLOB or CLOB are affected
- A patch with the fix is planned to be delivered with the Qlik Sense August 2022 Patch 1 version
Information provided on this defect is given as is at the time of documenting. For up to date information, please review the most recent Release Notes, or contact support with the ID QB-10789 for reference.
Workaround:
A workaround could be a downgrade of Qlik ODBC connector package to the previous working version. Alternatively you can reach out to Qlik Support in order to get a temporary fixed ODBC connector package version 6.191 (a new official patch will be provided soon).
Cause
Product Defect ID: QB-10789
-
Transformation: Source Lookup - CLOB datatype - limitations on string functions
Environment Replicate Table/Field level transformation The information in this article is provided as-is and to be used at own discretion. Dependin... Show MoreEnvironment
- Replicate
- Table/Field level transformation
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
Field Transformation
CLOB data types have a limitation in that no string functions can be performed on them in a field transformation. There will be no errors in the log file, just that the field will be NULL.
On my Source table from SQL, I am using the source lookup function under Data Enrichment in order to retrieve the CLOB field value and then apply a string function (replaceChars) to it.
By using the source lookup function on the filed it is returned to the replicate task in a form where a string function can be applied.
This source lookup function is documented in the replicate user guide and the on-line help.
The general form of the function is as follows:
source_lookup(TTL,'SCHM','TBL','EXP','COND',COND_PARAMS)
For this example I am doing the lookup into the MyClob table:source_lookup(1,'DBO','MyClob','MyClob','MyID=?',$MyID)
I will then wrap this lookup in a replacechars function:ReplaceChars(source_lookup(1,'DBO','MyClob','MyClob','MyID=?',$MyID),'L','@')
NO_CACHING is important to ensure it keeps changing for each value and doesn’t re-use values
DBOis the schema in oracle
MyCLOB is the table
MyClob is the field that I want returned
MyID=? is the predicate for the lookup
$MyID is the value from the field that we are using in the predicate in place of =?NOTE: That the lookup may have a performance implication and will need to be tested to see if it meets all of your latency criteria.
I will show a series of screen shots below that demonstrate the results of this exercise for the target table:
Source DDL and data for MyClob table:
Target data for MyClob table (NO Transformations):
Target data for MyClob table (ReplaceChars() function on MyString and MyClob fields):
NOTE: There are no errors in the task log - just null values in the field.
MyString field transformation: replaceChars($MyClob,'M','@')
MyCLOB field transformation: replaceChars($MyClob,'L','@')
Target data for MyClob table (ReplaceChars(SourceLookup) function on MyClob fields):
NOTE: You will see that the replaceChars function worked.
MyString field transformation: replaceChars($MyClob,'M','@')
MyCLOB field transformation: ReplaceChars(source_lookup(1,'DBO','MyClob','MyClob','MyID=?',$MyID),'L','@')
Please feel free to try other string functions on the Clob field after once the source lookup returns the value. They should all work.
-
How to troubleshoot Hadoop connectivity issues
In this article, we will be walking through the basic troubleshooting steps required to look into Hadoop endpoint connectivity issues. These steps wi... Show MoreIn this article, we will be walking through the basic troubleshooting steps required to look into Hadoop endpoint connectivity issues.
These steps will focus on investigating two connections:
- HDFS connectivity
- HIVE connectivity
The Scenario
We have implemented two issues by blocking port 50070 and adding the wrong ODBC parameter for the Hive. The outlined troubleshooting steps will highlight how to identify these issues.
In Replicate UI Go to Server Tab --> Logging --> Server Logging Levels and keep the COMMUNICATION and SERVER logging component to Trace during the troubleshooting.
As we begin, we can already see that the connection test fails when we click Test Connection. It fails with SYS-E-HTTPFAIL, Failed to connect to HDFS...
To view the Server logs go to Replicate UI --> Server Tab --> Logging --> Server Logging Levels --> View Logs --> Open the server log to see the below connectivity failed messages:
00011144: 2022-09-25T00:43:40:393877 [COMMUNICATION ]T: requested URL <http://10.51.2.44:50070/webhdfs/v1/?user.name=admin&op=GETFILESTATUS>, action is GET (at_curl_http_client.c:504)
00011144: 2022-09-25T00:43:40:393877 [COMMUNICATION ]T: Clearing (if there is) curl old custom request (at_curl_http_client.c:950)
00011144: 2022-09-25T00:43:40:393877 [COMMUNICATION ]T: calling user pre submit callback (at_curl_http_client.c:1960)
00011144: 2022-09-25T00:43:40:393877 [COMMUNICATION ]T: user pre submit callback returned with status 0 (at_curl_http_client.c:1964)
00011144: 2022-09-25T00:43:40:393877 [COMMUNICATION ]T: preparing response structure; body will be written as dynamic string (at_curl_http_client.c:1428)
00011144: 2022-09-25T00:43:40:395850 [COMMUNICATION ]T: AT_CURL->at_curl_easy_perform: curl_action_status == CURLE_OK failed (at_curl.c:883)
00011144: 2022-09-25T00:43:40:395850 [COMMUNICATION ]T: Failed to curl_easy_perform: curl status = 7, Couldn't connect to server (Failed to connect to 10.51.2.44 port 50070: Bad access) [1000143] (at_curl.c:883)
The error Couldn't connect to server in the logs and Failed to connect in the UI indicates a connectivity issue and we will begin investigating there.
1. Ping test.
This is a quick and common way to verify Hive and HDFS connectivity. It verifies if we have network connectivity at all.
In a command line, we execute:
ping 10.51.2.44
Where 10.51.2.44 is the IP address of the destination server.
Result: PASS
2. Telnet Test
Depending on whether or not we are using Hive or HDFS, the ports used are different. It verifies if the port specified is being listened on.
In a command line, we execute:
telnet 10.51.2.44 50070
Where 10.51.2.44 is the IP address of the destination server.
Result: FAIL
This indicates that port 50070 may not be available (blocked by a firewall, etc).
3. CURL Test
This test is only required for HDFS. We might need to use different authentication mechanisms with CURL based on the deployed authentication mechanisms (like Username\password, Kerberos with key tab, and Kerberos with AD Etc.,).
We must use the exact URL found in the Qlik Replicate log file.
In a command line, we execute:
curl -i -v "HTTP://10.51.2.44:50070/webhdfs/v1/user/admin/USER?user.name=admin&op?GETFILESTATUS"
Where 10.51.2.44 is the IP address of the destination server.
Result: FAIL
Since both the CURL and Telnet tests failed, we look into port 50070 and identify that it has been blocked by a firewall. We enable the port and further tests show Telnet and CURL succeeding.
We can also verify the port using PowerShell and the following command:
Test-NetConnection -ComputerNamer 10.51.2.44 -Port 50070
This is an example of a successful CURL test:
However, when we test the connection in the UI, the connection still fails.
This time it displays a different error:
Cannot enable SSL for the connection when connecting to a server that has not enabled...
To view the Server logs go to Replicate UI --> Server Tab --> Logging --> Server Logging Levels --> View Logs --> Open the server log to see the below connectivity failed messages:
00028952: 2022-09-25T01:02:09:511445 [COMMUNICATION ]T: requested url <http://10.51.2.44:5 0070/webhdfs/v1/?user.name=admin&op=GETFILESTATUS>, action is GET (at_curl_http_client.c:504)
00028952: 2022-09-25T01:02:09:511445 [COMMUNICATION ]T: Clearing (if there is) curl old custom request (at_curl_http_client.c:950)
00028952: 2022-09-25T01:02:09:511445 [COMMUNICATION ]T: calling user pre submit callback (at_curl_http_client.c:1960)
00028952: 2022-09-25T01:02:09:511445 [COMMUNICATION ]T: user pre submit callback returned with status 0 (at_curl_http_client.c:1964)
00028952: 2022-09-25T01:02:09:511445 [COMMUNICATION ]T: preparing response structure; body will be written as dynamic string (at_curl_http_client.c:1428)
00028952: 2022-09-25T01:02:09:912486 [COMMUNICATION ]T: http response code is 200 (at_curl_http_client.c:1186)
00028952: 2022-09-25T01:02:10:127510 [SERVER ]T: going to connect using connection string <ssl=1;HiveServerType=2;Driver={Cloudera ODBC Driver for Apache Hive};HOST=10.51.2.44;PORT=10000;Schema=hive_lz;AuthMech=3;UID=hive;PWD={****};> (hive_odbc_client.c:374)
00028952: 2022-09-25T01:02:10:546553 [INFRASTRUCTURE ]T: AR_ODBC_DRIVER_FUNC->ar_SQLDriverConnect returned -1 (ar_odbc_func.c:1117)
00028952: 2022-09-25T01:02:10:546553 [SERVER ]T: RetCode: SQL_ERROR SqlState: HY000 NativeError: 1160 Message: [Cloudera][DriverSupport] (1160) Cannot enable SSL for the connection when connecting to a server that has not enabled SSL. If the server has SSL enabled, please check if it has been configured to use a SSL protocol version that is lower than what is allowed for the connection. The minimum SSL protocol version allowed for the connection is TLS 1.2. [1022502] (ar_odbc_conn.c:584)
The logs confirm that the HDFS connection worked this time, but not the Hive connection failed.
We verify the Hive connection following the same method as before.
1. Ping test:
In a command line, we execute:
ping 10.51.2.44
Where 10.51.2.44 is the IP address of the destination server.
Result: PASS
2. Port Test
This time we use PowerShell, but you can use telnet as well.
We will use port 10000.
In PowerShell, we execute:
Test-NetConnection -ComputerNamer 10.51.2.44 -Port 10000
Where 10.51.2.44 is the IP address of the destination server.
Result: PASS
3. ODBC Test
We will test the ODBC connection from the Qlik Replicate server:
Result: Pass
4. Review the Hive connection string
Locate the Hive connection string in the Qlik Replicate log and compare it with the DSN setup.
ssl=1; HiveServerType=2;Driver={Cloudera ODBC Driver for Apache Hive};HOST=10.51.2.44;PORT=10000;Schema=swathi_lz;AuthMech=3;UID=hive;PWD={****};
The log indicates that we are using SSL.
But as we review the Hive DSN Setup, we can see that we have SSL unchecked:
Result: Everything matched other than ssl=1.
Let’s review and fix the endpoint settings in Qlik Replicate and perform the test connection by removing the internal parameter of ssl=1.
But what if we receive different error messages?
Generally, driver issues can lead to blockers for Hive connections. If you locate a different error in the UI or the log files, research these errors using either Google or our knowledge base here in the Community, as fixes may already exist.
Environment:
Qlik Replicate
Hadoop Target -
Qlik Replicate LOGSTREAM Timeout while waiting to get data from audit file
In some cases when Replicate can't read from the Staging Folder of a LogStream task for whatever reason (corrupted folder, lack of disk space etc), th... Show MoreIn some cases when Replicate can't read from the Staging Folder of a LogStream task for whatever reason (corrupted folder, lack of disk space etc), there could be difficulties to resume the LogStream task after the initial issue is solved.
You might see errors like the below when trying to resume from timestamp:
[UTILITIES ]E: Failed to write to audit file <audit_folder directory>
[UTILITIES ]E: Timeout while waiting to get data from audit file [1002521] (at_audit_file.c:637)[UTILITIES ]E: Error reading audit batch [1002509] (at_audit_file.c:679)
Environment
- Qlik Replicate 2021.11 (not exclusive)
- Linux RHEL / Windows
Resolution
- Stop all LogStream and Replication tasks
- Kill the replicate sessions and process as the audit file is being locked by the process
- Rename the audit folder of the problematic audit file (the folder is the one described under the endpoint settings: "Staging Folder" in the Replicate UI)
- Resume LogStream task from timestamp (a few hours before the initial error), then resume replication tasks from the same timestamp
- If not solved, a reload of the task will be needed
Cause
Usually, the Replicate process (repctl) is locking the audit file that was being written/read while the issue occurred.
-
Qlik NPrinting: Days to Keep / Reports to Keep Setting Ignored
While running Qlik NPrinting Tasks, many reports are generated. However, the setting for Days to Keep / Reports to Keep seems to be ignored. Environ... Show MoreWhile running Qlik NPrinting Tasks, many reports are generated. However, the setting for Days to Keep / Reports to Keep seems to be ignored.
Environment:
Qlik NPrinting all versions
The first thing to consider is that Qlik NPrinting generates one report for each Newsstand user. For this reason, there will be one file for each user, and not "just one", in the Qlik NPrinting Application Data folder. Those reports may be absolutely identical.
In that sense, the "report to keep" setting does not apply here, and should be considered as "report to keep per user". There is currently an improvement at R&D. Check improvement on days to keep in Nprinting for details. In addition, see Publish task "Reports to Keep" does not work during first 24 hours.
As for the setting apparently being ignored, by default runs the check to delete old reports once every 24 hours. If no new report has been run in the meantime, the old reports will not be deleted, even if more than one day has passed.
Should there be a feeling that the periodic check is not run, the settings can be enforced:- Stop the Qlik NPrinting services Proper Order to Restart Qlik NPrinting Services
- Navigate to and open C:\Program Files\NPrintingServer\NPrinting\Scheduler\scheduler.config
- Add the below lines before the </appSettings> closing tag
<add key="cleanup-period" value="1800" /> <add key="cleanup-period-files" value="43200" /> - Restart the services
The first setting is the recurrence in seconds (default: 30 minutes) for the task that will check for reports in the newsstand and unlink the older ones.
The second setting is the recurrence in seconds (default: 24 hours) for the task that will scan and remove the actual files, after checking that the conditions about days and reports to keep, plus the presence of newer reports that supersede them, are met. In this case, we would change the default value to a shorter time (43200 seconds, 12 hours) to make sure the task is run more frequently. -
QlikSenseUtil as a backup tool
Qlik Sense Util comes bundled with Qlik Sense Enterprise on Windows. It is able to assist in the backing up process to help with the manual process. K... Show MoreQlik Sense Util comes bundled with Qlik Sense Enterprise on Windows. It is able to assist in the backing up process to help with the manual process.
Key elements that need to be backed up are:
- the certificates used by Qlik Sense for internal communication
- the Repository Database used to store meta-data about the Qlik Sense site
- the applications used by the Qlik Sense site
- the Extensions, thumbnail images, etc used by Qlik Sense apps
Other options components can include:
- Logs of activity on the site
- Supplemental material used by the Qlik Sense site (e.g. QVDs, Excel files, etc)
By using the QlikSenseUtil tool, an administrator can back all of these up and configure the tool to be integrated in a silent, incremental backup policy.
How to get started with QlikSenseUtil
- Locate the QlikSenseUtil tool
- Default location: C:\Program Files\Qlik\Sense\Repository\Util\QlikSenseUtil\QlikSenseUtil.exe
- Custom path example: Drive:\InstallationPath\Repository\Util\QlikSenseUtil\QlikSenseUtil.exe
- In Notepad or any text editor of your choice, construct the desired command line syntax for the backup.
- A sample command:
C:\"Program Files"\Qlik\Sense\Repository\Util\QlikSenseUtil\QlikSenseUtil.exe -backup -databaseHostname="FQDNofTheDatabase" -databasePassword="SuperUserPassword" -path="Drive:\BackupLocation" -rootPath="\\ServiceCluster\Path" - For a full listing of the flags possible, consult later in this article or the Readme.txt in the QlikSenseUtil directory.
Note: Executing a backup will stop Qlik Sense services, so only execute this tool when downtime is acceptable
Note: Executing as the Qlik Sense service account is recommended since the client certificate used by Qlik Sense is only accessible to the service account
Note: For silent backups, do remember to use the -f flag
Note: Logs for this tool will be written to %USERPROFILE%\AppData\Local\Temp\logs for the account executing the tool
- A sample command:
- Once the command is written out, launch a command prompt and paste in the command
- Observe the files backed up to the path specified in the path flag
Video Walk-through:
(video from https://www.youtube.com/watch?v=uuF4FMnaD1I?)
Flags:
backup: specifies a backup operation (mandatory)
path: path to the folder where to store the backup (mandatory)
certificatePassword: password for protecting the certificates. If not specified then the certificates will not be backed up
databaseHostname: Hostname where the database is running (mandatory)
databasePassword: Superuser or postgres account password (mandatory, if password was configured)
databasePort: port number used by the database)
f: silent backup
h: helpTo restore your environment, see Backup and restore Qlik Sense Enterprise on Windows.
Related Content:
-
How to allow Anonymous Hub access in Qlik Sense Enterprise on Windows
Three requirements need to be met in order to allow Anonymous users access to the Hub: The product must not be Qlik Sense Business or Qlik Sense Ente... Show MoreThree requirements need to be met in order to allow Anonymous users access to the Hub:
- The product must not be Qlik Sense Business or Qlik Sense Enterprise SaaS.
- The license must support anonymous access, see Which Qlik Sense Enterprise licenses support anonymous access?
- Proxy must be configured to allow Anonymous users
- A Stream dedicated to anonymous users must be created
Useful information:
- Users who access Qlik Sense anonymously are identified as NONE\anonymousd9e61933-a7e5-4836-8540-5b137a600e69, where the GUID is a unique identifier.
- To create a license rule, granting login passes to anonymous users you can use two identifiers:
- userDirectory=NONE
- user.IsAnonymous()
Configure the Proxy to allow Anonymous users
By default, anonymous access is disabled. The steps provided require a restart of the Proxy service after completion.
- Open the Qlik Sense Management Console.
- Choose the Virtual Proxy to be used for anonymous access
- Click Edit.
- In the Properties menu (right side) choose Authentication
- From the Anonymous access mode drop-down select Allow anonymous users
Set up a license rule
Anonymous users will need to have a license rule available for them which gives them either an Analyzer Capacity License or a Login Access Token. The instructions in this article will focus on Analyzter Capacity Licenses.
- Open the Qlik Sense Management Console.
- Navigate to License Management
- Click Analyzer capacity rules
- Click Create new
- Give the new rule a Name (mandatory) and Description (optional)
- Create one of two possible rules:
- In the Basic settings, select
user: userDirectory =
value: NONE - In the Advanced settings (see Fig 2), provide the condition:
user.IsAnonymous()Fig 2
- In the Basic settings, select
- Click Apply
Set up a Stream for anonymous access
You can use the default Everyone stream, which has already been set up for anonymous access, or create your own.
We will create two rules. One to allow logged in (authenticated) users access and publishing permissions. One to allow anonymous user access, but no publishing permissons. Note that based on your requirements, this may need to modified.
- Go to the Qlik Sense Management Console
- Go to Manage Content and Streams
- Click Create New
- Name your new Stream
- Click Security Rules
- Click Create associated rule (see Fig 2)
This will be our "Everyone (anonymous) can read the stream." rule.Fig 3
- Provide a Name (mandator) and Description (optional)
- In Basic only select Read
- In Advanced define
Condition: user.IsAnonymous()
Context: Only in hub
See Fig 4.Fig 4
- Click Apply
- Click Create associated rule (see Fig 2)
This will be our "Everyone (logged in) can access this stream and publish." rule. - Provide a Name (mandator) and Description (optional)
- In Basic only select Read and Publish.
- In Advanced define
Condition: !user.IsAnonymous()
Context: Both in hub and QMC - Click Apply
Test!
We are now ready to test access.
- Go to the Qlik Sense Hub (hosted by the Virtual Proxy you have configured)
- If you are logged in, log out.
- Our new Anonymous stream is now avaiable.
-
Using Qlik Application Automation to create and distribute Excel reports in Offi...
Getting data out of Qlik Sense Enterprise SaaS, and distributing it to different users in formatted Excel, has been a manual task until now. Thanks to... Show MoreGetting data out of Qlik Sense Enterprise SaaS, and distributing it to different users in formatted Excel, has been a manual task until now.
Thanks to the release of Qlik Application Automation, it is now possible to automate this workflow by leveraging the connectors for Office 365 - specifically Microsoft SharePoint and Microsoft Excel.
Here I share two example QAA workspaces that you can use and modify to suit your requirements.
Video:
Considerations
- QAA cannot currently attach files using the Mail block, instead this approach distributes a SharePoint link.
- QAA has a limit of 100,000 rows when getting data out of a Qlik Sense straight table object.
- The On-Demand example uses an extension in QSE SaaS to send data to the Automation. An update to the Qlik Sense Button object is expected soon, which will provide a native way to pass selections to an Automation.
Example 1 - Scheduled Reports
- Download the 'Scheduled Report.json' file attached to this document.
- Create a new Automation in QSE SaaS, give it a name, and then upload the workspace you just downloaded by right clicking in the editor canvas, and selecting 'Upload workspace'.
- Select the 'Create Binary File (Personal One Drive)' block, select 'Connection' in the block configurator to the right, and then create your connection to Microsoft SharePoint.
- Select the 'Get Straight Table Data' block. Under 'Inputs' in the block configurator, lookup your the App Id, Sheet Id, and Object Id for the relevant QSE SaaS table you wish to output.
- Select the 'Create Excel Table With Headers' block, select 'Connection' in the block configurator, and then create your connection to Microsoft Excel.
- Select the 'Send Mail' block. Under 'Inputs' in the block configurator update the 'To' to reflect the addresses you wish to deliver to.
- With the 'Send Mail' block still selected, select 'Connection' in the block configurator and add your Sender details.
- To test, Save and then Run the Automation
- If you receive any warnings or errors, navigate to the relevant blocks and ensure your Connection is selected in the block configurator.
- Select the 'Start' block. Under 'Inputs' in the block configurator, change Run Mode to Scheduled and define your required schedule.
Example 2 - On-Demand Reports
Note - These instructions assume you have already created connections as required in Example 1.
- Download the 'On-Demand Report v3.json' file attached to this document.
- Download and install the 'qlik-blends' extension. See:
https://github.com/rileymd88/qlik-blends/files/6378232/qlik-blends.zip - Create a new Automation in QSE SaaS, give it a name, and then upload the workspace you just downloaded by right clicking in the editor canvas, and selecting 'Upload workspace'.
- Ensure your Connections are selected in the block configurator for each of the following blocks, 'Create Binary File (Personal One Drive)', 'Create Excel Table With Headers', 'Add Rows To Excel Worksheet Table (Batch)', 'Create Sharing Link', and 'Send Mail'. Save the Automation.
- Select the 'Start' block and ensure Run Mode is set to Triggered. Make note of the of URL and Execution Token shown in the POST example.
- Open your chosen QSE SaaS application, and Edit the Sheet where you wish to add a Button to trigger an On-Demand report.
- Under 'Custom Objects' look for 'qlik-blends' from the Extensions menu and drag this into your Sheet.
- Under the 'Blend' properties to the right, add-in your POST webhook URL and Token as noted in Step 5.
- We will now add three measures to the 'qlik-blends' object. It is important you add them in the order described. Add the first measure, using the following function in the expression editor: GetCurrentSelections()
- Add the second measure, using the following function DocumentName()
- The final measure will be the Object ID of the table you wish to use. To find the Object ID, select 'Done Editing'. Then right click on the table, select 'Share', select 'Embed', then look for the Object ID under the preview. Copy this value, go back into Editing mode and paste this as your third measure value.
- With the 'qlik-blends' object selected, under Form select 'Add items'. For 'Item type' select Text. Under default value you can choose to add a default email address. For 'Label' and 'Reference' type 'Email'. It is critical that Reference is updated to 'Email'. Turn 'Required input' on.
- You can change the Appearance properties to suit your preferences, such as updating the Button label and message, enabling Dialog, and changing the Color under Theme.
- Back in the Automation, under the Start Block. Set 'Run asynchronously' = yes to allow multiple requests to run at the same time (This will also increase the max run time from 1min to 60min)
- Once happy, test the On-Demand report by entering an email and clicking the button.
This On-Demand Report Automation can be used across multiple apps and tables. Simply copy the extension object between apps & sheets, and update the Object ID (Measure 3) for each instance.
Environment
- Qlik Application Automation
- Qlik Cloud
- Microsoft Office 365
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
-
License and operation monitoring apps error 401 Unauthorized
Monitoring App reload fails with:Error message: Error: QVX_UNEXPECTED_END_OF_DATA: HTTP protocol error 401 (Unauthorized): Requested resource requires... Show MoreMonitoring App reload fails with:
Error message: Error: QVX_UNEXPECTED_END_OF_DATA: HTTP protocol error 401 (Unauthorized): Requested resource requires authentication.
The error happens when the Qlik Sense services account does not match the account used in the reload task.Resolution:
The data connections must be updated accordingly with the Qlik Service User ID and Password.- Open the Qlik Sense Management Console
- Navigate to Data Connections
- Locate monitor_apps_REST_ data connection
- Click Edit
- Enter the correct User ID and Password (as seen in Fig 1)
- Save the changes.
Fig 1
-
How to: Getting started with the Splunk connector in Qlik Application Automation
This article gives an overview of the Splunk Connector in Qlik Application Automation.The Splunk connector has been created with the following use cas... Show MoreThis article gives an overview of the Splunk Connector in Qlik Application Automation.
The Splunk connector has been created with the following use cases in mind:- Being able to perform searches to a Splunk instance
- Inserting new events into Splunk using the HTTP Event Collector
Connecting the Splunk connector
The Splunk connector makes use of basic authentication. When connecting the connector, you will be prompted with the following popup screen:
Hostname: The hostname or IP address of the Splunk instance. Exclude the http(s):// part and the port number of the URL.
Username: The username you use to login to Splunk
Password: The password you use to login to Splunk
Use SSL: Select whether the management service of Splunk makes use of SSL. Default yes.
Management port: The port on which the management service of Splunk is running. Default is 8089. Providing no value will omit the port from the final URL.Performing a search on Splunk
To perform searches using the Splunk connector it is possible to perform a synchronous search as well as an asynchronous search. The synchronous search has a timeout of 50 seconds, with the asynchronous search it is possible to have a longer window before the automation will timeout.
Synchronous Search
The synchronous search can be done using the Synchronous Search block. In the Search Query block you will have to provide a splunk search query.
Asynchronous Search
The asynchronous search block can be manually done using the block Asynchronous Search block. Then perform polling using the Get Search block using the returned Search ID. When finished, results can be obtained using the Get Results From Search block.
Alternatively it is possible to use the block Do Asynchronous Search and get Results. This block will perform the polling, only a search query has to be provided to obtain a list of results.
Inserting data into Splunk
The other use of the Splunk connector is to insert data into Splunk by making use of the HTTP Event Collector. An example usecase for this would be to send reload logs of a Qlik Sense app into Splunk.
To do this we will first have to create an HTTP Event Collector on the Splunk side.To do this in Splunk go to Splunk in the web browser and go to Settings > Data > Data Inputs.
On this page you will see Http Event Collector. Press the Add new button.Provide a name for the token and go to the next page. Select indexes to which this token can write. Click on Review and Submit. You will be given a token that you need to configure on the block.
In Qlik Application Automation make use of the Insert Data block. For the inputs, enter the HEC token value. Provide the port for the HTTP event collector. This can be obtained from the Splunk HTTP Event Collector page when clicking on the Global Settings button. You can also find whether SSL is enabled for the HTTP Event Collector, select this value for the Hec_Use_Ssl parameter. Next provide the index to which you're writing your events and provide a source string.
For data, you can send a JSON message. Event is a mandatory key to write in this JSON message.The following example will perform a reload and send the reload log to Splunk:
In Splunk it is then possible to view the logs of the reload:
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.
