STT - Monitoring Qlik Cloud Capacity

[Troy_Raney]

Environment

• Qlik Cloud

Transcript

Hello everyone and welcome to the May edition of Techspert Talks. I'm Troy Raney and I'll be your host for today's session. Today's Techspert Talks session is Monitoring Qlik Cloud Capacity with our own Tyler Waterfall. Tyler, why don't you tell us a little bit about yourself?
Hey everybody. Yep. I'm Tyler Waterfall. I've been with Qlik for long time now, 13 years. I was a customer using QlikView at a retailer before and I loved it so much that I joined the company. Currently I've been working on enabling data Analytics internally with Qlik. So I love the topic of Analytics and using our products and also been in charge of getting Capacity Consumption Reporting Analytics available to customers and also to us internally.
Thanks. So today Tyler's going to be talking about how to understand Capacity usage. We're going to explore the Reporting app in detail and perhaps offer some tips about how to manage that consumption. Now Tyler, when we're talking about Capacity licensing, what exactly is counted in Capacity?
Sure. So Capacity is just a way for us to give customers a chance to use our different features and you have a certain Capacity in your subscription and there are various features: Reports generated; 3rd-party Automations run; data moved; and one that's pretty common is Data for Analysis which is just a measure of how much data is available to analyze on your Qlik Cloud deployment. For example, there's apps that load external data into your Qlik Cloud deployment, and we count the data that's loaded into the app. And then any data that you're storing on your Qlik Cloud Tenant and QVDs or other file formats, call those data files. It's a cumulative assessment of how much data is available for analysis for that calendar month. And so it's not necessarily the app size, it's the amount of data that's loaded in. And that carries over as long as the app is there and available for analysis. But later we can talk through some of the tips on managing that.
What is the main tool for how admins can monitor their data consumption on their Tenants?
The primary tool for looking at detailed usage is the Consumption Report app. The Tenant admin is able to distribute internally to your Qlik Cloud Tenant.
So that's this Consumption Report? Okay.
So this is the app that gets generated.
And how do admins create this app or where do we get it from?
Sure. So in the Qlik Cloud Administration, if you come down to the Settings down here at the bottom and scroll down a little bit, there's a section called Detailed Consumption Reports, and you can click the Distribute button and an Automation in the background will fetch the Consumption Report for your Tenant and place it in your Personal Space. And just a little note: we have a process to update it daily. And so when you request this, it's just grabbing that latest updated Consumption Report and putting it on your Tenant. And so it's not an app that you can just click reload. You need to request an updated version each day.
Right. So it's not an app that you can schedule a reload to update the data. It's a unique app and each time you need to generate a brand new one?
Yeah. And we're aware that the manual process isn't the most fun. So for certain deployments you can use an Automation actually to distribute this app on a daily basis to a Managed Space. If you are interested in that approach if you come in and create a new Automation using a template. There's one called Capacity Consumption App Deployer. Here it is. It will set it up on a schedule. It will distribute the app each day to a Managed Space. That's pretty slick.
Okay. You can specify how often you want it - an updated version, but this will generate. And could it possibly even like replace the old one?
Yeah, it will publish and replace to that Managed Space.
Okay.
That's kind of nice.
Very cool. Can we take a look at the app?
Sure. Yeah, this is the overview page. Just shows me here's the different features that are available. It doesn't mean that I have all of them in my subscription.
Okay, so this overview sheet gives you a high-level summary of the features that this Tenant is using and the amount of Capacity from a monthly view of the past 12 months. What are these different sections?
Sure. Yep. I'll just talk about them here on this page. So Data for Analysis (like we talked about) that's how much data have you loaded externally into your Qlik Cloud Tenant via apps or in data files that's available.
Okay.
Data Moved and 3rd - Party Transformations. These are Qlik features to move data. And so that's cumulative GBs that you've been moving.
So that's on the Qlik Talend Cloud Data Integration side?
Yep.
Okay.
Yep. And then Open Lakehouse compute is another feature for moving data. A little bit newer. 3rd -party Automation runs. So you can run the standard Automations. Those aren't counted towards your Capacity, but other third party Automations like sending Slack messages or doing other things, those are counted here.
So I understand, it's not every Automation. It's just Automations that use some sort of third-party block. Is that right?
Correct. Yep.
All right.
And then there's also Reporting. So you can generate Reports and distribute Reports. Job Executions, Job Duration, that's related to the Talend Data Integration.
Okay.
And then Qlik Answers. So that's using MCP and asking questions with the Answers assistant.
Right. That's the popular AI feature.
Yep. And we're just getting familiar with it.
Okay.
Each of these will go into more detail with different sheets.
Let's jump into the first sheet there. Before we go into any more detail, my eye is drawn to that first chart on the left that's saying we're up to 96% usage of our Capacity on this Tenant. What does that mean?
Sure. On this Tenant, my subscription entitles me to 250 GB of Data for Analysis.
Okay.
And within the current month, I have reached a high water mark of 241 GB, 96%.
Okay. So, how is that calculated?
On the right here, I have a daily bar chart to show the Data for Analysis. This doesn't fluctuate all too much. Just kind of goes up and down. And within each month, here's the Peak Day. We look at the date that had the highest amount of Data for Analysis available.
Okay. So the Peak Day of each month that kind of determines what the cost is?
Correct. So if we go back to April, April 17th was the Peak Day. Hover over this bar and see the amount of data available within apps and data files (QVDs) was 230.5. So that becomes what we assess for the usage for the month of April.
So how can you see exactly what's included for that Peak Day?
Yeah, let's take a look at the Peak Day for this month. We're up a little more than last month. I can select this day; and down below I have some more details. So if I go Details by Day, it will show me for each day what apps and data files are contributing to this 241 GB for this Peak Day.
All right.
And I can look at this date and it's sorted for me by data volume.
The top item there is 10 GBs of data.
Correct. And that's being contributed by this app. The way that we calculated the contribution was the app reload. So during the app reload, it fetched 10 GB of external data in that reload.
Okay. So I just want to clarify, that's not the RAM footprint of the app in memory. It's how much data was consumed by that app. Am I understanding that right?
Correct. And maybe this app loads 10 GB of data from Snowflake and stores it into QVDs and then drops the data. And so if you look at the app size, it's going to be a very small app, but it loaded 10 GBs of data from an external source.
Okay. What's that fourth item down, data file?
Yep. So the data files, those are like QVD for example. So it's very possible that this app fetched 10 GB of external data and stores it into this QVD. And so this QVD is now available for analysis across other apps. Say for example, if I look at the No, so ones that don't count, I might have an app here and it's just loading that QVD. The app doesn't get counted because it's loading data that already exists on your Tenant.
So, I just want to make sure I understand, that little flag you checked there on (the No), those are apps that are already reloaded once and so that data is already counted. Is that right?
Well, there's two possibilities. It's an app that exists in a Personal Space. So, we don't count those ones. So as you are exploring new data or I want to see what it's like to load this group of tables, I can do that in my Personal Space and it's not going to count here. That's one scenario where it's not going to count towards Capacity. The other scenario is: this app just loads QVDs.
Right. Because that QVD data is already counted. So just reading from a flat file that's on the Tenant doesn't count any additional data against the Capacity. Okay.
Correct. And like for this example, the App Import, this app was probably just imported into your Personal Space…
and again any data loaded in Personal Space doesn't get counted against Capacity.
Correct.
All right.
Correct. So as soon as you want to share and collaborate, you put it in a Managed Space or a Shared Space, that's when we'll count it. But as you're exploring, the Personal Space can be a great place to explore
That’s nice, especially if like you're just checking to see how much loading that data might count against Capacity. It's good to use it in Personal Space to do that testing and exploration.
Yep.
So, how else would you recommend admins use this app to review their consumption?
Sure. Here's an app down here. It's 4 GB. Not terribly big, but it's being counted based on the last reload, which was the 27th of October. That's a stale app. If I were looking to do some housekeeping, I can go back and look at some of these apps that maybe are stale and maybe can be archived or put in a Personal Space or deleted because they're still counting even though they're not getting updated.
That's a good tip. So, keep an eye out for apps that aren't being reloaded, aren't being used, and if it's just sort of stale, at least reach out and ask the question, could this be moved?
And actually, I'm curious now on this app. So the biggest app that I have. So if I select the app and then I remove the date selection. So that app was first reloaded on the 13th of May.
Yeah. What do you think was happening here since it started as 10 GB, but now it's almost nothing?
Yeah, it was reloaded on this day, and in the day you can see there's two different reload IDs. The last time that it was reloaded, there was no external data that was fetched. That reset the app going forward. So this is a common scenario: people fetch data from that Snowflake example store it into QVDs and then I'm just going to load data from QVDs.
Right. So now it's not counting because it's not loading from external data anymore?
Correct.
Is there a best practice for avoiding situations like this in the future?
Yeah, maybe exploring apps and data within a Personal Space to get started.
Yeah.
So, if I'm not sure and I think I don't even know how much data is here, maybe just consider loading it in a Personal Space.
Do you have any tips for identifying those stale apps?
Sure. Yeah. If you look at the latest day, so 18th of May, this is what's currently counting. So, you'll notice that 10 GB app is no longer there because it got reset. Well, there's a lot of stuff. I'll make sure I'm just looking at the ones that count. But still, if you see the little scroll bar, I have a lot of apps and data files, but I could look at the biggest ones. This will give you a sense of things.
Well, I do want to ask how are Answers measured against Capacity?
That's a great question. So, there's at least three different ways that you can engage with Answers.
Okay.
There is Answers are enabled and you can ask questions; and the Assistance Gateway. There's also the MCP.
Okay. So, there are three different channels for asking Answers a question and this lets you filter, right?
Yes. And they're measured slightly differently. So Answers itself is the questions that you ask.
Ah.
So that's pretty straightforward. MCP is kind of in the background and so it can ask multiple questions but we group questions together for the MCP. So it's not every single, you know, agent that's in the background running.
That makes sense. That's nice.
For example, I have one user who's very active. And if I wanted to see how that user is engaging with Answers, I can see that our…
Okay. So that's a list of user IDs there in that table?
Correct. And at the moment, the Consumption Report is PII free. We just have the GUIs here. You can look them up just in the console.
So out of the box, it's kind of stripped of all identifiable information?
Correct. So, one other thing to point out would be this information page. There are some links to the Help.
But this is definitely a nice reference to explain what all those different meters are, how to use the app, and even a video. This is great. Awesome. Now, it's time for Q&A. Please submit your questions to the Q&A tool on the left-hand side of your ON24 console. We've already got a few to start with. First question, how is Capacity calculated if QVDs are used or if apps use resident tables?
It's a great question. So, you can check out on the help site. We list the Data for Analysis calculation. There's two aspects of Data for Analysis. One is data loaded into Qlik Cloud from external sources and then data files uploaded or created in Qlik Cloud. And so when a QVD is uploaded or created, that data file is counted. And then if an app loads that QVD, we've already counted that QVD. So the size of that QVD being loaded into the app is not counted. And it's similar to when an app is doing a resident load. So perhaps you've loaded a table, whether it's an external data source or a QVD, and then you do resident loads. That's very common. All those resident loads, that's not pulling data externally. The data is already in the app. So if you have an app that just loads QVDs and does resident loads, it will have no contribution to the Data for Analysis.
Okay. So that's the big differentiator is external data versus data that's already on the Tenant?
Correct.
That makes sense. Thanks. Next question. How to identify the apps that use the most and within the app if we can identify what is contributing to the most Capacity consumption?
Also great question when we look at the Data for Analysis we were looking at the details by day and so I can see just I select the date I can see this is sorted by the impact as we went through this app ID is contributing the most; but the following question is the harder question: what is contributing to that app? You can't get to that level of detail here in the app maybe the App Analyzer is a way to check this app and see what data is loading and investigate what external data is being loaded.
That's a great tip actually, the App Analyzer er, I think is pretty good on what all those different data sources break down as for an app and that's in the suite of monitoring apps that can be installed using another Automation, and I'll provide a link with more information about those. Next question. If my app is 10 GBs or so, how can I use the App Analyzer tool because it's not showing full results? So, we have somebody who's familiar.
Yeah, great question. We'll follow up with you on that one because that's outside of the scope of this Consumption Report.
Yeah, I'm not aware of any limitations like that hitting a 10 GB threshold.
Well, I did see a Community Forum question about it might truncate results if it's over a certain size.
Okay.
There might be a setting within the App Analyzer load script to toggle what the limit is.
All right, well, we'll follow up, but as a tip, the Forums is a great resource for getting answers like that. If you go on the Qlik Community, we've got over 200,000 active users who are very familiar with a lot of different topics. So, if you have a question, you can post there and get some support or reach out to Qlik Support. All right, next question: Will the consumption app be added to the monitoring apps deployer to simplify partner deployments?
That would be amazing. There aren't plans for that right now. We're really focusing our energy not so much on small incremental improvements to this app as much as fundamentally better way for customers and partners to access their consumption and their telemetry data directly from the Tenant. So that's where our focus is. That'll be coming. I can't give you dates obviously, but that's coming in the near future where you'll have better access to telemetry data, better tools.
Okay, that's exciting.
Yes.
Next question: Is there a way to block or control the next package of 25 GB Capacity for your Tenant without that being automatic? I didn't realize that was automatic.
I'm not aware of the automatic. So, I'll try to answer in a couple ways. So if you're referring to Data for Analysis, looking at it from the perspective of an administrator of your Qlik Cloud and you want to control how much Data for Analysis is used, to my knowledge we don't have any way to limit that.
Okay.
And if the question is around Data Integration, I don't know. We'll have to follow up on that one.
Okay, we'll see if we can get some more details. Next question: are parquet files better for storage in Qlik Cloud or is QVD better? What are the pros and cons of them? I've heard this question before. I think it was asked in a Techspert Talks about App Optimization, how to get the best performance out of apps. And the answer was basically: it's six of one, half a dozen of the other. That they're both good. QVD is a slightly better just because it's read optimized, but they're pretty much the same. What do you think?
I'll just confess that I'm not an expert in this area. So I'm just more familiar with QVDs and they've worked in the past. If you're thinking consumption impact, like you said, I don't think there would be much of a distinction because again, it's the external data that's fetched.
Next question: Is there something like Personal Space in Data Integration to test how much data is moved Capacity-wise before going into production?
Okay, the quick Answer is I don't think so, which is not very definitive, but I don't believe so, but I do know with Data Integration, there are things in place today to account for this question because you're really about, you know, what if it's my first load? What if the load fails at the very end? Those are not counted. So, we have things in place to mark specific data loads as billable or not. And so, that might already be answering your question. It's not a Personal Space concept, but more of a first-load concept. And so that would be something just to follow up on the contract just to see how that's defined.
I tend to recommend reaching out to your account manager. It's nice to know that those initial loads aren't counted. Okay, next question: What AI usage is measured and importantly, what is available or left before incurring overages and additional costs?
This is what we're providing so far in the Consumption Report at least, just to show the Answers that are counting towards your Capacity. If you're looking for more details, believe there is a monitoring app around Answers. So that might be something to explore.
Okay, next question: Is there a best practice for implementing incremental data loading?
So that depends on what you are striving to optimize with incremental loading.
Okay.
I like incremental loading because I don't want to fetch all the data every time. If you're asking with relation to Data for Analysis, for example, if I have event data stored in Snowflake (example) and I'm doing a daily update, I don't want to load 24 months of data every day. So, I'll do an initial load, store that to a QVD, and then just fetch the incremental data each day. So, pretty much a day's worth. So then my app size that's contributing to Data for Analysis is just that days worth of external data loaded rather than 24 months worth. And so there's definitely documentation about how to do incremental data loads, but that's similar to what's listed on the best practices for managing data.
Thanks. Last question we have time for today: Is there a way to build a personalized Capacity Consumption Report using Qlik Cloud APIs? For example, we want to enrich the consumption data by showing the actual app names instead of just IDs.
You're asking a very valid question that's been asked by many. As I described earlier, we are working on the next generation of this which is very much data available to you on your Tenant. Currently, you can fetch data through REST endpoints to fetch the app names and usernames, the metadata, but we're definitely working towards something that is a little more user-friendly with names and not just IDs.
Fantastic. Well, Tyler, thank you so much for going through all the details of the Consumption Report and explaining and clarifying exactly what is counted in Capacity. I think it'll be helpful for a lot of people going forward.
Yeah, thanks Troy for having me, and thanks everyone for joining. It's great to share this with you and hear your questions and we're excited for the things that are coming with regards to the next generation of telemetry and Usage Reporting that you'll have available to you. So stay tuned.
Thank you everyone. We hope you enjoyed this session and special thanks to Tyler for presenting. We always appreciate having experts like Tyler to share with us. Here's our legal disclaimer and thank you once again. Have a great rest of your day.