STT - Qlik Cloud Best Practices

[Troy_Raney]

Environment

• Qlik Cloud

Transcript

Hello and welcome to the May edition of Techspert Talks. I'm Troy Raney and I'll be your host for today's session. Today's presentation is Qlik Cloud Best Practices with our own Eric Thomas. Eric why don't you tell us a little bit about yourself?
Hey Troy. Yeah, I’m Eric Thomas. I am a Senior Technical Support Engineer with Qlik. I've been working with Qlik products since about 2017, mainly focus on Qlik Sense SaaS support as well as application development and integrating Qlik Sense in third-party applications.
Awesome. All right, today we're going to talk about how to load data from on-premise data sources; how to schedule reloads especially via apis and automations; we're going to see a demo of that, and how to manage governed apps; and also how to manage users and group permissions. Now Eric, imagine a scenario where people that are moving to Qlik Cloud and still have most their data on-premise. What's the best way to connect to that data?
Sure, well first of all, this is my tenant. For customers that are able to open up their firewall and open up their ports; they should be able to do direct connections from Qlik Cloud. But I know most customers will be concerned about that and want to go a different route; and that's why we've done work creating the Qlik Data Gateway with Direct Access. You can install that directly on a server behind your firewall and it'll just work as a Gateway between your data sources and Cloud Services so you can move your data back and forth.
Can you show us how that's set up?
Sure, I have a quick demo, I won't go through really the whole installation process. It’s installed in the background.
Okay.
What we have here is just a basic app; it's connecting to my on-premise postgreSQL database.
And that's using the Direct Access Gateway, right?
Yep. Once it is installed, you'll have access to most of the same data sources that you had before. If you just type in Direct Access,
Oh, it highlights all those connections, that's awesome.
Provided the Gateway is installed correctly and it shows connected, you can actually check that in the Management Console. I'll show that real quickly. Here's my 2 Data Gateways; I've on-prem and MySQL, these are on different servers. This one's the one I'm using for today, shows connected.
Okay. First of all, is this free?
It is free. It comes included with any Qlik Cloud license, so I guess that's the caveat, you have to have a Qlik Cloud license.
Okay. And I noticed you had 2 connections there; is it important to have a Data Gateway installed for each data source you have, or each firewall?
It's probably more of the latter. I would say for most customers, they shouldn't have to worry about having multiple Data Gateways installed. It really does come down to (I guess) how segmented your network is, and if you can only access certain data sources from certain environments; but otherwise if you have (you know) a pretty open network that can be connected to, you can just have one Data Gateway.
Okay. You've got it installed, and how do you configure that?
Sure, so this is my one postgreSQL connection here, an interface doesn't change all that much; it really instead of just being a standard connector, you're just picking what Data Gateway you're using here. So, for example, I showed you my on-prem ones, the one that's working and connected. My postgreSQL server is actually on the same server as Data Gateway. Probably not standardly recommended but shows that I can just use localhost there and connect it directly to it on the Data Gateway server. Everything else is just standard connection there.
I'm just kind of curious, when it comes to best practices with that connection is it better to use a fully qualified domain name or the IP address or does it matter it shouldn't matter if the server that it's installed on can recognize the address or Alias you're using right it should work all right so it's picking up all the data from that postgreSQL database that's pretty cool yep and then I just have really basic apps just pulling some postgreSQLdata in and storing it to EVD all right for people who wanted to find more information where could they look. Yeah,. Sure, hell page specifically dedicated to Qlik Data Gateway Direct Access they give you an overview of kind of what some of the architecture looks like here and then it goes through step by step the process of setting up goes to requirements then other things like what port should be opened luckily Qlik Data Gateway is pretty straightforward it's just TCP 443 pretty standard port and then internally we have those ports here and as far as downloading it goes you can actually get it directly through the Qlik Cloud Management console if you go into this Data Gateway section that we already looked at there is a deploy button here the acknowledging will give you installer I recommend just open up Qlik Cloud on your on your server that's behind your firewall and downloading it directly onto the server and that basically opens up like it says in the name a Gateway so you can connect to any on-premise data directly to the cloud tenant using this installer that's great exactly and definitely simplifies not having to open up a bunch of inbound ports well that was awesome and fairly easy it seems so a next scenario I wanted to go through it so once customers have their data and their apps the next big admin step would be setting up reloads for that data to make. Sure, all those apps are up to date and what's the best way to do that in Qlik Cloud. Sure, Qlik Cloud definitely differs from Qlik Sense Enterprise on Windows you can still do a standard schedule reload if I go to any of my apps here I can set up a Reload schedule and I could go through each one and schedule it so if I do have you know a chain of extract transform presentation layer type apps a standard ETL I could schedule those out customers that are used to task chaining they'll either want to take advantage or application automations or if they have a third party scheduler they could schedule out their apps completely based off our Qlik apis all right how could you set up a task chain with automations. Sure, if you have a example here it's based off an existing automation template that we have so if I go up to new automation.
Okay. All templates we do have a couple standard ones created by Qlik you just want to make your own blank automation absent parallel that kind of thing we're interested in this one right here which is reload apps is a task chain you can see kind of the basic mock-up of what they're going for here even links to a community article that goes into a ton of detail all right well. Yeah, let's take a look at the one you've already set up. Sure,.
Okay. and real Qlik with automation you've got a library of different blocks that are available on the left your canvas in the middle and all the configuration settings on the right yep that's exactly right so you have a start block that kind of kicks everything off and it'll follow down the chain.
Okay. can you walk us through it each block is doing. Yeah,. Sure, we won't go into start yet do reload essentially what I'm doing here I'm looking for a specific app I have a kind of etls already set up I have an extract app a transform app and a presentation or a UI layer app you know I want them to kick off sequentially so basically as my extract finishes I want my transform to start as my transform finishes I want my presentation layer to start so this one it's going to reload my extract app we're going to wait for the reload to complete and then I have a initial block afterwards basically looking for the status of this reload so basically we're saying here if the reload is successful we can go to this one which is another do reload if it's not successful we'll go down the chain here till our no and it's going to send an email set up to myself that says my extract reload failed.
Okay. but this is a pretty successful run I know I know all these apps reload so they typically go down the yes path here so let's say first one successful we're going over and we're starting off our transform app that we're checking again was it successful see I have do reload two that's the name of this block is it successful yes.
Okay. but we'll go to do reload three which is our presentation layer was a successful yes no if if no then it goes to send an email again and I haven't I have them named slightly different so this one says your presentation reload failed that's kind of how I kind of handle this ETL structure for these three apps that that's great and I love how visual it is too so basically this is your new Qlik Cloud task chain different from how it was in the Enterprise on Windows but I guess I will caveat you do not have to specifically use the mailbox there's actually quite a few out there if you want to do like teams or slack there's a couple other messaging options available take a look and see which one works best for you great so what are some ways to trigger this type of automation. Sure, we'll go back to the start block couple different options here you can manually trigger it from this button you can schedule it for a specific time ones that I've been leaning towards are the triggered and web hook we have a bunch of web hooks that are pre-configured you can see the connectors that are available a lot of customization options here I personally am using a triggered version right now I like this because what it does gives you a simple URL here that you can put directly in a rest browser so that you can trigger this remotely you trigger it through like Postman or put this directly into your third-party schedule if it's part of a different kind of ETL reload structure because my understanding is a lot of customers are kind of used to having some sort of third-party scheduling tools so this would tie right into that. Yeah, exactly I usually run this via Postman so basically I copied my URL here so you can see I've done that in Postman the URL is right there on the post and you've added the header right below and that key there all right so you've just triggered that same automation we're looking at from Postman yep exactly awesome so is there a way to get some sort of feedback from the automation so that from Postman you can tell that it succeeded. Yeah, there is this one runs so Qlikly that it's not giving us the ID we need but typically it'll give you an ID that shows that's queued.
Okay. you can take that ID and use one of our apis to track the status of that so if you are using like a third-party scheduler you could check the status of it and then have other processes set up based on how that occurs and I just pulled up start Qlik.dev where we document a lot of our apis.
Okay. so you could come here and search the library of apis for the type of command that you're looking for. Yeah, it's really well done and this is the one that I would utilize basically retrieves the status of the run all we need to know is the ID of the Automation and then the Run ID or the kind of ID of the ongoing task.
Okay. so this is like an additional API call that you would put into Postman to send to Qlik Cloud to get a response yep you put it in Postman or your third party scheduler and it'll give you some kind of response like this sample response is it still going is it in progress or did it complete successfully or you know worst case scenario it failed. Yeah, but you've already built in the alerting from The Qlik Cloud side if it failed but it's nice sometimes you get feedback that it isn't hanging or anything like that so what about a scenario where a customer has a large resource intensive production app and you've also got developers are working on a copy of that same app what's the way for developers to work on that version without having to use the same large amount of data as in the production app. I can go to how I set that up let's say I have my developer space.
Okay. and this is a shared space right this is a shared space so you could do collaboration you can have multiple people editing the document you can multiple people doing the reloads.
Okay. yep and I have a qvd called Source qvd activity and I have my space aware governed app and I have another space this is a manage space for your prod versions.
Okay. here I have a space aware covered app and another source qbd.qbd.
Okay. but basically this is the production app and it's loading from this data source and the developer version similarly named and running from the data source in the shared space where it's located yes exactly.
Okay. so what's the difference in these qvds can we take a closer look at those. Sure, this is my production qvd just under 2 million rows all right but if I go to my Developers app or developer space to open that data set this one is just sitting at just under 2 000 but it's dramatically smaller dramatically smaller much easier to use so let's go ahead and open our app itself and I could show you how the script works or something like this.
Okay. in a year I have two data load scripts set up already this is probably the more traditional one basically just loading a couple of my files from our source qbd difference is this one is pointing at a specific space every space has a data file so this one's pulling from App developers data files and it's looking for source qbd Qlik Cloud has a concept called space aware data syntax and this essentially means if we don't provide a space it'll reload from the context that the app is located so for example we have an app developer space we have data files and we have that Source qvd in our developer space then we have production apps in our production app space that has the data files that has its own set of qvds we named them the same so that we could utilize the same script so really here if I reload this in my app developer space it's going to be pulling from that much smaller qvd if I reload this in my production apps manage space that'll be utilizing that much larger full data set that's really smart so without changing the script the two apps can mirror each other and just when the developers make a change they want to publish to the production app they just publish that to the manage space and it picks up the larger data set automatically. Yeah, you will have to reload it oh we'll say automatically but yes.
Okay. okay but that seems like a super smart and efficient way to do it alright so once an admin gets all this set up I imagine that the next step would be to start granting access to users so they can see the production app I know admins don't want to Grant access on an individual basis what's the recommended way to set that up to make it easier on tenant admins. Yeah,. Sure, I've gotten this request a couple times in support cases well you can do and I think works really nicely is to utilize the group functionality of that Qlik cloud services has so you could go into the space details see how the app developers pulled up.
Okay. go to my members section and I could add a member individually and all right to find their permissions. Yeah, let's say dell here they want to kind of get away from. Yeah, the the much better option in my opinion is to basically have logical groups set up in your third-party active directory in this case I have a identity provider set up to go to Azure ad.
Okay. and what are the requirements for setting up a third party IDP like this. Yeah, it is required in your license to bring your own third party IDP that can pass over group claims otherwise you're using the standard Qlik IDP which actually doesn't have groups at least at the moment and we're using Azure active directory here and then the other requirement is if I go into these settings here let's scroll down just a bit it's under feature control we have creation of groups and this essentially does what it says here all right so basically means if they exist on the active directory you're using Qlik Cloud will pick those up. Yeah, exactly and I have my groups over here in Azure just so you can see what ones I'm working with these groups should all be available in my Qlik Cloud tenant.
Okay. well let's take a scenario where developers been working on a new app they want to make available for all the sales team how can you grant access the entire sales team without assigning individual permissions then. Yeah,. Yeah, so if it's forgiven to sell sales team it's probably a production app let's go to our production app space we'd go into our space details go into members here let's go ahead and go add members and then we're going to enter our group so if a Qlikly look over here it's U.S dot sales that's our sales team there it goes it pulls up it denotes that it is a group so I'll go ahead and add that in notifications will not be sent to users you can add them all at the same time and as new users are added to that sales team they are automatically giving these permissions.
Okay. so once we've done that we have our Us sales team down here and those members should be able to access this space in those apps and do those permissions that we gave them awesome so that's a Qlik way of granting everybody in that Active Directory Group access to this space and everything inside it absolutely that is sweet so let's take it from another perspective I know admins and especially support BC is a lot if a user is complaining about not having access to something that they're expecting to how can you troubleshoot that. Yeah, I would definitely start with do some backtracking if you go into space details if you go into members like we looked at before who should be getting here you can search them by name or start using groups you can go back to your Azure ad double check that the user is actually in that group that you expect them to that would be the first thing but you also can go to a diagnose claims and point that we have here this right here you just add this at the end of your tenant so I see API so this is like an API call against your Qlik Cloud tenant and what is it asking. Yeah, so this is asking what claims are coming over from our external IDP Azure in this case so you'll see some of the standard you know what's this user's name it's a user user IDs things like that what roles they have in the tenant the more interesting ones for troubleshooting would be down here you can actually see and under the extra claims portion what groups this users part of if you get that request we'll have that user look at this endpoint with diagnosed claims and you'll see for that user what they are getting from the Azure side of things so if you are using U.S sales make. Sure, that they have U.S sales as part of their group claims that wasn't there there's something you need to rectify on the Azure side of things that's awesome and that's a pretty specific API call where can people find that API call or is there an article about how to do this sort of thing for diagnosed claims that should be in the Qlik.dev apis you should be able to find that in here they're all documented.
Okay. and just for everybody on this call we're going to include all the links that we're showing and mentioning along the recording of this session once we're all finished well now it's time for Q a please submit your questions to the Q a panel on the left side of your on24 console I see we got a lot of questions that already come in Eric why don't we just take them from the top first question does using an alias create any issues Qwith connecting to on-premise data sources. Sure, I'm gonna assume we're talking about using Data Gateway here data connections are from the context of the server or machine that the Gateway is installed on Alias should be fine provided the server is on and can resolve it you know in my example I used localhost that's not something that you know cloud services would be able to resolve right so. Yeah, I would say yes.
Okay. great next question why should I be concerned with resources in Qlik Cloud why is it better to have smaller qbds for the Developers. Yeah,. Sure, maybe didn't go into that as in depth so the example that comes to my mind was from a customer that was specifically using dedicated capacity on one of their managed spaces they're also using it on their developer spaces what was occurring for them is this iPad hundreds and hundreds of millions of rows unfortunately some poorly optimized expressions and so there are limitations on resources in Qlik Cloud dedicated capacity gives you a little extra but they're getting close to their resource usage and they had two of the same app essentially they had a Dev version that devs were kind of actively updating and modifying and then a production version we were both using the same amount of data and the same unfortunate expressions and then causing them to blow through a ton of resources so this was in the you know we did actually this exact solution for them of having two spaces using space aware data syntax to allow them to have a light and lean Dev version that they could test maybe small chunks of data and validate performance and some of their expressions and optimize those and then they had a full-fledged production version using you know their full gamut of data that makes Sense. Yeah,.
Okay. all right moving on is there a way to change the fonts of all the objects in an app at once. Yeah, that's something we really talked about today but um yes I'm not a hundred percent on this one if you can change every single one with just one minor change but you should you should be able to utilize a custom theme and set the font for all the QV objects there's there should be some documentation out there on custom themes that I would look into for something like that there definitely is and we've done a few textbook talks around themes talking about custom themes when it comes to fonts so I think you're right on with that next question how can we use color to highlight KPIs in a table got a lot of visualization questions. Yeah,. Yeah, that's something we've covered uh much today if you want to highlight values in a table.
Okay. each column or each field and expression Dimension has a background expression option so you could do a little bit of logic to do the coloring there let's so if your KPI was you know if it's greater than 0.75 percent it's green if it's 50 or greater orange else it's red you could do something like that in the the background expression field on each field in the table to to get that kind of functionality going that's a great tip and I'd also recommend doing a Qlik search in Qlik Community if you're looking for some more specific syntax there's a lot of users and luminaries and even Qlik staff that are monitoring those forms that might be able to help you out with some more specifics next question is it possible to use Qlik Sense desktop and where can I find it I guess that could be still a little confusing for people with Qlik Cloud. Yeah, and actually. Yeah, Qlik Sense desktop's still out there still in use and you can actually download it directly from Qlik Cloud if you go up to your profile in the upper right corner here let's go to profile settings under Tools go the couple of the tools available and here's a Qlik Sense desktop section so it'll just get you the latest version and then you can actually authenticate it against Qlik Cloud so you can log into the Qlik Sense desktop and it'll use your same Qlik Cloud license yes Qlik Cloud desktop still out there you can get it directly through the tools section if of your profile settings awesome thanks for showing that next question can you still use extensions with Qlik Cloud yes yes absolutely in the Management console there is a extension section here where you can add an import different extensions you get from something like Qlik developer awesome great all right how can I make all the sheets in my app public when I move to a shared space there's no publish all button that I'm aware of so you would have to go into each individual sheet and right Qlik them and do the make public option for each sheet so. Yeah, unfortunately no way to publish all but does give you the control over what you want to make public or not.
Okay. next question is there any way we can duplicate the published apps to a development workspace definitely gotten this question before yes well kind of whenever you publish an app it will leave the original version intact so if that's coming from your personal space you'll have an app in your personal space that's the kind of origin version it's going from a shared space it'll still be a copy in your shared space so I guess you can go that route and you can duplicate that from your shared or personal space kind of the only caveat I've seen Distributing apps from Qlik Sense Enterprise for Windows those we just have a published version in a managed space so there's no way to really duplicate those there are some tricks out there you can find using Qlik CLI which is like a command line interface that's like streamlined a bunch of Qlik apis ultimately this is something that's looked at to be changed in the future make it easier to duplicate and manage manage steps so. Yeah, I guess more to come on that one great next question what are the user roles and is it possible to create new ones I took a look at some of those briefly in your demo. Yeah,. Yeah, we did have those pulled up during the demo so just from for a managed space if I go to add members real Qlik we can look at the roles has restricted view can view can contribute can manage can consume data can publish so those are the ones in a managed space.
Okay. self-explanatory enough what they give you access to there's a couple different ones in the developer space can view can edit can edit data in an app and can manage and can consume data like we had previously so those are those are permissions as far as the second part of that question you can't create new ones at the moment a functionality for editing or adding your own has been discussed and maybe something we see down the line but as of right now you're stuck with the predefined permissions and roles.
Okay. next question how do you hide the qbd files from users in a shared or managed space that's another kind of permissions question can consume data is the important factor there this is can list and use data sources you can also have your users if you go into this catalog you can have them just look for apps to make it easier so they're not seeing apps and qvds. Yeah, that can consume data is the toggle I would change for those users.
Okay. so if they have can consume data but not viewed then it'll just give them access the data but they won't actually see it in there. Yeah, can consume data is to let some see data connections and data files that are in that space so that would if that's off they'll just see apps.
Okay. great is there a way to migrate bookmarks from Qlik Sense Enterprise on Windows to Qlik Cloud that's a good one bookmarks are always a little bit tricky as far as the utility data does that there may be something out there in the community but from what I remember let's say we're going the traditional process of just exporting apps you export an unpublished app from Qlik Sense Enterprise for Windows that should include all content including bookmarks I think you have to go through that process of publishing what sheets that you want to use same thing with publishing the bookmarks if it is a published version of the app on Qlik Sense Enterprise for Windows it's only going to be the published content so you had a bunch of community sheets that weren't public or personal sheets in a published app those wouldn't make it across and same thing with the bookmarks I would suggest getting all the bookmarks active in an app and exporting it that should include all the bookmarks.
Okay. that's a big topic migrating from Qlik Sense Enterprise to Qlik Cloud and lots of little details like that tons of questions on that. Yeah, we could tell the whole session on that one or more is Data Gateway capable of reloading on-prem flat files like qvs and csvs from Data Gateway.
Okay. there's no connector that I'm aware of in the app that goes directly to like a file location what you can do is use any of the existing file locations connectors I mean we have a ton out there you can do Amazon S3 Azure storage Dropbox Google Cloud Google Drive SharePoint SFTP OneDrive you know so that would be what I would suggest. Yeah,.
Okay. next question how do you do Version Control and Qlik Cloud for example restoring previous versions of an app. Yeah, that's a good one too you can duplicate apps like we kind of talked about in that previous question you have your developers copying versions before they publish it and kind of renaming it you know may put a date on it or version number however you do your naming you can also do some pretty cool things with application automation I know there's a great article on actually doing this there's application blocks and application automation to serialize an app make it base64 you can upload that to something like GitHub what you know whatever repository you're using and then there's actually another block to import an app that's been serialized that's a good point I remember we had a text for talk around application Automation and we actually demoed a bit of that so. Yeah, I'm. Sure, we'll include a link for. Sure, all right what role is necessary to allow someone to trigger a Reload. Yeah, rolls let's see can reload can manage at least I believe let me pull up the roles real Qlik you'll need can manage the caveat will be you'll need can consume data for every space that you're utilizing a connection of depends on where your data connections are but. Yeah, can consume data and can manage.
Okay. that makes Sense all right last few questions are there any settings on the Azure side to ensure the Qlik Cloud can connect to Azure IDP I think there is an awesome Community article on connecting Qlik Cloud to Azure ad and it goes just step by step every step you're doing in Azure every step you're doing in Qlik Cloud to make that connection that that's what I follow to actually set mine up so. Yeah, try maybe I can provide you that great. Yeah, we'll include that link for. Sure, what monitoring apps are available or keeping track of the apps and their usage in the Qlik Cloud tenant for keeping track apps and usage you're best looking at one called the entitlement analyzer that one does have app usage and it also goes into how you're utilizing your different license allocations for your users we have our official support articles that shows all our current monitoring apps for both Qlik Cloud and on-premise so maybe you can look through there and find one that suits the need great all right last question what connectors should I use to load data stored in Google spreadsheets Google spreadsheets can be done with the Google Drive connector let me just double check. Yeah, Google Drive and spreadsheets is what I would go for. Yeah, that seems pretty straightforward and that allows you to create a connection to your Google Drive and load specific spreadsheets all right well thank you very much Eric I think this will help a lot of people get comfortable with capabilities of Qlik Cloud and you've introduced a lot of best practices for managing users and apps and reloads it's nice to address a lot of topics in one session so I appreciate it. Yeah, thank you thanks for having me if you have any other questions feel free to reach out to us on Qlik community great thank you everyone we hope you enjoyed this session and thank you to Eric for presenting we appreciate getting experts like Eric to share with us. Here's our legal disclaimer; and thank you once again have a great rest of your day.

[NahuelBourdichonAlten]

hello excellent video. I write because I am very confused in how to organize my space, I come from qlik sense and created a stream for each layer extraction, transformation and app and then programmed the task, in cloud I don’t know how to do it.
example: I create an space (which shared, data or management ?), I save the qvd in the same space ? I keep the automation file  in the same space.
There will be a guide how to do it because I cannot find information in web

thanks 

[Sonja_Bauernfeind]

Hello @NahuelBourdichonAlten 

I would recommend posting about your specific need in the Move to Qlik SaaS section of our forums.

For information about spaces (both for apps and data), see Working in spaces. For general information about how to manage your source data, see Adding and managing your analytics data.

All the best,
Sonja