Hello everyone and welcome to the October edition of Techspert Talks! I’m Troy Raney and I’ll be your host for today's session. Today's presentation is exploring Qlik Application Automation with Jeffrey Goldberg. Jeff why don't you tell us a little bit about yourself?
Thanks Troy. For those of you who don't know me, I’ve been with Qlik now for almost two years on my second tour. I worked for Qlik for four and a half years in pre-sales. I focus now as a product manager. I’m certainly focused on Qlik Application Automation, but I’m really also focused on APIs, integration, identity management and anything that really has to do with enhancing the platform either through extensible components, extensions like visualization extensions, and the APIs that support those things. So, that's why Application Automation kind of falls into my realm because a lot of this is; while it's a no code tool which we'll talk about in a minute; is all driven by the APIs that are available in our platform; as well as other cloud platforms.
All right. Today, we're going to be talking about what Qlik Application Automation does; how it works; we'll get to see a little bit of a demo there; some best practices; and really we're going to be focusing a bit on how to troubleshoot common issues and what to do when things don't go as expected. Jeff, for those of us who aren't familiar, can you tell us a little bit more about what Application Automation can do from a high level overview?
Absolutely! The thing about Qlik Application Automation is it's not just a feature. It's actually its own unique product within the cloud platform; which is called an Integration Platform as a Service. And we've now integrated into Qlik Cloud. And what it enables you to do is: leverage analytics as part of an application integration experience. When we looked at Blendr, and brought it in and made it Qlik Application Automation, what we really wanted to do is: we talk about Active Intelligence, but how do we make it real? How do we help accelerate decision making and make it so that people can reach that state of Active Intelligence without having to do manual work?
So, it's kind of replacing a lot of scripting and tasks with visual blocks, so you can kind of see how it's mapped out.
Yeah. So, it's a great way of thinking about it. It's basically a visual way of scripting your daily workflow. There really hasn't been a good way without being a programmer to connect different applications that you use every day. So, you know using Hubspot and Salesforce to migrate or move contacts between the two systems. And you need to make sure that data stays in sync between the two of them.
This is where Application Automation can come into play, because it can take care of doing that work for you without having to involve someone from the IT organization or having to know how to code in order to facilitate that that kind of experience.
And what are some popular use cases for it?
A lot of what people are doing right now, since we've released Qlik Application Automation is analytics automation and the analytics operations, right? Sometimes this is called Passive Intelligence.
So, there's really like four different types of use cases. What we want to get people to is that state of Dynamic Actions, right? Where you're leveraging analytics that drive data decision-making in an automated way, but recognizing that there's a good amount of upfront work that has to be done in order to make sure that you're processing that data and getting it to a state where you can go ahead and look at dimensions and measures, perform selections on that data using the blocks that then allow you to do the evaluations which enable you to then trigger the actions in the downstream business applications you're integrating with.
So, based on the data you get to actually trigger things and initiate workflows?
Cool. How should someone get started using Qlik Application Automation?
Yeah. So, let's just jump right in, right? So, I’m on Qlik Cloud, and I am in the Analytics area of the platform. So, when I want to start building automations, I can just go to Add New, and choose New Automation. And then I can…
Give it a name.
And that brings you right into the Automation Editor. And the Automation Editor has three different sections. There's the Block Menu, which is on the left and you can see that there's a number of different connectors and block types there. So, we have basic blocks lists. We have some advanced blocks cloud storage blocks and then we have connectors and you can see it's showing the connectors that I use most often in my other automations.
Yeah, that's cool.
There's a number of other connectors down below that you can choose from. And we're adding new connectors on a monthly basis. And then when you want to access those connectors, you can just click inside, and that will bring you to the actual blocks that are available. And you can search for blocks through the search menu or you can use these different tags that are up at the top to filter the list and get started.
What's the center called?
The middle is the canvas. This is where you actually drag and drop the blocks. And then, when we drag and drop the blocks, on the right hand side, that's the block configurator.
Okay. Can you walk us through setting up a reload automation?
Yep. Type “Do” in the search, and I drag and drop Do Reload. First things first, you see the blue pulsing square? That just shows that when I let go of the mouse button, it's automatically going to attach to that point in the flow.
And the details of that on the right?
Yep. Now, because I’m using the Qlik connector here, I’m already authenticated to my Qlik Cloud tenant, so there's no need to provide any credentials.
I can go right into, and I can search for my apps here. I have this app called TC. You can see that I have the app grid here. And then I can say I want to wait for this reload to complete, or I can set it to run asynchronously. If I set it to run asynchronously, then what I can do is add another Do Reload. And I can go over to here and do another lookup, and I can look for my second app. Oops. I can do, I do look up and I can look for my second app. What happened just now is: you see that red box that's around App ID? The reason why that turned red is: I started typing and then I deleted. It's basically telling me: I can't run this flow without supplying an App ID. What's nice about the platform is it actually tells you intuitively when you've got errors in your configuration that you need to work through; and we'll show a couple more of those.
I actually love that part, that it kind of tells you right away the things that are missing.
It's really nice, right?
Before it creates an error. Yeah.
Yeah. So, I’m going to also do a start reload and continue here. And what this is going to do is: it's actually going to kick off the reloads for these two applications at the same time. And there's another block inside of here called Wait for Reloads to Complete. And you use this block to basically say “I want to wait for these two reloads to complete before I do anything else,” right? Another way that you could do it though; you could go back and; one of the things that's nice is: in the canvas you can just drag and drop. And whatever is not in the sequence doesn't get run. I can go ahead and I can go into the Basic Blocks here, and I can add a Condition.
And if I switch this Do Reload to wait for the reload to complete, then I can go into the Condition. I can choose from the input, you'll see that there's the output from the Do Reload ready to use as data that I can use to drive the continued execution of the flow. So, in this case, that output from the Do Reload, you can see there's the App ID, and there's also the status, right? So, if the status equals Succeeded, that's when we want to run the second reload. And we'll also wait for this reload to complete. And if I want to I can add another Condition here to do something else or send a message to Microsoft Teams. Drop that in here. And I can also drop another one here at the bottom, and I can even change the flow, right? So, the editor is really nice and flexible. Let's say you want to just grab a hold of a whole chunk of an automation. Wherever, whatever block you click, and you disconnect from the flow. It's going to take all of the blocks below it.
That's a nice tip for how to use the canvas and mix it up.
So, once it's connected, how does the send message work?
Go into here, and you'll notice I already have a connection to Microsoft Teams, but I’m going to delete that and you can see immediately I already have this…
And if I go ahead and I try to save this, it's basically throwing (before I even can save) an Invalid Block message. And it's telling me: you've got a problem with this block. This block doesn't have these fields filled out; because you don't have these fields filled out, it's not going to work. Now, I can go ahead and save it anyway.
It's nice to have that option.
Yeah. It's nice, right? But if I go and I go and run, and then you're going to get this error. And it's basically going to say “oh, I couldn't find Microsoft Teams, not either never existed or may have been deleted.” What you need to make sure that you do is: first you need to make sure that you have a connection for Teams. Specifically, sometimes you have to have different permission sets. When you choose to link, it will pop-up this box. It will ask you to pick an account and then it will connect. So, now that that connection's all set, you can see that it's there I can go into the inputs. And you can see, I’ve got the red that basically is telling me: I need to fill this out.
It's looking for what profile to use in Teams, I guess?
Yep, exactly. It's looking for the different team that you want to send the message to.
And I want to send this to the output channel, and I want to say: the first reload completed successfully. Save that. And you can see, I still have an Invalid Block, because I have this Team's Message, but let's just go ahead and remove this one. And now you can see that it will save appropriately. And if I run this; if you go into the overview, you can see: what your last run was; you can see what your history is; and you can see we have our failed item. We have this one that's running, and once the automation finishes, you get this really great Per Block run. Which doesn't necessarily put the blocks in order, but you can go in and you can look at the execution; and you can see the information that was sent into the request, and then what came out of the request which is pretty cool when you're trying to troubleshoot different things that are going on.
Yeah. So, if something failed like one of the blocks, could you actually see where it broke down there?
Yep. Yep. So, what would happen is you would see in the Per Block if there was an error where it broke down, and you would also see that the error is here. But you're right; you could come back to the Per Block, or better yet you could go to the Chronologically tab. And what this will do is: it actually shows you the sequence of everything that was kicked off. So, and it will give you a preview as well. So, you can click on the drop down here, and you can also see the same information. So, I’d recommend using the Chronologically tab.
And can we take a look at what the message it's sent to you in Teams looks like?
Absolutely. I’ll move it over.
That's very simple, but yeah, it's nice to know what it looks like. Now, I understand there's a ton of possibilities with Teams and with Slack that we don't have time to go into today, but back to this reload: what can affect the performance of these reloads?
So, depending on the size of your applications, they may take a long time to reload depending on how much data you're actually reloading. But it could also be a result of the reload queue that sits inside of the cloud platform. Depending on when your request goes into that queue, that's when it's going to run. So, if you've got a lot of concurrent reloads happening at the same time, you may find that your load ends up running later on.
Okay, now I’ve heard the terminology and I’m hoping you could explain the difference between a task chain or automation and an orchestration?
It's a great question Troy. I actually don't see a whole lot of difference between a task chain per se and an orchestration.
But then it's about focusing on operational efficiency and productivity. There is aspects of data movement. It's really about moving small bits of transactional data in support of business processes. It's not about like bulk data movement. When people are talking about orchestration and orchestration of pipelines, they're talking about “we're moving data,” right? We're not just doing this. When we think about orchestration, right? It's directing automations within end-to-end processes, right? So, if you think of just an API call opposed to going to a screen and clicking three things in order to do something; it's very much event driven or dependency driven versus “oh, I’m just sick and tired of having to click five buttons at a time.” I think that's the way a lot of people think about the differences between the two things.
Now getting back to SaaS, suppose there are several automations built without those wonderful emails or Team messages/notifications. How can you view all of the automations have been created and see which ones might have issues?
Yep, great question. If we go out to the Management Console. Inside of the Management Console, you will see that there's an Automations tab. And you can go ahead and see from here all the automations that exist on your tenant. And you can see when the last run; whether the last run failed or not. So, if we go and say “oh we want to look at Icons.” So, let's back out here, go to the Catalog, Icons, right we can see here's our last run brings us right to the error. We can go into the Chronology. I only have one block in here. So, that's the, that's the issue. But I can then also look at the full history here. So, it's not that I have to go back to the Management Console; that's just one place that you can look at it from.
Okay, is there a way that users can manually trigger a flow from an app?
So, there are some different ways that people have done that. Using the Action Button can use the navigation to the website URL, and there the URL of the automation in there. And the way that you get that automation is by going into Run Mode here and choosing Triggered. There's an example here that represents the URL for this automation.
When you use that website URL, it pops up another page. Now let me just go into my copy of Consumer Sales app. This is the app I use to test with.
So, I understand you've been improving the Action Button, since it's open source.
I’m working on updating this button, so that instead of going to navigation and choosing ‘open a website or email’ you can actually go into the button here, and you can choose Execute Automation. And then you'll see the list of automations that exist in your tenant.
So, it's picking up all the automations you've already created in your tenant there?
It's picking all of those up and so then you can choose: “Oh. I want to do Hot Dog Alert.” And you're also going to be able to choose “Oh, I want to Include My Selections.” So, if I go ahead and I do this, and then I do this, and I press this button, and then I click on this button, and then I go out to my hub, go and look at my hot dog alert; you should see that it started, right? Now, because I’m just testing this because this is still something that I’m making sure works properly; all I’m doing is passing inputs and we'll have videos once it comes out. But you can see I’m passing into this automation the App ID and the Bookmark ID. When you do that selection check box, it creates a bookmark. And now you can go ahead and reference that bookmark inside of the automation for that App ID, and now you have the starting point of your flow. So you can start working with different charts and stuff like that inside of the application, and getting the data that you need.
That's really awesome.
Yeah. To be clear: not out yet.
What are some best practices or details that admins should be aware of when managing automations to avoid potential issues?
Yep. So, automations today will only run for a maximum amount of one hour.
You can't create a runaway automation that just runs forever. Now there are some other types of execution that have some potential consequences; and you want to make sure that when you're building your automations, you're thinking about it, right? In the case of something like this automation where we're using a triggered web hook, right? It's great that we're using a triggered webhook URL, but let's say I’m putting it someplace where it gets called a lot.
You're going to queue up a lot of automations, and they're all going to run. Now, automations don't run simultaneously. Automations always will run one at a time. The thing is if you have like a really small automation like this one which runs for like one second; and you are triggering this URL like crazy, then you're going to hit your threshold of available automations. Because the way that we offer this is a number of automation runs per month. So, as you're building your automations. you want to be cognizant of the use case.
Yeah, and let's address one issue you brought up there. Is Qlik Application Automation free?
There is a free tier for Qlik Sense Enterprise SaaS customers.
And Hybrid Data Delivery customers. The free tier gives you 5000 runs per month currently with the ability to run 5 automations simultaneously. That's what we call concurrency. But we do have 3 different offerings that you can purchase that expand your capacity. So, you can buy more what we call “automation runs.” So, you can go from 5,000 to 20,000 or 60,000 or a 100,000 and you can contact sales to get more information about the pricing of those.
That's great. Can you show us another example of what's possible with Qlik Application Automation?
I would love to. In fact, what I want to do is show how you can take advantage of analytics to compel action in other systems.
Let's say I’ve got a Qlik Sense app. I’m looking at sales data, and I’m also looking at this Account and Opportunities detail. You can see I’ve got these Marketo scores and I’ve got these Hug ranks. What the Marketo score is reflecting is the interest that someone has in the materials that we're sending them. What the Hug rank does is it shows how we (you know fictional company) are engaging with you.
So, a high Marketo score means a highly engaged customer, and the Hug score shows how well engaged the company is with that customer. So, this helps identify sales opportunities?
Exactly. I know that there's a certain threshold combination between the Marketo score and the Hug rank where it’d be cool if I could actually make automatically the sales person gets an activity inside of Salesforce that they have to act upon?
So, I see you've completed a flow would you walk us through how this works?
Yep, absolutely. I’m using an event which is App Reload Finish. So, I can go over to my Run Mode and I choose App Reload Finished. And then, the first thing I’m doing is I’m making sure that this is equal to the app that I’m looking at which is my Sales Pipeline Analysis which it is.
And I can start selecting Stage, and I’m looking for accounts and opportunities that are in champion stage.
Once I have that, I want to get the straight table data from this table. Now, I’m selecting that field value first is to reduce the amount of data that I have. The get straight table data only will bring back a 100,000 cells, right? And that's a limitation inside of Qlik Engine. You wanna definitely make sure you have an understanding of the selections you wanna make on the data. If you try to grab something that's going to have a million cells, automation's going to fail.
That's a good troubleshooting tip.
Once you have your straight table data, you add another Condition, right? And this is where you can add your Thresholds. So, in here I’m saying: well, if the Marketo scores equal to 110, and the hug rank is equal to 9 (and this is just for the sake of demo), I want to create a task. And inside of that task, there's specific fields that have to be filled out; like the What ID, the Subject of the task.
Yeah, all those fields and Salesforce that are necessary.
And you can see I can actually use data that I’m grabbing from the straight table in the message that I want to put inside of Salesforce. And then, I’ll also send a Hero Card to basically say “Hey, you've got these records. Do you want to go and look at them?” And we can actually embed the links so that they just click the button.
Yeah, that's so convenient.
Yeah. We have a playlist looking at how we construct this in real time. And you can see inside, there's this Active Intelligence Salesforce walkthrough.
That's a great resource for learning more. By the way, the link to this playlist and more will be included with the recording of this session.
And when that application reloads, it does all of this automatically now.
So, that lets people easily take action on those insights which means a lot less manual work. How does all look from the application side?
When this automation runs, it actually creates these activities right here and I can go into the task and I can basically see “Oh, Jeremy Webb from Tristique Associates has a Marketo score greater than 100, but a hug rank less than 15. Contact them and engage them about our new Application Automation capability and Qlik Cloud.”
And that was created by the automation?
Yep. And we can confirm that it worked because we can go into the automation here. And we can see this is what we sent. And then we can see here's the out, which is a Success=True.
Can we see again what that looks like in Teams?
You can see that it's created these items. There's that Active Intelligence. But one of the nice things is as you're actually building this there's ways that you can actually troubleshoot your flow by moving blocks around, right? So, if you do find yourself in a situation where you have to do some troubleshooting, you have a lot of flexibility in the editor to move blocks around to kind of troubleshoot what's going on.
I see running the test runs of individual blocks as well. That's really cool.
Yeah. Test blocks are really nice as well because they basically give you back sample data. So, like let's say: you're doing something for the first time. You don't want to have to do a full run. You can do a test run on a block based on information. So, if I go and I hit “Do this test run” here, what it will do is: it will run everything. So, let's go ahead and grab this out. You can see it didn't like my Condition because didn't get the App Reload Finished. So, we'll switch this back to Manual Mode, right? But if I do a test run, it will only run to the block that I’ve done the test run at.
So, in this particular case, it labels the output as a Test Run, and it basically says that it made the selection. And then, I can also do the test run here. It will go ahead and it will show “Oh, yeah okay. So, it brought back that test run information.”
Great! Now, it's time for Q&A. Everyone please submit your questions to the Q&A panel on the left side of your On24 console. First question: is there a function to allow developers to request application promotion from one environment to another?
Oh, that's a good question. We are getting a lot of - we are receiving a lot of feedback about things like “Oh, I want to do app promotion. I have two tenants and I want to basically say: this is where I do my development. This is where I do production.” Right? In a lot of cases, we have customers that use the same tenant, right? And they have different spaces. But in the event that you really do have two tenants, currently with Application Automation you notice (and I mentioned this at the beginning) when we first… Let me go back to the - you'll recall when we earlier - when I talked about the use of the Qlik Blocks. One of the nice things is: you don't have to worry about setting up your credentials because it's bound to the tenant that you're actually working with. Now, that's really nice because you don't have to do that; but what right now we don't have the ability for you to connect to another tenant like with an API key using the same set of blocks. So, what you can do is: you can use the call URL block (as an example). And say let's say: you're going to run this automation on your production tenant as the as a solution that's going to perform the app promotion, right? You could use the call URL block with an API key to contact that the development tenant; reference the app; export it to base; and then import it as base into the production tenant. So, that is one of the ways that you can do app promotion. We are looking at a number of different ways to streamline that process. Either through automation or through the core cloud UI. Stay tuned for those updates as time goes on, and we learn more about how our customers use Application Automation.
That's great. Next question was: what happens if a flow fails? Is there a way to trigger an alert email to the admin?
Yes, actually. If we look at this example right? So, we have succeeded, but if it didn't succeed you can easily go ahead and use something like the Mail Block, right ? If you wanted to send a mail (an email), you could use a Mail Block. And you do have to configure that with your own SMTP server. It doesn't use the SMTP server of what's in the Management Console. So, you do have to specify that again. But you could also send a message with Slack. You could send a message with Teams just like we did previously, right? I can copy this block and I can paste this block, and I can drag it into here.
It's great that there's so many options available for communicating.
Yeah, exactly. And that was one of the things we wanted to make sure of is; that there's so many different ways, right? You can use if you subscribe to Mailchimp, you can use Mailchimp. If you subscribe to SendGrid, you can use SendGrid to send messages. Out there's lots of different ways that you can go ahead and produce messages based on whether or not your automations are succeeding or failing. The other thing that you can do is: there is an Error Block, right? And when you put the Error Block in, this will actually create the message. And you can choose to stop the automation or continue the automation. And by doing that, that's what's going to show that will show up, right? If I come back out into the overview, if we look at things like the Chronologically or the Per Block, and you have an error; you can actually specify setting up that error.
That's kind of interesting. Next question: referencing Qlik Data Transfer can trigger a reload of an app. Is there any plan to trigger an automation instead when the data is synced?
I’m not sure about Qlik Data Transfer. We are working with the data integration team for Hybrid Data Delivery to create APIs and events that will turn into blocks inside of Application Automation that will respond to events based on data changes. We also are looking at when you're landing data inside of data files which is what the Qlik Data Transfer tool does. There should be APIs and events that that we can hook on to. So, I’d say stay tuned. Stay tuned for that.
Is it possible to write data to Qlik Cloud?
In terms of writing data to Qlik Cloud specifically; certainly you can go ahead and grab data from different data sources like SQL Server, like Hubspot, like any of these connectors. They're all producing data. You could do something like writing a load script manually inside of here because there is the possibility to set a load script inside of an app. But if you're looking to move like data files, currently there's no way to move data files in the Application Automation tooling similar to Qlik Data Transfer. That may be changing in the near future. It's all - it's a matter of making sure that the APIs are public and guidelines compliant. But there's numerous different ways you can bring data into the cloud through APIs. And you don't even need to use Qlik Blocks in order to do that, right? You can just go ahead and use APIs to create files or move files and move data around to different sources and targets if you're trying to integrate Application Automation to do non-Qlik things (which is a totally perfectly fine use case).
Where can I find an overview of all the automations I’ve created?
Yeah. So, in terms of the overview, right? The best place to go is the Hub. All automations that you create are in your personal space. So, if you go over to the Catalog, you can click on all filters and you can choose Automations, because automations are first-class citizens just like apps or your monitored charts. And you can see them and then you can go ahead and you can look at your list if you need to see what their performance is if you've got access to the Management Console. The Management Console gives you access to look at that list. We have had some feedback come from customers as well as the field saying it would be good to have a mechanism for supporting that in this UI as well. So, that people can see what they've got running and have more of an overview. So, something we're thinking about.
How can you tell which app the automation is for from the editor?
Ah, yeah. So, you really can't. Because automations aren't necessary (like I just said) they're first-class citizens, right? So, they're this they kind of operate at the same level as an app. They're not child object of an app like a chart or a story or anything like that. Because an automation doesn't even need to interact with Qlik Sense. It can interact with Hubspot, and Salesforce, or Marketo in Salesforce, or ServiceNow. Based on other information that's coming into the system (either through another data connection that you're making to) like a SQL data source or from another application in your Enterprise portfolio. So, it's important to understand that if you are trying to associate automations with the apps that they're tied to, your best bet is to go into the edit of an app, and add tags, right? So, you could add a tag that aligns this; like I’ll just align this with GMS. And now, if I go ahead and do a search on all things with tag GMS, you're going to see I’ve got copy of Consumer Sales and then I also have my first automation. And that would be…
That's a smart way to do it.
That would be the way that I would organize it.
Okay, last question: I’m looking for a way to send ‘task fails/success emails’ to specific users. Is there a way that can be done through automation?
Absolutely. If you go back to using with Slack and Teams a little harder because the way that they're structured, you typically will send information to channel. Although you can send in Slacks to (I believe) to specific users. If you're sending an email, it's just a matter of knowing the email address of the person that you want to send it to. And it could be that you're, let's say you've got Section Access inside of an app. If you leverage the Users Table, let's say you're an admin on that that app. You're going to see everything. I would use the Section Access table and use the email as your user identification field. And that way, you could pull that email address out; and you could send emails that way. Or you could have a list that you sideload either through a text file or Excel or a database should do that.
That's really smart. Well, thank you very much, Jeff. This has been great.
Troy, it's been an absolute pleasure to spend some time with you today and go over the platform. I know we showed a demonstration of some Active Intelligence, and then we also went over some different places where you're going to spot errors, and stuff like that. There is a tremendous amount of content that we've actually produced so far. You saw the YouTube channel. There's also our community. We have an Application Automation and integration forum on community. And then we also have a number of how-to articles so that you can learn how to use the different patterns and blocks, and you can go ahead and upload those patterns directly into the editor so that you can save yourself some time. Thanks for the time today to chat.
Thank you everyone. We hope you enjoyed this session. Thank you to Jeffrey for presenting. We appreciate getting experts like Jeff to share with us. Here's our legal disclaimer. And thank you once again. Have a great rest of your day.