Using Qlik Lineage API's with your On-Premise Environment(s)

[Dalton_Ruer]

Starting Point

Before you get to excited, the Qlik Lineage API's do NOT work against your Qlik Sense Enterprise on Windows environments. But I used that title because that is how I searched when a customer said "I really like tracking the lineage for my SasS applications, but I also need it for the Qlik Sense Applications I still have on-premise." 

Don't get discouraged just yet, as I'm going to show you a way forward that will let you capture the lineage you want using a product Qlik provides that you might not be aware of ... Qlik Lineage Connectors. 

In a nutshell, the Qlik Lineage Connectors collects all the lineage information for your on-premise environments and publishes the needed METADATA for them to the Cloud, where you can use the API's. No data is moved to the cloud, and you do not need to reload anything in the cloud to make it work. 

Installation & Configuration

Note: You will need at least a bare bones Qlik SaaS tenant to serve as the catalog for your lineage data. Follow the instruction for the Qlik Lineage Connectors application, as the point of this post is to help you after that so you can configure and get the Qlik Lineage API's working on all of that precious on-premise lineage metadata. 

The metadata can only be published to a Managed space within your Qlik SaaS Tenant. In the following screen shot you will notice I conveniently named that space "OnPremise" so there would be no doubt in my old mind what the space was when I see it a few weeks from now. 😉

The following screen shot shows what that space looked like before configuring and running the Qlik Lineage Connectors in my environment: 

Executing Qlik Linage Connectors On-Premise

You can run/test the process by going to the Scheduler tab and simply pressing the run/play icon. As you will see in the image below I configured my environment to be limited to a single stream that had 22 applications in it. 

As the process was going I began seeing "items" appear in my "OnPremise" space. 

Output

I used the word "items" because they are a type of object called an "External link." You can click on the link and it acts just like other hyperlinks and would launch the application in your QSEoW environment. However, it also has the advantage which is driving this post ... the ability to see the Lineage Metadata for those on-premise applications. Simply right click on the "external link" and choose either "Lineage" or "Impact analysis", just like you would any of your SaaS applications. 

 Yeah! We are in business: 

API Calls

If you checkout the Lineage APIs you will notice a pretty simple format. 

https://{Qlik Tenant}/api/v1/lineage-graphs/nodes/{QRI_ID} 

If you look at the online documentation or examples people have shared they all involve "applications." Which shows you that the QRI_ID involves a pattern of "qri:app:sense://{Application Id}"

Alas, these "external links" are not "applications" they are simply pointers to the on-premise applications along with the lineage metadata. So, that format won't work for us. However, we can easily get that QRI_ID value using the Qlik "items" API call. 

Items API

In the documentation you will notice that one of the parameters that can be utilized for the "items" API call is something called "resourceType." That parameter allows you to filter the API to exactly what you are looking for which we need to do in this case. The "resourceType" value we want is "genericlink."  

Notice that in my screenshot (from Postman) that in addition to passing "genericlink" for the "resourceType" parameter, I have also configured another parameter called "spaceId" that then limits the list to the "genericlink"(s) found in my "OnPremise" Managed Space. 

If you look at the blue highligted area in the screenshot above it output exactly what we are looking for in the form of the "secureQri" value. Woohoo! We can programmatically iterate through the output from calling the items API and pass the QRI_ID values to the lineage API.

Lineage API

Two important parameters for the Lineage API are called "level" and "up." The "level" parameter lets you define what aspects of the lineage information you want. While the "up" parameter lets you declare how many values to return. 

In my call, notice that I have indicated that I wish to get all values, and I want the resources.

You simply change the "level" parameter if you wish to obtain Table names or Field names and the information about them. Or set it to all and it will return all of the metadata you could dream of.