Qlik Sense APIs capabilities

[swudali4786]

Use Case Example:

1) Messages in the form of XML/JSON originate from Microsoft BizTalk Server based on communication (B2B) with a Purchase Order system. The XML/JSON message contains the data passed by the PO system about a new or an existing PO. Messages are generated whenever a PO request comes in.

2) Qlik sense then needs to load the data in the XML/JSON messages into an existing Qlik Sense application.

PUSH from Biztalk: Is there a way for BizTalk hub to call a Qlik API  that can take the data in the XML message as an input and load the new data into a Qlik sense application thru an incremental process?

PULL from Qlik Sense: The XML messages are not being persisted in Biz Talk so I am not sure if a rest API connector mechanism can help in this case where reload tasks from QMC can be triggered via QRS API.

Any thoughts around this is much appreciated.

[Vincenzo_Esposito]

It is a long debate when it comes to manage real-time data, and data coming from a MOM have to be considered real time some extent. In 20+ years experience across different areas, gave me the belief the word "Real time" is more related to the user's feeling that a strict scientific definition. I met people happy to call a 30 seconds delay system "real time or near real time".

Qlik Sense is definitely not a stream data analysis, meaning that there is a time to digest the incoming data different from zero. The less the time Qlik takes to read data, the more often it can be updated. I haven't see reload time under 30 seconds, so far.

Said that, the best way I can advice to address your needs is:

1) To buffer all messages to a persistent system (e.g. Database)

2) Create a task to read these data from the persistent system

3.a) You can trigger the reload task via API (e.g. BizTalk trigger the task each <n> messages)

3.b) You can schedule the reload task to run each <n> seconds

Hope this help

[swudali4786]

Hi Ves,

Thanks for your prompt response. This use case is for a "near real time" scenario. 

Having BizTalk persist to a database is a great idea. Another option that I could think of was to

i) Have BizTalk  team write all outgoing messages (XML files) to a folder on some disk. (set up an archival strategy to make sure folder size doesn't increase to unmanageable sizes as time progresses).

ii) Set up a general web connector that connects to this folder structure and the XML files.

iii) In QMC set up reload data and app tasks.

iv) Have BizTalk trigger the reload thru Capability API  whenever a new XML file arrives into the folder(Or is there any other API to trigger a reload data and app that is more appropriate?)

for item ii) is there any other option apart from Qlik general web connector that can help connect to the folder containing XML files?

Also I believe volume of messages is a critical factor in the triggering mechanism -  if the first message arrives into the folder and BizTalk triggers a reload thru an API and the reload task is still running. Meanwhile if another message arrives will the reloads be queued in QMC? Sorry if this sounds like a dumb question but I have not run into a similar situation before and want to evaluate what options should i be considering to arrive at an optimal solution.

Thanks in advance for your help!

Regards,

Srik

[Vincenzo_Esposito · Accepted solution]

That was my point when I wrote Qlik Sense is not a data stream analizer. You cannot trigger the reload task on the message incoming, because it fails if a reload is already in progress. Hence my suggestion to accumulate the messages on a persistent layer and consume each 30sec. or 1 min.