Qlik Community

QlikView Documents

Documents for QlikView related information.

Using EDX in QlikView 11

tle
Contributor

Using EDX in QlikView 11

A QlikView Technology White Paper

Published: November, 2011

www.qlik.com

Introduction

Event Driven Execution (EDX) allows the running of QlikView batch jobs based on external events. The primary usage of EDX is to have an external scheduler or Extract,Transform, and Load (ETL) tool (for example, Informatica) run QlikView batch jobs as part of a larger batch process. There are also other use cases, forexample, triggering a QlikView batch when a file arrives and user-initiated batches. This document describes the requirements for each of these three use cases.

The example code (attached below as QMSEDX-v1.exe) and executable (attached below as QMSEDX_CommandLine_v1.exe) that is delivered together with this document uses the new EDX API, which means this document and the example code are useful both when using the EDX functionality for the first time and when migrating from previous versionsof the EDX API.

Use Cases

Using an External Scheduler or ETL Tool

Many large organizations have other batch requirements than just QlikView. Such organizations often use a standard tool to run all batch processes, so that the IT training costs are minimized and common operations (for example, logging and re-launching) are handled in one central place. In addition, QlikView batch processing is almost always reliant on new data being available and in many cases this data is created by other batch processes. An external scheduler or ETL tool can be used to prepare the data and then run the QlikView batch to load it.

Most scheduler and ETL tools are designed to run external batch programs via a command line interface. The reasons for this include:

  • Batch processes are long-running, so starting aseparate process to handle them provides a simple way for the scheduler tomonitor the batch processes.
  • The scheduler can be configured to run the batchusing a particular operating system account that provides the required securitycontext.
  • If the batch process fails, the standard operatingsystem error handler can be used to pass this information back to the scheduleror ETL tool.

The main alternative to using a command line tool is using some sort of web services. Unfortunately,web calls cannot be used for long-running processes, since a timeout eventually occurs. A common solution to this problem is to start the process with a web request and then follow it up regularly with subsequent requests to find out if the original request has finished successfully or not.

File Arrival Detection

In many IT departments, the inter-system communication is not orchestrated by ETL or even Enterprise Application Integration (EAI) tools. Instead, files are pushed. For example, a non‑QlikView batch job that extracts data from a production database is typically owned by the IT team running the database, which means the team is responsible for creating and configuring the batch job and ensuring that it runs correctly.On the QlikView side, a program that checks if a file has arrived must be running, so that the file can be used in a subsequent QlikView batch.

In this scenario, simply starting the QlikView batch process should be enough. There is no need to follow its execution. The program is typically a Windows service as it runs all the time.

User-initiated Action

TheQlikView Automation API provides a number of functions that run QlikView batch scripts, for example, Document.Reload(). However, it is normally not a good idea to let users run batch jobs, since a QlikView batch job typically replaces the entire data set of an application used by many users.This means a single user can interfere with the work of all the others unless the QlikView batch is carefully configured. Because of this, the batch functionality is disabled in a QlikView Server deployment.

  • The marketing department at a pharmaceutica lcompany has an organizational division between the analysts and the product managers. The analysts are in charge of loading a dozen external data sources of varying quality. Once the data relating to a particular product is ready, the running of a QlikView batch loads the data and publishes the application to the relevant product managers.
  • A software company with thousands of employees wants each employee to have access to their sales data up to the last 30 minutes. The computer resources needed to support such data usage are massive. Since the salespeople only use the system once a week on average, it is decided to allow each user to run a QlikView batch to generate a QVW (for the user)whenever needed.

EDX in QlikView 11

In QlikView 11, EDX runs through the QlikView Management Service (QMS) API. This is a major change from QlikView 10, where EDX is realized by calls directly to a QlikView Distribution Service (QDS). The QMS API is a web service API that uses theSimple Object Access Protocol (SOAP). Client applications make HTTP (web) requests to QMS on port 4799. The system is secured by NT LAN Manager (NTLM) as well as special protective measures to avoid certain types of hijacking attacks known as “time limited service key”. This combination of security means the client application must be written in .NET and therefore the provided example code is .NET projects/solutions developed in Microsoft Visual Studio 2010.

The client application uses NTLM to authenticate a Windows account to QMS. QMS then checks which Windows groups the Windows account for the client application is member of to determine the function calls the user is allowed to make. Most of the QMS API requires membership in a local group called “QlikView Management API”, but to run EDX, a separate group, “QlikView EDX”, should be used. Both groups are local Windows groups on the server where QMS runs.

The client application makes calls to instruct QMS to start a task and in return receives an error code indicating success or failure, as well as an execution ID. The execution ID uniquely identifies the execution of the task as opposed to the task itself. The client application then periodically polls QMS to check the progress of the task execution. This poll request returns a data structure that contains, among other things, the execution status, start time, stop time (if already finished), and a list of new execution IDs. These subsequent execution IDs represent the execution of tasks that are triggered because the initial task has finished. The client application can then follow an entire set of inter-related tasks that together make up a full parallelized batch flow.

Lastly, the example code outputs log information and returns an error code to the operating system. The error code can be recovered in a standard way, for example, by using the %ERRORLEVEL% environment variable in a BAT file.

Changes Compared to Previous Versions

From a functionality perspective, the following has changed compared to using EDX in QlikView 10:

  • Because communication is with QMS instead of QDS, a QDS cluster can be used to load balance EDX calls. If high availability is required for the batch environment, QMS is a single point of failure. Consider either virtualization of QMS or an active/passive system.
  • The overall success or failure code of aQlikView batch can be returned. Previously, the status of the task could be fetched (that is, the definition of the batch job), but not the execution ofthe batch job.
  • The execution of subsequent tasks can be followed.

Notes on Using the QMS API

Finally,some notes on using the QMS API in QlikView 11:

  • The EDX password must not be null. Use an empty string, if no password is configured.
  • The QDS ID is required in the function call that starts the EDX batch job. However, higher-level privileges than membership in “QlikView EDX” are required to get a QDS ID. The simple solution is to pass an empty GUID for the QDS ID, which means QMS will search for the task in all configured QDS instances. Almost all projects today only have one logical QDS configured, so this can be a useful solution.
  • EDX can be used without a QlikView Publisher license. Simply pass the name of the QVW instead of a task name and the reloadschedule of the QVW will run.
Labels (1)
Attachments
Comments
Not applicable

Cheers.

Hmm ... on second thought, having had a look at Arthur's profile - looks like he is with Qliktech, so, although unlikely, no harm in asking him if there is some way to get a degree of support

Or get some consutancy from qliktech expert services and have them set up an EDX mechanism that works for you

0 Likes
Partner
Partner

Hi,

is it possible to run partial reload (instead of full reload) using qmsedx.exe?

Milda

0 Likes

Hi Milda,

Indeed, if the reload task in the QMC has been configured to work that way. It's not something this .EXE controls, rather than how the task has been set up.

Miguel

0 Likes
Partner
Partner

Hello,

I have a problem with the EDX trigger and variables.

Here is the script of my QVW application:

TRACE var1: $(var1);

TRACE var2: $(var2);

SET var1=;

SET var2=;

The aim is to display what is stored in the vars, then clear them.

When I trigger the task with EDX with

QMSEDX.exe -task="test.qvw" -vn=var1 -vv=toto

here is the result:

23/08/2013 17:29:58: 0014  TRACE var1: toto

23/08/2013 17:29:58: 0014  var1: toto

23/08/2013 17:29:58: 0015  TRACE var2:

23/08/2013 17:29:58: 0015  var2:

If I open my application, "var1" and "var2" are empty, because they are cleaned at the end of the script.

Now if I trigger the task with

QMSEDX.exe -task="test.qvw"

(no variable set) here is the result:

23/08/2013 17:34:17: 0014  TRACE var1: toto

23/08/2013 17:34:17: 0014  var1: toto

23/08/2013 17:34:17: 0015  TRACE var2:

23/08/2013 17:34:17: 0015  var2:

var1 has always the "toto" values, whereas it should be empty.

It's like if the parameter is stored somewhere, because even if I reload my application in the management console, I still have a value "toto" for "var1"...

If I trigger with

QMSEDX.exe -task="test.qvw" -vn=var2 -vv=titi

it gives

23/08/2013 17:36:39: 0014  TRACE var1:

23/08/2013 17:36:39: 0014  var1:

23/08/2013 17:36:39: 0015  TRACE var2: titi

23/08/2013 17:36:39: 0015  var2: titi

var2 is correct, and var1 is cleared...

0 Likes
alex_stone
Contributor

Hi Lee,

     Is there any sample that without use C# code?

0 Likes
Partner
Partner

Hi Arthur.

I can't get a connection to the endpoint webservice when using certificates, ie. the communication uses https.

The error message from QMSEDX_commandline.exe is:

The QMSEDX.exe still seems to be wanting to connect using http:

Could not find endpoint element with name 'BasicHttpBinding_IQMS' and contract

'QMSAPI.IQMS' in the ServiceModel client configuration section.

Could you please provide me with the correct config for QMSEDX.exe.config.

Mine looks like this (looked like this before making the edit below):

<endpoint address="https://localhost:4799/QMS/Service" binding="basicHttpBinding"

                bindingConfiguration="BasicHttpBinding_IQMS" contract="QMSAPI.IQMS"

                name="BasicHttpBinding_IQMS"/>

<endpoint address="https://localhost:4799/ANY/Service" binding="basicHttpBinding"

                bindingConfiguration="BasicHttpBinding_IQTService" contract="QMSAPI.IQTService"

                name="BasicHttpBinding_IQTService" />

Edit:

I've tried adding wsHttpBinding and using that binding with no luck:

<wsHttpBinding>

   <binding name="WSHttpBinding_IQMS" closeTimeout="00:01:00" openTimeout="00:01:00"

     receiveTimeout="00:10:00" sendTimeout="00:01:00" bypassProxyOnLocal="false"

     transactionFlow="false" hostNameComparisonMode="StrongWildcard"

     maxBufferPoolSize="524288" maxReceivedMessageSize="65536"

     messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true"

     allowCookies="false">

     <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"

      maxBytesPerRead="4096" maxNameTableCharCount="16384" />

     <reliableSession ordered="true" inactivityTimeout="00:10:00"

      enabled="false" />

     <security mode="Transport">

    <transport clientCredentialType="Certificate" proxyCredentialType="None"

       realm="" />

    <message clientCredentialType="Windows" negotiateServiceCredential="true" />

     </security>

   </binding>

    </wsHttpBinding>

<endpoint address="https://localhost:4799/QMS/Service" binding="wsHttpBinding"

                bindingConfiguration="WSHttpBinding_IQMS" contract="QMSAPI.IQMS"

                name="WSHttpBinding_IQMS"/>


Edit 2:

Client is using QVS v11.0 SR2, all services communicating via certificates.


Kind regards

Magnus Åvitsland

BI Architect Consultant

Framsteg AB

0 Likes
paul_w_routledg
Contributor II

This utility was working really well on a Qlikview project I set up. I am now setting up an new Qlikview project on a new server and it doesn't work.

Calling QMSEDX.EXE from the command line works ok on the new server, same as the old server. Bu the real aim is to call QMSEDX from Control-M and when I try that on the new server it fails.

The old server was Windows Server 2008 (R1) and the new server is Windows Server 2008 R2 and I suspect that might be the source of the problem. We are using the same version of the Control-M agent on both servers. The agent runs as a service and I think the security is tighter on Server 2008 R2 and it can't call the QMSEDX console application because a Service has no 'console' as such. I'm guessing here, I'm no .Net expert.

This is the error that I found in the windows event viewer:

Application: QMSEDX.exe

Framework Version: v4.0.30319

Description: The process was terminated due to an unhandled exception.

Exception Info: System.ServiceModel.FaultException

Stack:

Server stack trace:

   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)

   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)

   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)

   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(System.Runtime.Remoting.Messaging.IMessage, System.Runtime.Remoting.Messaging.IMessage)

   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(System.Runtime.Remoting.Proxies.MessageData ByRef, Int32)

   at QMSEDX.QMSAPI.IQMS.TriggerEDXTask(System.Guid, System.String, System.String, System.String, System.Collections.Generic.List`1<System.String>)

   at QMSEDX.QMSAPI.QMSClient.TriggerEDXTask(System.Guid, System.String, System.String, System.String, System.Collections.Generic.List`1<System.String>)

   at QMSEDX.Program.TriggerTask(QMSEDX.QMSAPI.IQMS, System.String, System.String, System.String, System.Collections.Generic.List`1<System.String>)

   at QMSEDX.Program.TriggerAndMonitorTask()

   at QMSEDX.Program.Main(System.String[])

Any advice would be very welcome.

0 Likes
Not applicable

Hi all,

Has anyone made any progress on the error paul.w.routledge reported?

I am experiencing more or less the same problem, that is the executable runs successfully and triggers the EDX, however in the command line afterwards an error is returned:

Unhandled Exception: System.ServiceModel.FaultException: System.Exception: Could

not find a result for the specified execution ID "c6ecc697-bd25-44f6-94a2-87a58

30a5c8e"

Server stack trace:

   at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRunt

ime operation, ProxyRpc& rpc)

   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean on

eway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan tim

eout)

   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCall

Message methodCall, ProxyOperationRuntime operation)

   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]:

   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage req

Msg, IMessage retMsg)

   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgDa

ta, Int32 type)

   at QMSEDX.QMSAPI.IQMS.GetEDXTaskStatus(Guid qdsID, Guid executionID)

   at QMSEDX.QMSAPI.QMSClient.GetEDXTaskStatus(Guid qdsID, Guid executionID)

   at QMSEDX.Program.<>c__DisplayClasse.<PollSingleTask>b__d()

   at System.Threading.SpinWait.SpinUntil(Func`1 condition, Int32 millisecondsTi

meout)

   at QMSEDX.Program.PollSingleTask(IQMS client, Guid execId, Int32 pollInterval

, Int32 timeout)

   at QMSEDX.Program.<>c__DisplayClassb.<MonitorTriggeredTasks>b__a(TaskExecutio

nItem item)

   at System.Collections.Generic.List`1.ForEach(Action`1 action)

   at QMSEDX.Program.MonitorTriggeredTasks(IQMS client, List`1 executionItems)

   at QMSEDX.Program.TriggerAndMonitorTask()

   at QMSEDX.Program.Main(String[] args)

and then the executable (qmsedx.exe) fails.

As the trigger is running, I would be basically happy if someone could tell me h

0 Likes
Not applicable

What is the license for QMSEDX?

0 Likes
paul_w_routledg
Contributor II

I have solved my problem above.

We are calling EDX tasks from BMC Control-M and the Control-m service is running as the local system account.

I had to add the local system account to the QlikView EDX local group on the server to allow it to run EDX tasks.

(In fact I added the local system account to the QlikView Administrators local group which also allows it to run EDX tasks).


0 Likes
Version history
Revision #:
1 of 1
Last update:
‎2011-11-23 04:15 PM
Updated by:
tle