topic Re: Qliksense API - parallel asynchrounous data fetching using Promises in Integration, Extension & APIs

Qliksense API - parallel asynchrounous data fetching using Promises

pamaxeed — Thu, 08 Dec 2016 14:39:02 GMT

Hi all,

we have developed an extension and we have noticed that the extension are pretty slow during the data process fetching.
We implemented a sequential data fetching process which takes approximately 20 seconds for fetching 60.000 rows (3 columns = 2 dimension + 1 measure).

I was wondering if there is the possibility to use an asynchonous parallel data fetching mechanismus using promises using the backendapi from qlik.
When I execute the code it raises always an exception when trying to load multiple requests at the same time, works fine when it is limited to 1 or 2. Is this is a bug or it is simple not allowed to do it by the Qlik Engine?

Any idea or hint would be very much appreciated 🙂 !

 function fetchData() {
      var columns = config.DATA_PER_ROW;  
      var totalheight = rowCount; 
      var pageheight = Math.floor(10000 / columns);  
      var numberOfPages = Math.ceil(totalheight / pageheight);  
      var qTotalData = [];
      var deferred = $.Deferred()
  
      var promises = Array.apply(null, Array(numberOfPages)).map(function(data, index) {  
          return backendApi.getData([{
                qTop: (pageheight * index) + index,  
                qLeft: 0,  
                qWidth: columns,  
                qHeight: pageheight,  
                index: index  
              }]); 
      }, this);         


      Promise.all(promises).then(function(data) {  
        var qsData = data[0];
            for (var j=0; j<qsData.length; j++) {  
                for (var k=0; k<qsData.qMatrix.length; k++) {                                                  
                    qTotalData.push(qsData.qMatrix)  
                }  
          }  
          console.log(qTotalData);
          deferred.resolve(qTotalData); 
            
      }).catch(function(error) { console.log(error)});   
  }

The code raises a general exception (undefined) :-/.

Cheers,

Patric

Re: Qliksense API - parallel asynchrounous data fetching using Promises

pamaxeed — Thu, 08 Dec 2016 15:37:46 GMT

For all who are interested in:

It seems that the getData() Method from the backendApi has a bug.

I have used the model object instead and the getHyperCubeData method.

 var promises = Array.apply(null, Array(numberOfPages)).map(function(data, index) {  


       var page = {  
                qTop: (pageheight * index) + index,  
                qLeft: 0,  
                qWidth: columns,  
                qHeight: pageheight,  
                index: index  
          };                                                                                             
          return model.getHyperCubeData('/qHyperCubeDef', [page]);  
      }, this);      
      
      Promise.all(promises).then(function(data) {  
          console.log(data);
          for (var j=0; j<data.length; j++) {  
                for (var k=0; k<data.qDataPages[0].qMatrix.length; k++) {                                                         
                    qTotalData.push(data.qDataPages[0].qMatrix)  
                }  
          }

Cheers,
Patric

Re: Qliksense API - parallel asynchrounous data fetching using Promises

Alexander_Thor — Thu, 08 Dec 2016 15:53:28 GMT

Ye it's a tricky one, in the deep internals there is some websocket throttling and also some internal caching of data that happens when you use the backendApi that is not able to keep up when requesting a large number of data pages in parallel fashion.

When you are using the the method on the model directly then you are bypassing that and sub-sequentially the backendApi might be a bit wonky to rely on.

You could turn of the caching mechanism, it's not supported but seems to work fine

Here is an example GoogleMaps-Sense/GoogleMaps-Sense.js at master · mindspank/GoogleMaps-Sense · GitHub

Re: Qliksense API - parallel asynchrounous data fetching using Promises

pamaxeed — Fri, 09 Dec 2016 15:28:35 GMT

Hi Alexandre,

I just wondering why it is still so slow.
To fetch ca. 160.000 rows it takes approximately 32 seconds.

What is wrong in the implementation? How standard qliksense objects are fetching the data?

Kind regards,

Patric

Re: Qliksense API - parallel asynchrounous data fetching using Promises

Alexander_Thor — Mon, 12 Dec 2016 20:09:49 GMT

None of Qlik's visualization fetches all of the data up-front but instead pages in additional data as needed when you scroll of zoom in different objects.

Sending that much data over the wire is probably kinda slow, you probably want to implement some kind of gradual rendering of whatever you are doing. A lot of the metadata is already provided up-front for you to calculate min/max and labels.

Re: Qliksense API - parallel asynchrounous data fetching using Promises

Anonymous — Fri, 15 Jun 2018 10:53:21 GMT

Function paint only have access to the backendApi, I do not have access to the model. Can i access the model in any other way?

Re: Qliksense API - parallel asynchrounous data fetching using Promises

ErikWetterberg — Fri, 15 Jun 2018 14:28:28 GMT

Hi,

You'll find it at this.backendApi.model.

Erik Wetterberg

Re: Qliksense API - parallel asynchrounous data fetching using Promises

Alexander_Thor — Wed, 27 Jun 2018 08:57:40 GMT

So the reason that the getData method on the BackendApi semi-failed under async operations was due to us having an internal cache and throttle to avoid congestion on the websocket pipe.

However since Feb 2018 we did introduce a way to turn this cache off which would allow you fetch multiple pages via the backendapi in a async way.

https://help.qlik.com/en-US/sense-developer/April2018/Subsystems/APIs/Content/BackendAPI/setCacheOptions-method.htm