Qliksense API - parallel asynchrounous data fetching using Promises

[pamaxeed]

Hi all,

we have developed an extension and we have noticed that the extension are pretty slow during the data process fetching.
We implemented a sequential data fetching process which takes approximately 20 seconds for fetching 60.000 rows (3 columns = 2 dimension + 1 measure).

I was wondering if there is the possibility to use an asynchonous parallel data fetching mechanismus using promises using the backendapi from qlik.
When I execute the code it raises always an exception when trying to load multiple requests at the same time, works fine when it is limited to 1 or 2.  Is this is a bug or it is simple not allowed to do it by the Qlik Engine?

Any idea or hint would be very much appreciated 🙂 !

 function fetchData() {
      var columns = config.DATA_PER_ROW;  
      var totalheight = rowCount; 
      var pageheight = Math.floor(10000 / columns);  
      var numberOfPages = Math.ceil(totalheight / pageheight);  
      var qTotalData = [];
      var deferred = $.Deferred()

      var promises = Array.apply(null, Array(numberOfPages)).map(function(data, index) {  
          return backendApi.getData([{
                qTop: (pageheight * index) + index,  
                qLeft: 0,  
                qWidth: columns,  
                qHeight: pageheight,  
                index: index  
              }]); 
      }, this);         


      Promise.all(promises).then(function(data) {  
        var qsData = data[0];
            for (var j=0; j<qsData.length; j++) {  
                for (var k=0; k<qsData.qMatrix.length; k++) {                                                  
                    qTotalData.push(qsData.qMatrix)  
                }  
          }  
          console.log(qTotalData);
          deferred.resolve(qTotalData); 

      }).catch(function(error) { console.log(error)});   
  }

The code raises a general exception (undefined) :-/.

Cheers,

Patric

[pamaxeed · Accepted solution]

For all who are interested in:

It seems that the getData() Method from the backendApi has a bug.

I have used the model object instead and the getHyperCubeData method.

 var promises = Array.apply(null, Array(numberOfPages)).map(function(data, index) {  

       var page = {  
                qTop: (pageheight * index) + index,  
                qLeft: 0,  
                qWidth: columns,  
                qHeight: pageheight,  
                index: index  
          };                                                                                             
          return model.getHyperCubeData('/qHyperCubeDef', [page]);  
      }, this);      

      Promise.all(promises).then(function(data) {  
          console.log(data);
          for (var j=0; j<data.length; j++) {  
                for (var k=0; k<data.qDataPages[0].qMatrix.length; k++) {                                                         
                    qTotalData.push(data.qDataPages[0].qMatrix)  
                }  
          } 

Cheers,
Patric

[Alexander_Thor]

Ye it's a tricky one, in the deep internals there is some websocket throttling and also some internal caching of data that happens when you use the backendApi that is not able to keep up when requesting a large number of data pages in parallel fashion.

When you are using the the method on the model directly then you are bypassing that and sub-sequentially the backendApi might be a bit wonky to rely on.

You could turn of the caching mechanism, it's not supported but seems to work fine

Here is an example GoogleMaps-Sense/GoogleMaps-Sense.js at master · mindspank/GoogleMaps-Sense · GitHub

[pamaxeed]

Hi Alexandre,

I just wondering why it is still so slow.
To fetch ca. 160.000 rows it takes approximately 32 seconds.

What is wrong in the implementation? How standard qliksense objects are fetching the data?

Kind regards,

Patric

[Alexander_Thor]

None of Qlik's visualization fetches all of the data up-front but instead pages in additional data as needed when you scroll of zoom in different objects.

Sending that much data over the wire is probably kinda slow, you probably want to implement some kind of gradual rendering of whatever you are doing. A lot of the metadata is already provided up-front for you to calculate min/max and labels.

[Anonymous]

Function paint only have access to the backendApi, I do not have access to the model. Can i access the model in any other way?

[ErikWetterberg]

Hi,

You'll find it at this.backendApi.model.

Erik Wetterberg

[Alexander_Thor]

So the reason that the getData method on the BackendApi semi-failed under async operations was due to us having an internal cache and throttle to avoid congestion on the websocket pipe.

However since Feb 2018 we did introduce a way to turn this cache off which would allow you fetch multiple pages via the backendapi in a async way.

https://help.qlik.com/en-US/sense-developer/April2018/Subsystems/APIs/Content/BackendAPI/setCacheOpt...