Qlik Community

Ideas

Vote for your favorite Qlik product ideas and add your own suggestions.

Announcements
Coming your way, the Qlik Data Revolution Virtual Summit. October 27-29. REGISTER

Partial Reload as a standard option for a QMC Task

Luminary
Luminary

Partial Reload as a standard option for a QMC Task

The partial reload would be very beneficial for working with IoT or online data.

It enables to have almost real-time data in the user app without the need to reload the whole data model => requires fewer resources.

It can already be triggered with API, but should definitely be an option for a Task Trigger in QMC.

Along with this, there should also be an option to set a timeframe when an hourly Task will NOT be executed (skipped). For example, many companies run consolidation and cleanup procedures on their databases during the night, so it does not make sense to reload data in specific hours.

In the context of a partial reload that runs every minute, there needs to be a possibility to postpone a partial reload and run a full reload for the rest of the data model.

4 Comments
Luminary
Luminary

I'd even love to have an option to decide in script and do either full or partial reload based on conditions.

Luminary
Luminary

Yes! That is an excellent idea.

There is an IsPartialReload() system function, so why not ReloadMode() function or variable, similar to how ErrorMode works.

Employee
Employee

Hello @RadOresky,

Thanks for your idea. Going forward please use one, and only one, label per post. Please see the submission guidelines for more details.

Luminary
Luminary

Hello 

I fully agree. Partial reload seems really a good way to reach 'low latency' dashboard without having to burn CPU by reloading the same data all the time. Currently it seems limited to user session , is "difficult" to activate and requires good understanding of the ADD / REPLACE / IsPartialReload() statements to function correctly.

However, I do not the scheduler as the best spot for this feature. Indeed, there are other techniques like application pipelines and QVD to optimize data load in the scheduler and thus reduce load duration. The scheduler role is not to deliver high frequency reload but more to maintain data consistency between all user sessions even in case of reload failure. 

I see a bigger interest directly in the dashboard to allow user to update the dashboard very rapidly without impacting other users (so limited to user sessions). As Qlik Sense engine is taking care of refreshing transparently charts anytime the application data are modified, it should definitely be possible to create an automatic refresh / reload mechanism based on partial reload to limit CPU impact... 

So I did some experiments to evaluate the potential 

First thing is that it is possible to activate it via a simple widget ( https://www.youtube.com/watch?v=_fgUBRndn30 ) but it still requires user action. 

Second thing was then to build a simple extension with an ON/OFF button and a timer to activate periodic partial reload . The reload is local to user session only and not STORE is possible, but this can be used to query new data from a rest API and update dashboards accordingly . 

 

In my example , the refresh period is decided at development time and cannot be modified after app is published, and of course , it require a specific security rule so that users get the right to launch the reload.  

This can be combined with scheduler driven reload to align periodically the different users sessions.  

If interested, I imagine I can share the extension code , but without any warranty of its value 🙂 

Regards

Thomas