Qlik Script: Multi-Threaded GROUP BY, especially Sum()
We need Multi-Threaded GROUP BY in script, especially Sum() - which is basic to implement with multi-threading.
We have many Customers with millions of rows. They have to have some aggregations in dedicated, small and quick apps. They have fast cpus with many cores, for example 48 physical cores. But GROUP BY in script is single-threaded...
What takes seconds in visualization, it takes hours in script!
This feature would great improve the scalability of SaaS for an organization like mine that has large data volumes and we probably aren't even close to being the biggest. Strongly support adding this capability.
It definitely is unintuitive that Qlik is so quick at aggregation in the front end but so slow at it in the back end. I'd definitely like for more multi-threading support for sure.
I have found some ways to work around this a little though - e.g. don't use group by with a where clause - you're better off loading the data with a where clause and then doing group by as a resident load. Some of those findings are written up in a blog I wrote a few years ago... A Journey In Load Optimisation - Ometis
NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.
