topic Impact of Large Dataset with Group by in Load Statement in App Development

Impact of Large Dataset with Group by in Load Statement

Tony — Sat, 16 Nov 2024 06:18:25 GMT

Hello,

While loading a dataset of 100 million records consisting of 5 columns, (one column is doing a Max aggregation function, and the other 4 columns are part of a Group by clause,) the computer crashes. No blue screen. When It crashed with the Group by the whole screen just froze completely, and then it rebooted.

Performing a load without the Group by, then using a Resident load with a Group by, all window objects failed. The desktop just died. I had to force a restart by holding down the power button for 30 seconds. But never had the blue screen in any of this.

Any thoughts on limitations with the Group by that may be contributing to this issue would be most appreciated.

Thank you,

Tony

Re: Impact of Large Dataset with Group by in Load Statement

dplr-rn — Mon, 18 Mar 2019 16:10:01 GMT

Group by is more expensive. But i have not heard of system crashes

You can optimize group by sorting the data (order by) by the same group and then applying group by. check below

https://community.qlik.com/t5/QlikView-Documents/Optimize-Group-By-Performance/tac-p/1481478

hope it helps

Re: Impact of Large Dataset with Group by in Load Statement

Tony — Mon, 18 Mar 2019 16:15:34 GMT

Thank you, dilipranjith, for your very helpful reply. I'll try out your suggestion,

Tony