Skip to main content

Suggest an Idea

Vote for your favorite Qlik product ideas and add your own suggestions.

Announcements
This page is no longer in use. To suggest an idea, please visit Browse and Suggest.

Split updates into smaller batches in Qlik Compose for data warehouse on billion row tables

Carlo_P
Contributor III
Contributor III

Split updates into smaller batches in Qlik Compose for data warehouse on billion row tables

We seem to be hitting a resource limit in Snowflake when we are updating or inserting data on tables with billions of rows. When those tables are being updated or inserted to it tries to perform the transaction in one operation and either runs forever or never completes even on a 4XL warehouse. The more inserts or updates the bigger the warehouse size needs to be which makes processing unpredictable. We never know on a given load how many changes will be coming on that day so we cannot dynamically size the warehouse based on load and we cannot leave it on a bigger size in case this happens as it will needlessly burn credits. Snowflake has asked us to process those sequentially to ensure it is not competing for memory with other queries on a 4XL. We would like to have an option to set a limit on the size of updates and inserts and if that limit is reached it should be broken up into smaller batches. Ideally the size threshold limit should be configurable to adjust in the project to be able to adjust based on the warehouse size selected or it could be dynamically adjusted based on the size of the warehouse used. This would allow us to keep processing on a small warehouse size and would make our loads more predictable. We are actively working with Snowflake support on this issue but we think adding this option in compose would allow for more flexibility and control over our workloads and would allow us to significantly reduce our costs.

4 Comments
bobvecchione
Employee
Employee

Hi - Thanks for the request. Do you know if there are more of one operation than another? For example, we know Snowflake costs for the Merge when there are a lot of INSERTS is more expensive that just the INSERT operation itself.

 

We have added a control for this.

 

Let me know if you actually know the breakdown of operations.

 

Thanks

 

Bob

rdewolfe
Contributor
Contributor

This would be very helpful for us using Qlik Replicate .  We have Azure SQL endpoints where the connection times out when performing very large operations, and we have no ability to get the sync to proceed other than reloading the full table.  This appears to be an option for other endpoints but not Azure SQL

 

Meghann_MacDonald

From now on, please track this idea from the Ideation portal. 

Link to new idea

Meghann

NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.

Ideation
Explorer II
Explorer II
 
Status changed to: Closed - Archived