Suggest an Idea

[ubanerjee]

When loading a few tables in our environment, snowflake WH spills data over to remote storage. These tables could benefit from using a larger warehouse. However, changing the default warehouse for every job seems wasteful, and increases the cost more than necessary. 

ability to override default warehouse by mappings or facts can help make jobs more efficient and cost effective.

[TimGarrod]

Hi @ubanerjee  - have you considered using Pre-Load and Post-Load features in Compose to adjust the WH size when these mappings run / execute ?    

They can be assigned to the task (so re-used across tasks) in order to adjust the WH Size.  (I realize this is not the same as assigning a different warehouse to the specific mapping  - however Compose is built to run statements in parallel based on number of connections you assign which means statements for a single table are not always assigned the same session in snowflake to ensure we can execute the statement as quickly as possible  when a new slot becomes available within the workflow). 

[ubanerjee]

@TimGarrod Thanks for the suggestion, we have implemented the solution and it works as expected for the most part and is good for our use case.

When there are other Pre-Load ETLs that require the larger warehouse this solution requires us to run a dummy ETL set with the Alter statements  prior to running the actual ETL set. Not ideal, but works. 

[Meghann_MacDonald]

From now on, please track this idea from the Ideation portal. 

Link to new idea

Meghann

NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.