idea Re: Batch process staging tables in Snowflake in Suggest an Idea

Batch process staging tables in Snowflake

tdphill1064 — Mon, 26 Sep 2022 13:00:25 GMT

Instead of doing loads in Snowflake during Batch process where the Snowflake table is zeroed out to have no records. Create a staging table to pump the data in then do the switch of the main table and staging table to limit the downtime of the loading process. It allows for users to access the old data until the new data is added.

Re: Batch process staging tables in Snowflake

Jesse_W — Thu, 14 Jul 2022 18:25:42 GMT

This would be very useful for larger tables that we are loading. It would be a good option on the loads to use a staging table or not.

Re: Batch process staging tables in Snowflake

bobvecchione — Tue, 19 Jul 2022 13:13:37 GMT

Hi Folks,

Thanks for this request. I can share that we are making a number of Snowflake target improvements with Replicate and will consider this as a part of that list.

Once we know more around timing/dates...I will share here.

--bobv--

Re: Batch process staging tables in Snowflake

tdphill1064 — Thu, 21 Jul 2022 17:23:58 GMT

Thanks for the reply. It will be excellent if they can get this added.

Re: Batch process staging tables in Snowflake - Status changed to: Open - New

Nulee_Massaro — Tue, 30 Aug 2022 11:39:56 GMT

Thank you for reaching out. Can you please elaborate on the issue you are having with Snowflake? What is your main concern?

Re: Batch process staging tables in Snowflake

tdphill1064 — Wed, 28 Sep 2022 20:31:47 GMT

The main concern is that when you do a batch load Qlik zeros out the table requested when doing a full load. I have seen other products use a method of having a stage table and letting the main table still contain all the data. Once the full load is done in the staging table the replication software switches the staging table to the main table naming convention and instead of having 30 minutes to hours for the load to complete there is an outage only on the switching of the table names which happens in milliseconds.

Re: Batch process staging tables in Snowflake

AlexOmetis — Thu, 29 Sep 2022 09:40:15 GMT

Yes, this would be really useful to limit the impact of long-running full loads - either when pushing DDL changes that haven't been auto-processed, as part of recovering from an issue or where change data capture isn't available.

From now on, please track this idea from the Ideation por...

Meghann_MacDonald — Mon, 18 Sep 2023 16:58:43 GMT

From now on, please track this idea from the Ideation portal.

Link to new idea

Meghann

NOTE: Upon clicking this link 2 tabs may open - please feel free to close the one with a login page. If you only see 1 tab with the login page, please try clicking this link first: Authenticate me! then try the link above again. Ensure pop-up blocker is off.

Re: Batch process staging tables in Snowflake - Status changed to: Closed - Archived

Ideation — Mon, 18 Sep 2023 16:58:44 GMT