idea Re: Qlik Compose CDC ETL Task/Instructions run Performance with Snowflake as target - Status changed to: Open - On Roadmap in Suggest an Idea

Qlik Compose CDC ETL Task/Instructions run Performance with Snowflake as target

Hrishikesh — Thu, 02 Sep 2021 20:29:31 GMT

Currently we have a Project with 106 Entities and growing with Snowflake as our target database

Whenever we run the CDC ETL task, it runs for several minutes (sometime > 10 mins.).

Our generated ETL set has > 5,000 SQL instructions to run.

We observed that sometimes data gets modified for only a few Source Entities out of total 106, but Compose runs all the 5,000 instructions which in turn causes the ETL task to run several minutes.

We are looking for near real-time data refresh in our Data Warehouse and Data Marts.

Can Compose be smarter to recognize the condition "where count(*) > 0" in __CT tables in landing and only run the corresponding mapping related ETL instructions?

I think that will improve performance a lot instead of "always" running 5000 SQL instructions, as only the ETL for modified entities will be run.

Re: Qlik Compose CDC ETL Task/Instructions run Performance with Snowflake as target - Status changed to: Open - On Roadmap

TimGarrod — Mon, 02 May 2022 22:32:30 GMT

Re: Qlik Compose CDC ETL Task/Instructions run Performance with Snowflake as target

Hrishikesh — Mon, 02 May 2022 22:58:14 GMT

Hi @TimGarrod , saw this idea moving to Open state, when is it targeted to be developed?

Looking forward, as this will be an exciting feature for overall ETL Performance improvement

Re: Qlik Compose CDC ETL Task/Instructions run Performance with Snowflake as target

TimGarrod — Mon, 02 May 2022 23:06:06 GMT

Hi @Hrishikesh , phase 1 of reducing / eliminating execution of some statements based on run-time parameters is undergoing development now. I don't have an explicit date - as it depends on dev / testing etc. However, I will update you here when I know more.

Re: Qlik Compose CDC ETL Task/Instructions run Performance with Snowflake as target

Hrishikesh — Mon, 02 May 2022 23:52:00 GMT

Ok thanks for the update, will this eliminate "Populating staging table..." ETL statements and the staging table data processing steps that follow when there is no landing data available in __CT tables?

Re: Qlik Compose CDC ETL Task/Instructions run Performance with Snowflake as target - Status changed to: Delivered

TimGarrod — Fri, 30 Dec 2022 16:58:15 GMT

Compose November 2022 SR (a patch release ontop of 2022.05) has additional filtering of ETL instructions to reduce the churn when no data is present to be processed.