Skip to main content
Announcements
NEW: Seamless Public Data Sharing with Qlik's New Anonymous Access Capability: TELL ME MORE!
cancel
Showing results for 
Search instead for 
Did you mean: 
lguevara
Partner - Contributor III
Partner - Contributor III

Qlik Compose ETL Task can be partitioned?

When we have a lot of data, what characteristic have ETL Task in Qlik Compose to improve this movement of data, for example can be partionated.

Labels (3)
1 Solution

Accepted Solutions
Nanda_Ravindra
Support
Support

@lguevara ETL tasks partitioning is not something which is available in Qlik Compose for Data warehouse. But if you are asking how to improve the ETL performance, then  we will have to see the ETL log and understand where the bottleneck is and based on that we can make recommendation.

Note: The ETL performance has been significantly improved in Qlik Compose 2022.5 and higher for some of the endpoints like Synapse and snowflake.

Below are some guide lines:

1. After the Full load, consider running the index maintenance\stats updates. 

2, Make sure there are no unmapped columns in Data warehouse table in mapping section.

3. Check with your DBA's to see if the compose user has the ability to make 10 connections to the database - this is our default setting. If possible have them increase the limit to the user to 20 and change the 'Maximum number of database connections' setting in the ETL to match this number.

4. Get a powerful box

5. Check with your DBA to see why the statement is taking long to complete the transaction - your DBA will have to run the explain plan to see where the issue is

6. If the full load is taking longer, change the ETL setting to "Optimize for an initial load", please make sure to uncheck the box after the initial load task completes and regenerate the ETL set.

Hope this helps!

Thanks,

Nanda

View solution in original post

1 Reply
Nanda_Ravindra
Support
Support

@lguevara ETL tasks partitioning is not something which is available in Qlik Compose for Data warehouse. But if you are asking how to improve the ETL performance, then  we will have to see the ETL log and understand where the bottleneck is and based on that we can make recommendation.

Note: The ETL performance has been significantly improved in Qlik Compose 2022.5 and higher for some of the endpoints like Synapse and snowflake.

Below are some guide lines:

1. After the Full load, consider running the index maintenance\stats updates. 

2, Make sure there are no unmapped columns in Data warehouse table in mapping section.

3. Check with your DBA's to see if the compose user has the ability to make 10 connections to the database - this is our default setting. If possible have them increase the limit to the user to 20 and change the 'Maximum number of database connections' setting in the ETL to match this number.

4. Get a powerful box

5. Check with your DBA to see why the statement is taking long to complete the transaction - your DBA will have to run the explain plan to see where the issue is

6. If the full load is taking longer, change the ETL setting to "Optimize for an initial load", please make sure to uncheck the box after the initial load task completes and regenerate the ETL set.

Hope this helps!

Thanks,

Nanda