Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
When we have a lot of data, what characteristic have ETL Task in Qlik Compose to improve this movement of data, for example can be partionated.
@lguevara ETL tasks partitioning is not something which is available in Qlik Compose for Data warehouse. But if you are asking how to improve the ETL performance, then we will have to see the ETL log and understand where the bottleneck is and based on that we can make recommendation.
Note: The ETL performance has been significantly improved in Qlik Compose 2022.5 and higher for some of the endpoints like Synapse and snowflake.
Below are some guide lines:
1. After the Full load, consider running the index maintenance\stats updates.
2, Make sure there are no unmapped columns in Data warehouse table in mapping section.
3. Check with your DBA's to see if the compose user has the ability to make 10 connections to the database - this is our default setting. If possible have them increase the limit to the user to 20 and change the 'Maximum number of database connections' setting in the ETL to match this number.
4. Get a powerful box
5. Check with your DBA to see why the statement is taking long to complete the transaction - your DBA will have to run the explain plan to see where the issue is
6. If the full load is taking longer, change the ETL setting to "Optimize for an initial load", please make sure to uncheck the box after the initial load task completes and regenerate the ETL set.
Hope this helps!
Thanks,
Nanda
@lguevara ETL tasks partitioning is not something which is available in Qlik Compose for Data warehouse. But if you are asking how to improve the ETL performance, then we will have to see the ETL log and understand where the bottleneck is and based on that we can make recommendation.
Note: The ETL performance has been significantly improved in Qlik Compose 2022.5 and higher for some of the endpoints like Synapse and snowflake.
Below are some guide lines:
1. After the Full load, consider running the index maintenance\stats updates.
2, Make sure there are no unmapped columns in Data warehouse table in mapping section.
3. Check with your DBA's to see if the compose user has the ability to make 10 connections to the database - this is our default setting. If possible have them increase the limit to the user to 20 and change the 'Maximum number of database connections' setting in the ETL to match this number.
4. Get a powerful box
5. Check with your DBA to see why the statement is taking long to complete the transaction - your DBA will have to run the explain plan to see where the issue is
6. If the full load is taking longer, change the ETL setting to "Optimize for an initial load", please make sure to uncheck the box after the initial load task completes and regenerate the ETL set.
Hope this helps!
Thanks,
Nanda