Ability for Compose to start a databricks cluster prior to scheduled run
We have a CDC task in C4DL for a databricks project that is scheduled to run daily at 4am. At 4am the task fails due to Cluster being unavailable. We are requesting that compose start up the cluster and then begin processing the CDC task instead of failing. This is similar to how replicate task with databricks target starts the cluster when changes are detected in source system that need to be applied to the target.
Hi @lpmarino - I'm closing this request because this ability is already supported via databricks clusters starting up on connection attempts and retry mechanisms in Compose.
If you have a databricks cluster shutdown and start a Qlik Compose job - the job connecting to databricks will auto-start the cluster. While the cluster startup may cause a timeout in Compose (typically it won't) , the ability to configure RETRY in Compose supports retrying the timed-out operation.
I have tested this in an environment and kicking off the Compose job will auto-start the databricks cluster.
If I have mis-understood the request at all , please comment.
Retry can be found un the project settings and "Environment" tab -
