Skip to main content

Suggest an Idea

Vote for your favorite Qlik product ideas and add your own suggestions.

Announcements
QlikWorld 2023, a live, in-person thrill ride. Save $300 before February 6: REGISTER NOW!

Ability for Compose to start a databricks cluster prior to scheduled run

lpmarino
Contributor III
Contributor III

Ability for Compose to start a databricks cluster prior to scheduled run

We have a CDC task in C4DL for a databricks project that is scheduled to run daily at 4am. At 4am the task fails due to Cluster being unavailable. We are requesting that compose start up the cluster and then begin processing the CDC task instead of failing. This is similar to how replicate task with databricks target starts the cluster when changes are detected in source system that need to be applied to the target.

2 Comments
TimGarrod
Employee
Employee

Hi @lpmarino  - I'm closing this request because this ability is already supported via databricks clusters starting up on connection attempts and retry mechanisms in Compose. 

If you have a databricks cluster shutdown and start a Qlik Compose job - the job connecting to databricks will auto-start the cluster.  While the cluster startup may cause a timeout in Compose (typically it won't) , the ability to  configure RETRY in Compose supports retrying the timed-out operation.  

I have tested this in an environment and kicking off the Compose job will auto-start the databricks cluster.

If I have mis-understood the request at all , please comment. 

Retry can be found un the project settings and "Environment" tab - 

TimGarrod_0-1642540702350.png

 

Status changed to: Closed - Declined
TimGarrod
Employee
Employee
 
Status changed to: Closed - Already Available