Use lower cost Databricks "Job Compute" rather than All-Purpose clusters
Databrick's DBU pricing is tied in part to the compute type you use. All-purpose compute is the most expensive. The least expensive is job compute. Please offer the ability to utilize job compute to run Compose jobs. On the Premium platform, it could reduce customer's DBU costs by almost half!
Thank you for the submission. We do have a feature on the roadmap for Replicate (Delta and Non-Delta endpoints) and Compose to support lower cost compute on databricks.
Since Compose and Replicate both execute sparkSQL against databricks, the current plan is to support databricks SQL endpoints as the compute layer.
SQL endpoints also provide a big reduction in cost on databricks as well as upcoming features like their Unity catalog for fine-grained access control.
