Suggest an Idea

[tjgarza]

The Databricks Runtime 7.3 is the latest runtime with Long Term Support.  It comes with Spark 3.0.1 and has reduced a lot of overhead time from queries.   In the documentation, it looks like only 5.x-6.x is supported and there may be some minor changes to support 7.3.  Can you please add this as a supported version?

[Ola_Mayer]

Included in Nov release

[tjgarza]

Thanks @Ola_Mayer !  From the documentation in the preview the Databricks endpoint only shows 5.x and 6.x while the new Delta connector show 7.x.  

Would the recommendation be to switch to the Delta connector?  And is the non-Delta version going to continue to be deprecated at some point?  

Microsoft Azure Databricks | 5.x and 6.x | ✔ |✔
Microsoft Azure Databricks Delta |7.x | ✔ | ✔

[Ola_Mayer]

Databricks connector is a Data lake implementation, while Databricks Delta is a cloud data warehouse.

Those are used to support different use cases.

The first one is landing data for in a data lake for additional processing primarily my Compose.

The second one is real time ODS generation.

[tjgarza]

Thanks @Ola_Mayer , we are using Compose now to build the Databricks Delta tables, so we will look into switching to using the Delta endpoint directly.

In the meantime, with the Databricks  (non-Delta) connector, there is still the required connection to a Databricks cluster, was it just a misprint that this doesn't show 7.x?  

[Ola_Mayer]

I'm checking this right now

[Ola_Mayer]

Just got answer back from R&D, it is a documentation bug and should show 7.x.

Just one comment, any questions related to Tech Preview version, should be submitted under Tech Preview section and not the regular Idiation.

Thank you in advance

Ola Mayer