Azure Databricks 3.5 LTS Runtime end of support

[TalendSolutionExpert]

Problem Description

After Azure Databricks announced the end of support (EOS) for Databricks Runtime 3.5 LTS, you cannot create clusters with this Runtime version anymore. The Databricks cluster page displays a warning message for the existing clusters that use this version:

"This cluster's runtime version is out of support. Select a new runtime and restart."

Talend 7.1.1 only support Databricks Runtime 3.5 LTS, so when you try to run Spark Jobs from this version of Studio in your existing 3.5 Databricks Runtime, if fails with the following error message:

"org.talend.bigdata.launcher.utils.BigDataLauncherException: {"error_code":"MALFORMED_REQUEST","message":"Invalid JSON given in the body of the request - failed to parse given JSON"}"

Root Cause

You are using an EOS version of Databricks.

Solution

Because there isn't any backport for recent Databricks distributions to lower versions, Talend recommends upgrading to 7.3. You can still use Runtime v5.5 LTS in Talend Studio v7.2.1 for Spark batch by using the TPS-3243 patch. Contact Talend Support to request the patch.

Workaround

As an option for Talend 7.1.1, you can create a 5.5 Runtime on the Databricks side and run the Jobs from 7.1.1 on that after selecting 3.5 LTS Runtime in the Spark configuration. However, this approach is not supported.

Warnings

The workaround suggested above was tested internally for one of the Databricks endpoints (for example, West Europe). The results may vary with different regions. For more information see, the Azure Products available by region page.