Submit Multiple Big Data batch job on Azure Databricks cluster

[Anonymous]

Hi ,

We have requirement of submitting multiple spark big data batch jobs on single Azure Databricks cluster. We tried submitting more than one job at the same  time to single databricks cluster, but jobs start failing with the issue :

“org.talend.bigdata.launcher.utils.BigDataLauncherException: Status 400 for /api/2.0/clusters/restart”

After some analysis I figured out that as per the talend official documentation, we can’t run multiple big data batch jobs in parallel on single databricks cluster. Please find the link below:

https://help.talend.com/reader/iLJ_d4Eb9ZgJEEFVKwPNig/uG6WXACe~Vz1fYBBPjpnPA

Is there a way I can run big data jobs in parallel on a single databricks cluster?

[Anonymous]

Hello,

Could you please create a new feature jira issue on talend bug tracker?

Best regards

Sabrina