topic Re: Trigger a Databricks Notebook from Talend in Talend Studio

Trigger a Databricks Notebook from Talend

rbk_tal — Sat, 16 Nov 2024 02:39:40 GMT

Hello everyone,

We are currently exploring the usage of DataBricks and integrating it with Talend.

While we have find the existing integration very useful, we have a requirement to trigger a specific Notebook within Databricks once our job is complete. If possible pass parameters to the notebook which is currently obtained via widget within databricks.

Is it possible to trigger a Databricks notebook from Talend and pass the parameters to the widgets available there ? Kindly let me know for any further information. Thanks.

Re: Trigger a Databricks Notebook from Talend

Anonymous — Tue, 21 Apr 2020 10:09:17 GMT

Hi rbk_tal
If the notebook can be triggered by job API, then you can use tRest or tRestClient component in Talend to call the API.
https://docs.databricks.com/jobs.html

Regards
Shong

Re: Trigger a Databricks Notebook from Talend

rbk_tal — Sun, 21 Jun 2020 16:24:29 GMT

Hi Shong,

I thank you for your response and I was able to connect to the Cluster using the RestClient. We have several APIs we can use. In case of someone else comes across this, we were able to find a couple of ways to connect too.

1. Best way - Suggested by Shong - RestClient - Databricks API.

Other options - to interact with data in Databricks and not related to original question of triggering a notebook.

1. Big Data Batch Job - you can directly connect to Databricks. I could not find the Dynamic schema option and debugging was a little tough.

2. There were a couple of other JDBC connection options available, not sure how safe they are though.

Best to try out and see what works out.

thank you once again Shong.

Re: Trigger a Databricks Notebook from Talend

Anonymous — Mon, 22 Jun 2020 01:38:37 GMT

great, thank you for your feedback!