Trigger a Databricks Notebook from Talend

[rbk_tal]

Hello everyone,

We are currently exploring the usage of DataBricks and integrating it with Talend.

While we have find the existing integration very useful, we have a requirement to trigger a specific Notebook within Databricks once our job is complete. If possible pass parameters to the notebook which is currently obtained via widget within databricks.

Is it possible to trigger a Databricks notebook from Talend and pass the parameters to the widgets available there ? Kindly let me know for any further information. Thanks.

[Anonymous]

Hi rbk_tal
If the notebook can be triggered by job API, then you can use tRest or tRestClient component in Talend to call the API.
https://docs.databricks.com/jobs.html

Regards
Shong

[rbk_tal]

Hi Shong,

I thank you for your response and I was able to connect to the Cluster using the RestClient. We have several APIs we can use. In case of someone else comes across this, we were able to find a couple of ways to connect too. 

1. Best way - Suggested by Shong - RestClient - Databricks API.

Other options - to interact with data in Databricks and not related to original question of triggering a notebook.

1. Big Data Batch Job - you can directly connect to Databricks. I could not find the Dynamic schema option and debugging was a little tough.

2. There were a couple of other JDBC connection options available, not sure how safe they are though.

Best to try out and see what works out.

thank you once again Shong.

[Anonymous]

great, thank you for your feedback!