Talend, CSV, Blob Storage, Spark and Databricks

zwheeler — Sat, 16 Nov 2024 03:26:46 GMT

I am new to talend so sorry for the long title.

1. created talend job, adds data to blob storage as CSV --- SUCCESS

2. Read CSV file from blobl storage to tLog -- SUCCESS

3. Read CSV file and turn into parquet file using tfileoutputparquet -- NOT WORKING

OBJECTIVE -- Turn CSV into Parquet File in Azure Blob Storage then read Parquet from Databricks

4. Set up AzureFSConfiguration Component - SUCCESS

5. Started Cluster, Got Cluster Id, Created Token - SUCCESS

6. Set up Spark Configuration on TALEND - COMPLETED

Question- On walkthrough

It configures data lake in the Spark Cluster, do i have to configure this for blob storage ?

If so what are the correct or proper credentials to set and add.

Currently, we have no configuration on the actual spark cluster under Advanced Option

We run talend job and get the following error:

[ERROR]: org.talend.bigdata.launcher.databricks.DatabricksJob - Run terminated with status TERMINATED and result FAILED

We import and add the component talend-bigdata-launcher-platform-1.2.7-20180828.jar

and get the following error when we run talend

The method withProductionMode(boolean) is undefined for the type SparkBatchJob.Builder

YOUR assistance is greatly appreciated, as we havebeen trying to complete this for a week with no success and need to show progrees by tomorrow COB.

topic Talend, CSV, Blob Storage, Spark and Databricks in Talend Studio