File to Microsoft Azure Databricks Delta Error when copy data

[lguevara]

Hi,

I configure my source as file and my target as Azure Databricks Delta; and test connection works fine.

When run task, create the table in databricks but no insert data.

Error

00011360: 2023-02-15T17:26:14 [TARGET_LOAD ]E: Failed (retcode -1) to execute statement: 'COPY INTO `default`.`regions` FROM(SELECT cast(_c0 as TINYINT) as `ID`, _c1 as `NAME` from 'abfss://myfilesystem1@datalake123134.dfs.core.windows.net') FILEFORMAT = CSV FILES = ('/staging1/File to Databricks/1/LOAD00000001.csv.gz') FORMAT_OPTIONS('nullValue' = 'attrep_null', 'multiLine'='true') COPY_OPTIONS('force' = 'true')' [1022502] (ar_odbc_stmt.c:4985)
00011360: 2023-02-15T17:26:14 [TARGET_LOAD ]E: RetCode: SQL_ERROR SqlState: 42000 NativeError: 80 Message: [Simba][Hardy] (80) Syntax or semantic analysis error thrown in server while executing query. Error message from server: org.apache.hive.service.cli.HiveSQLException: Error running query: Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key
at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:53)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:435)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:257)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties.withLocalProperties(ThriftLocalProperties.scala:123)
at org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties.withLocalProperties [1022502] (ar_odbc_stmt.c:4992)

Thanks

[kmadhavi2]

Hi,

Please cross check the value for fs.azure.account.key

Thanks,

Madhavi

[SwathiPulagam · Accepted solution]

Hi @lguevara ,

Make sure you added the below key value on databricks  Advanced Options --> spark config
fs.azure.account.key.datalake123134.dfs.core.windows.net xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==

 value you have to collect from your storage account secret.

Let us know how it goes.

Thanks,

Swathi