Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

[amshirkh]

Hi,

Getting the following error intermittently and task stops running:

Error forwarding bulk
Failed to create net changes table for bulk apply
Execute create net changes table statement failed, statement CREATE TABLE `qlik`.`attrep_changes7618E978E2D9CCF0` ( `seq` INT NOT NULL, `col1` STRING, `col2` STRING, `col3` STRING, `col4` STRING, `col5` STRING, `col6` STRING, `col7` STRING, `col8` STRING, `col9` STRING, `col10` STRING, `col11` STRING, `col12` STRING, `col13` STRING, `col14` STRING, `col15` STRING, `col16` STRING, `col17` STRING, `col18` STRING, `col19` STRING, `col20` STRING, `col21` STRING, `col22` STRING, `col23` STRING, `col24` STRING, `col25` STRING, `col26` STRING, `col27` STRING, `col28` STRING, `col29` STRING, `col30` STRING, `col31` STRING, `col32` STRING, `col33` STRING, `col34` STRING, `col35` STRING, `col36` STRING, `col37` STRING, `col38` STRING, `col39` STRING, `col40` STRING, `col41` STRING, `col42` STRING, `col43` STRING, `col44` STRING, `col45` STRING, `col46` STRING, `col47` STRING, `col48` STRING, `col49` STRING, `col50` STRING, `col51` STRING, `col52` STRING, `col53` STRING, `col54` STRING, `col55` STRING, `col56` STRING, `col57` STRING, `col58` STRING, `col59` STRING, `col60` STRING, `col61` STRING, `col62` STRING, `col63` STRING, `col64` STRING, `col65` STRING, `col66` STRING, `col67` STRING, `col68` STRING, `col69` STRING, `col70` STRING, `col71` STRING, `col72` STRING, `col73` STRING, `col74` STRING, `col75` STRING, `col76` STRING, `col77` STRING, `col78` STRING, `col79` STRING, `col80` STRING, `col81` STRING, `col82` STRING, `col83` STRING, `col84` STRING, `col85` STRING, `col86` STRING, `col87` STRING, `col88` STRING, `col89` STRING, `col90` STRING, `col91` STRING, `col92` STRING, `col93` STRING, `col94` STRING, `col95` STRING, `col96` STRING, `col97` STRING, `col98` STRING, `col99` STRING, `col100` STRING, `col101` STRING, `col102` STRING, `col103` STRING, `col104` STRING, `col105` STRING, `col106` STRING, `col107` STRING, `col108` STRING, `col109` STRING, `col110` STRING, `col111` STRING, `col112` STRING, `col113` STRING, `col114` STRING, `col115` STRING, `col116` STRING, `col117` STRING, `col118` STRING, `col119` STRING, `col120` STRING, `col121` STRING, `col122` STRING, `col123` STRING, `col124` STRING, `col125` STRING, `col126` STRING, `col127` STRING, `col128` STRING, `col129` STRING, `col130` STRING, `col131` STRING, `col132` STRING, `col133` STRING, `col134` STRING, `col135` STRING, `col136` STRING, `col137` STRING, `col138` STRING, `col139` STRING, `col140` STRING, `col141` STRING, `col142` STRING, `col143` STRING, `col144` STRING, `col145` STRING, `col146` STRING, `col147` STRING, `col148` STRING, `col149` STRING, `col150` STRING, `col151` STRING, `col152` STRING, `col153` STRING, `col154` STRING, `col155` STRING, `col156` STRING, `col157` STRING, `col158` STRING, `col159` STRING, `col160` STRING, `col161` STRING, `col162` STRING, `col163` STRING, `col164` STRING, `col165` STRING, `col166` STRING, `col167` STRING, `col168` STRING, `col169` STRING, `col170` STRING, `col171` STRING, `col172` STRING, `col173` STRING, `col174` STRING, `col175` STRING, `col176` STRING, `col177` STRING, `col178` STRING, `col179` STRING, `col180` STRING, `col181` STRING, `col182` STRING, `col183` STRING, `col184` STRING, `col185` STRING, `col186` STRING, `col187` STRING, `col188` STRING, `col189` STRING, `col190` STRING, `col191` STRING, `col192` STRING, `col193` STRING, `col194` STRING, `col195` STRING, `col196` STRING, `col197` STRING, `col198` STRING, `col199` STRING, `col200` STRING, `col201` STRING, `col202` STRING, `col203` STRING, `col204` STRING, `col205` STRING, `col206` STRING, `col207` STRING, `col208` STRING, `col209` STRING, `col210` STRING, `col211` STRING, `col212` STRING, `col213` STRING, `col214` STRING, `col215` STRING, `col216` STRING, `col217` STRING, `col218` STRING, `col219` STRING, `col220` STRING, `col221` STRING, `col222` STRING, `col223` STRING, `col224` STRING, `col225` STRING, `col226` STRING, `col227` STRING, `col228` STRING, `col229` STRING, `col230` STRING, `col231` STRING, `col232` STRING, `col233` STRING, `col234` STRING, `col235` STRING, `col236` STRING, `col237` STRING, `col238` STRING, `col239` STRING, `col240` STRING, `col241` STRING, `col242` STRING, `col243` STRING, `col244` STRING, `col245` STRING, `col246` STRING, `seg1` STRING ) USING csv options('multiLine'='true','FIELDDELIM'=',','nullValue'='attrep_null') LOCATION 'abfss://qlik@dlsmbtadmapnonprod.dfs.core.windows.net/qlik-db-staging/attrep_changes7618E978E2D9CCF0'
RetCode: SQL_ERROR SqlState: HY000 NativeError: 35 Message: [Simba][Hardy] (35) Error from server: error code: '0' error message: 'org.apache.hive.service.cli.HiveSQLException: Error running query: Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:863)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:637)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:144)
at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:91)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:53)
at org.apache.spark.sql.hi
Failed (retcode -1) to execute statement: 'CREATE TABLE `qlik`.`attrep_changes7618E978E2D9CCF0` ( `seq` INT NOT NULL, `col1` STRING, `col2` STRING, `col3` STRING, `col4` STRING, `col5` STRING, `col6` STRING, `col7` STRING, `col8` STRING, `col9` STRING, `col10` STRING, `col11` STRING, `col12` STRING, `col13` STRING, `col14` STRING, `col15` STRING, `col16` STRING, `col17` STRING, `col18` STRING, `col19` STRING, `col20` STRING, `col21` STRING, `col22` STRING, `col23` STRING, `col24` STRING, `col25` STRING, `col26` STRING, `col27` STRING, `col28` STRING, `col29` STRING, `col30` STRING, `col31` STRING, `col32` STRING, `col33` STRING, `col34` STRING, `col35` STRING, `col36` STRING, `col37` STRING, `col38` STRING, `col39` STRING, `col40` STRING, `col41` STRING, `col42` STRING, `col43` STRING, `col44` STRING, `col45` STRING, `col46` STRING, `col47` STRING, `col48` STRING, `col49` STRING, `col50` STRING, `col51` STRING, `col52` STRING, `col53` STRING, `col54` STRING, `col55` STRING, `col56` STRING, `col57` STRING, `col58` STRING, `col59` STRING, `col60` STRING, `col61` STRING, `col62` STRING, `col63` STRING, `col64` STRING, `col65` STRING, `col66` STRING, `col67` STRING, `col68` STRING, `col69` STRING, `col70` STRING, `col71` STRING, `col72` STRING, `col73` STRING, `col74` STRING, `col75` STRING, `col76` STRING, `col77` STRING, `col78` STRING, `col79` STRING, `col80` STRING, `col81` STRING, `col82` STRING, `col83` STRING, `col84` STRING, `col85` STRING, `col86` STRING, `col87` STRING, `col88` STRING, `col89` STRING, `col90` STRING, `col91` STRING, `col92` STRING, `col93` STRING, `col94` STRING, `col95` STRING, `col96` STRING, `col97` STRING, `col98` STRING, `col99` STRING, `col100` STRING, `col101` STRING, `col102` STRING, `col103` STRING, `col104` STRING, `col105` STRING, `col106` STRING, `col107` STRING, `col108` STRING, `col109` STRING, `col110` STRING, `col111` STRING, `col112` STRING, `col113` STRING, `col114` STRING, `col115` STRING, `col116` STRING, `col117` STRING, `col118` STRING, `col119` STRING, `col120` STRING, `col121` STRING, `col122` STRING, `col123` STRING, `col124` STRING, `col125` STRING, `col126` STRING, `col127` STRING, `col128` STRING, `col129` STRING, `col130` STRING, `col131` STRING, `col132` STRING, `col133` STRING, `col134` STRING, `col135` STRING, `col136` STRING, `col137` STRING, `col138` STRING, `col139` STRING, `col140` STRING, `col141` STRING, `col142` STRING, `col143` STRING, `col144` STRING, `col145` STRING, `col146` STRING, `col147` STRING, `col148` STRING, `col149` STRING, `col150` STRING, `col151` STRING, `col152` STRING, `col153` STRING, `col154` STRING, `col155` STRING, `col156` STRING, `col157` STRING, `col158` STRING, `col159` STRING, `col160` STRING, `col161` STRING, `col162` STRING, `col163` STRING, `col164` STRING, `col165` STRING, `col166` STRING, `col167` STRING, `col168` STRING, `col169` STRING, `col170` STRING, `col171` STRING, `col172` STRING, `col173` STRING, `col174` STRING, `col175` STRING, `col176` STRING, `col177` STRING, `col178` STRING, `col179` STRING, `col180` STRING, `col181` STRING, `col182` STRING, `col183` STRING, `col1...'

Setup:

Windows Server 2019 Datacenter 64 bit , 32 GB memory

Source: Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

Target: Azure Databricks Delta 

All prerequisites documented in this link are followed:

https://help.qlik.com/en-US/replicate/May2021/Content/Replicate/Main/Databricks%20Delta/databricks_d...

see attached for detailed logs.

[KellyHobson]

Hey @amshirkh ,

I wanted to follow up on this one (better late than never :)) in case others run into this issue. 

From another customer case with same "RetCode: SQL_ERROR SqlState: HY000 NativeError: 35 Message: [Simba][Hardy] (35).....Error running query: Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key" message. 

This requirement was missing on the client side

To be able to access the storage directories from the Databricks cluster, users need to add a configuration (in Spark Config) for that Storage Account and its key.
Example:
fs.azure.account.key.<storage-account-name>.dfs.core.windows.net <storage-account-access-key>

Best,

Kelly 

[lyka]

Hey @amshirkh 

Please let us know if the proposed solution worked.

Thanks

Lyka

[bnok]

We have the same issue and have verified that the spark config has the correct key for the storage account but the error continues to persist.

[bnok]

Are there other alternatives that could be causing "NativeError: 35 Message: [Simba][Hardy] (35).....Error running query: Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key" ?

[amshirkh · Accepted solution]

The issue for my project turned out to be extra end of line characters in some String/text types at the source. ( Oracle ). A simple transformation rule in the Qlik task (  remove the end of line character ) fixed the issue. 

The key error was misleading.

[Steve_Nguyen]

@amshirkh

thanks for the update.