Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

amshirkh — Thu, 14 Oct 2021 08:28:40 GMT

Hi,

Getting the following error intermittently and task stops running:

Error forwarding bulk
Failed to create net changes table for bulk apply
Execute create net changes table statement failed, statement CREATE TABLE `qlik`.`attrep_changes7618E978E2D9CCF0` ( `seq` INT NOT NULL, `col1` STRING, `col2` STRING, `col3` STRING, `col4` STRING, `col5` STRING, `col6` STRING, `col7` STRING, `col8` STRING, `col9` STRING, `col10` STRING, `col11` STRING, `col12` STRING, `col13` STRING, `col14` STRING, `col15` STRING, `col16` STRING, `col17` STRING, `col18` STRING, `col19` STRING, `col20` STRING, `col21` STRING, `col22` STRING, `col23` STRING, `col24` STRING, `col25` STRING, `col26` STRING, `col27` STRING, `col28` STRING, `col29` STRING, `col30` STRING, `col31` STRING, `col32` STRING, `col33` STRING, `col34` STRING, `col35` STRING, `col36` STRING, `col37` STRING, `col38` STRING, `col39` STRING, `col40` STRING, `col41` STRING, `col42` STRING, `col43` STRING, `col44` STRING, `col45` STRING, `col46` STRING, `col47` STRING, `col48` STRING, `col49` STRING, `col50` STRING, `col51` STRING, `col52` STRING, `col53` STRING, `col54` STRING, `col55` STRING, `col56` STRING, `col57` STRING, `col58` STRING, `col59` STRING, `col60` STRING, `col61` STRING, `col62` STRING, `col63` STRING, `col64` STRING, `col65` STRING, `col66` STRING, `col67` STRING, `col68` STRING, `col69` STRING, `col70` STRING, `col71` STRING, `col72` STRING, `col73` STRING, `col74` STRING, `col75` STRING, `col76` STRING, `col77` STRING, `col78` STRING, `col79` STRING, `col80` STRING, `col81` STRING, `col82` STRING, `col83` STRING, `col84` STRING, `col85` STRING, `col86` STRING, `col87` STRING, `col88` STRING, `col89` STRING, `col90` STRING, `col91` STRING, `col92` STRING, `col93` STRING, `col94` STRING, `col95` STRING, `col96` STRING, `col97` STRING, `col98` STRING, `col99` STRING, `col100` STRING, `col101` STRING, `col102` STRING, `col103` STRING, `col104` STRING, `col105` STRING, `col106` STRING, `col107` STRING, `col108` STRING, `col109` STRING, `col110` STRING, `col111` STRING, `col112` STRING, `col113` STRING, `col114` STRING, `col115` STRING, `col116` STRING, `col117` STRING, `col118` STRING, `col119` STRING, `col120` STRING, `col121` STRING, `col122` STRING, `col123` STRING, `col124` STRING, `col125` STRING, `col126` STRING, `col127` STRING, `col128` STRING, `col129` STRING, `col130` STRING, `col131` STRING, `col132` STRING, `col133` STRING, `col134` STRING, `col135` STRING, `col136` STRING, `col137` STRING, `col138` STRING, `col139` STRING, `col140` STRING, `col141` STRING, `col142` STRING, `col143` STRING, `col144` STRING, `col145` STRING, `col146` STRING, `col147` STRING, `col148` STRING, `col149` STRING, `col150` STRING, `col151` STRING, `col152` STRING, `col153` STRING, `col154` STRING, `col155` STRING, `col156` STRING, `col157` STRING, `col158` STRING, `col159` STRING, `col160` STRING, `col161` STRING, `col162` STRING, `col163` STRING, `col164` STRING, `col165` STRING, `col166` STRING, `col167` STRING, `col168` STRING, `col169` STRING, `col170` STRING, `col171` STRING, `col172` STRING, `col173` STRING, `col174` STRING, `col175` STRING, `col176` STRING, `col177` STRING, `col178` STRING, `col179` STRING, `col180` STRING, `col181` STRING, `col182` STRING, `col183` STRING, `col184` STRING, `col185` STRING, `col186` STRING, `col187` STRING, `col188` STRING, `col189` STRING, `col190` STRING, `col191` STRING, `col192` STRING, `col193` STRING, `col194` STRING, `col195` STRING, `col196` STRING, `col197` STRING, `col198` STRING, `col199` STRING, `col200` STRING, `col201` STRING, `col202` STRING, `col203` STRING, `col204` STRING, `col205` STRING, `col206` STRING, `col207` STRING, `col208` STRING, `col209` STRING, `col210` STRING, `col211` STRING, `col212` STRING, `col213` STRING, `col214` STRING, `col215` STRING, `col216` STRING, `col217` STRING, `col218` STRING, `col219` STRING, `col220` STRING, `col221` STRING, `col222` STRING, `col223` STRING, `col224` STRING, `col225` STRING, `col226` STRING, `col227` STRING, `col228` STRING, `col229` STRING, `col230` STRING, `col231` STRING, `col232` STRING, `col233` STRING, `col234` STRING, `col235` STRING, `col236` STRING, `col237` STRING, `col238` STRING, `col239` STRING, `col240` STRING, `col241` STRING, `col242` STRING, `col243` STRING, `col244` STRING, `col245` STRING, `col246` STRING, `seg1` STRING ) USING csv options('multiLine'='true','FIELDDELIM'=',','nullValue'='attrep_null') LOCATION 'abfss://qlik@dlsmbtadmapnonprod.dfs.core.windows.net/qlik-db-staging/attrep_changes7618E978E2D9CCF0'
RetCode: SQL_ERROR SqlState: HY000 NativeError: 35 Message: [Simba][Hardy] (35) Error from server: error code: '0' error message: 'org.apache.hive.service.cli.HiveSQLException: Error running query: Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:863)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:637)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties(SparkOperation.scala:144)
at org.apache.spark.sql.hive.thriftserver.SparkOperation.withLocalProperties$(SparkOperation.scala:91)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:53)
at org.apache.spark.sql.hi
Failed (retcode -1) to execute statement: 'CREATE TABLE `qlik`.`attrep_changes7618E978E2D9CCF0` ( `seq` INT NOT NULL, `col1` STRING, `col2` STRING, `col3` STRING, `col4` STRING, `col5` STRING, `col6` STRING, `col7` STRING, `col8` STRING, `col9` STRING, `col10` STRING, `col11` STRING, `col12` STRING, `col13` STRING, `col14` STRING, `col15` STRING, `col16` STRING, `col17` STRING, `col18` STRING, `col19` STRING, `col20` STRING, `col21` STRING, `col22` STRING, `col23` STRING, `col24` STRING, `col25` STRING, `col26` STRING, `col27` STRING, `col28` STRING, `col29` STRING, `col30` STRING, `col31` STRING, `col32` STRING, `col33` STRING, `col34` STRING, `col35` STRING, `col36` STRING, `col37` STRING, `col38` STRING, `col39` STRING, `col40` STRING, `col41` STRING, `col42` STRING, `col43` STRING, `col44` STRING, `col45` STRING, `col46` STRING, `col47` STRING, `col48` STRING, `col49` STRING, `col50` STRING, `col51` STRING, `col52` STRING, `col53` STRING, `col54` STRING, `col55` STRING, `col56` STRING, `col57` STRING, `col58` STRING, `col59` STRING, `col60` STRING, `col61` STRING, `col62` STRING, `col63` STRING, `col64` STRING, `col65` STRING, `col66` STRING, `col67` STRING, `col68` STRING, `col69` STRING, `col70` STRING, `col71` STRING, `col72` STRING, `col73` STRING, `col74` STRING, `col75` STRING, `col76` STRING, `col77` STRING, `col78` STRING, `col79` STRING, `col80` STRING, `col81` STRING, `col82` STRING, `col83` STRING, `col84` STRING, `col85` STRING, `col86` STRING, `col87` STRING, `col88` STRING, `col89` STRING, `col90` STRING, `col91` STRING, `col92` STRING, `col93` STRING, `col94` STRING, `col95` STRING, `col96` STRING, `col97` STRING, `col98` STRING, `col99` STRING, `col100` STRING, `col101` STRING, `col102` STRING, `col103` STRING, `col104` STRING, `col105` STRING, `col106` STRING, `col107` STRING, `col108` STRING, `col109` STRING, `col110` STRING, `col111` STRING, `col112` STRING, `col113` STRING, `col114` STRING, `col115` STRING, `col116` STRING, `col117` STRING, `col118` STRING, `col119` STRING, `col120` STRING, `col121` STRING, `col122` STRING, `col123` STRING, `col124` STRING, `col125` STRING, `col126` STRING, `col127` STRING, `col128` STRING, `col129` STRING, `col130` STRING, `col131` STRING, `col132` STRING, `col133` STRING, `col134` STRING, `col135` STRING, `col136` STRING, `col137` STRING, `col138` STRING, `col139` STRING, `col140` STRING, `col141` STRING, `col142` STRING, `col143` STRING, `col144` STRING, `col145` STRING, `col146` STRING, `col147` STRING, `col148` STRING, `col149` STRING, `col150` STRING, `col151` STRING, `col152` STRING, `col153` STRING, `col154` STRING, `col155` STRING, `col156` STRING, `col157` STRING, `col158` STRING, `col159` STRING, `col160` STRING, `col161` STRING, `col162` STRING, `col163` STRING, `col164` STRING, `col165` STRING, `col166` STRING, `col167` STRING, `col168` STRING, `col169` STRING, `col170` STRING, `col171` STRING, `col172` STRING, `col173` STRING, `col174` STRING, `col175` STRING, `col176` STRING, `col177` STRING, `col178` STRING, `col179` STRING, `col180` STRING, `col181` STRING, `col182` STRING, `col183` STRING, `col1...'

Setup:

Windows Server 2019 Datacenter 64 bit , 32 GB memory

Source: Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

Target: Azure Databricks Delta

All prerequisites documented in this link are followed:

https://help.qlik.com/en-US/replicate/May2021/Content/Replicate/Main/Databricks%20Delta/databricks_delta_prereq_hadoop_gen_target.htm

see attached for detailed logs.

Re: Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

KellyHobson — Wed, 06 Apr 2022 13:31:07 GMT

Hey @amshirkh ,

I wanted to follow up on this one (better late than never :)) in case others run into this issue.

From another customer case with same "RetCode: SQL_ERROR SqlState: HY000 NativeError: 35 Message: [Simba][Hardy] (35).....Error running query: Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key" message.

This requirement was missing on the client side

To be able to access the storage directories from the Databricks cluster, users need to add a configuration (in Spark Config) for that Storage Account and its key.
Example:
fs.azure.account.key.<storage-account-name>.dfs.core.windows.net <storage-account-access-key>

Best,

Kelly

Re: Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

lyka — Wed, 06 Apr 2022 14:16:07 GMT

Hey @amshirkh

Please let us know if the proposed solution worked.

Thanks

Lyka

Re: Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

bnok — Wed, 11 May 2022 17:40:25 GMT

We have the same issue and have verified that the spark config has the correct key for the storage account but the error continues to persist.

Re: Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

bnok — Wed, 11 May 2022 17:41:55 GMT

Are there other alternatives that could be causing "NativeError: 35 Message: [Simba][Hardy] (35).....Error running query: Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key" ?

Re: Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

amshirkh — Wed, 11 May 2022 17:51:15 GMT

The issue for my project turned out to be extra end of line characters in some String/text types at the source. ( Oracle ). A simple transformation rule in the Qlik task ( remove the end of line character ) fixed the issue.

The key error was misleading.

Re: Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

Steve_Nguyen — Wed, 11 May 2022 17:55:35 GMT

@amshirkh

thanks for the update.

topic Re: Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta in Qlik Replicate

Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

Re: Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

Re: Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

Re: Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

Re: Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

Re: Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta

Re: Replicate error in full load/cdc changes from Oracle to Azure Databricks Delta