Spark Connection Error

nnmh — Fri, 15 Nov 2024 23:05:02 GMT

I have an error while trying to use spark Batching in connecting spark with hive and hadoop

The error :

org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.TApplicationException: Invalid method name: 'get_database_req'

at org.apache.hadoop.hive.ql.metadata.Hive.getDatabase(Hive.java:2137) ~[hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]

at org.apache.hadoop.hive.ql.metadata.Hive.databaseExists(Hive.java:2122) ~[hive-exec-3.1.3000.7.1.1.0-565.jar:3.1.3000.7.1.1.0-565]

at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$databaseExists$1.apply$mcZ$sp(HiveClientImpl.scala:415) ~[spark-hive_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$databaseExists$1.apply(HiveClientImpl.scala:415) ~[spark-hive_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

at org.apache.spark.sql.hive.client.HiveClientImpl$$anonfun$withHiveState$1.apply(HiveClientImpl.scala:341) ~[spark-hive_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:271) [spark-hive_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:270) [spark-hive_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:324) [spark-hive_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

at org.apache.spark.sql.hive.client.HiveClientImpl.databaseExists(HiveClientImpl.scala:414) [spark-hive_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply$mcZ$sp(HiveExternalCatalog.scala:219) [spark-hive_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

at org.apache.spark.sql.hive.HiveExternalCatalog$$anonfun$databaseExists$1.apply(HiveExternalCatalog.scala:219) [spark-hive_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

at org.apache.spark.sql.hive.HiveExternalCatalog.withClient(HiveExternalCatalog.scala:97) [spark-hive_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

at org.apache.spark.sql.hive.HiveExternalCatalog.databaseExists(HiveExternalCatalog.scala:218) [spark-hive_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

at org.apache.spark.sql.internal.SharedState.externalCatalog$lzycompute(SharedState.scala:136) [spark-sql_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

at org.apache.spark.sql.internal.SharedState.externalCatalog(SharedState.scala:124) [spark-sql_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

at org.apache.spark.sql.hive.HiveSessionStateBuilder.org$apache$spark$sql$hive$HiveSessionStateBuilder$$externalCatalog(HiveSessionStateBuilder.scala:39) [spark-hive_2.11-2.4.0.7.1.1.0-565.jar:2.4.0.7.1.1.0-565]

any Help Please

Re: Spark Connection Error

Anonymous — Thu, 07 Apr 2022 03:28:04 GMT

Hello,

Could you please clarify in which Talend version/edition you are? What's your connection mode? Screenshots of hive and hadoop connection setting will be preferred.

Best regards

Sabrina

topic Spark Connection Error in Talend Studio

Spark Connection Error

Re: Spark Connection Error