Hello-
I am running a simple spark job, to convert file to parquet using Talend.
system throws error :
Caused by: java.lang.ClassNotFoundException: org.antlr.v4.runtime.misc.ParseCancellationException
and When i check modules, antlr is available.
do you have pointers ?
----------------------------------------------------------------------------
Job Server : On remote machine.
Spark: Amazon EMR
---------------------------------------------------------------------
Talend Real-time Big Data Platform
Version: 7.0.1
Build id: 20180411_1414
---------------------------------------------------------------------
java version "1.8.0_102"
Java(TM) SE Runtime Environment (build 1.8.0_102-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)
---------------------------------------------------------------------
***
[INFO ]: org.apache.spark.ui.SparkUI - Stopped Spark web UI at http://<Removed>:4040
[INFO ]: org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Interrupting monitor thread
[INFO ]: org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Shutting down all executors
[INFO ]: org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnDriverEndpoint - Asking each executor to shut down
[INFO ]: org.apache.spark.scheduler.cluster.SchedulerExtensionServices - Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
[DEBUG]: org.apache.hadoop.service.AbstractService - Service: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl entered state STOPPED
[DEBUG]: org.apache.hadoop.ipc.Client - stopping client from cache: org.apache.hadoop.ipc.Client@4d0b0fd4
[INFO ]: org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend - Stopped
[INFO ]: org.apache.spark.MapOutputTrackerMasterEndpoint - MapOutputTrackerMasterEndpoint stopped!
[INFO ]: org.apache.spark.storage.memory.MemoryStore - MemoryStore cleared
[INFO ]: org.apache.spark.storage.BlockManager - BlockManager stopped
[INFO ]: org.apache.spark.storage.BlockManagerMaster - BlockManagerMaster stopped
[INFO ]: org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint - OutputCommitCoordinator stopped!
[INFO ]: org.apache.spark.SparkContext - Successfully stopped SparkContext
[DEBUG]: io.netty.buffer.PoolThreadCache - Freed 3 thread-local buffer(s) from thread: rpc-server-3-4
Exception in thread "main" java.lang.NoClassDefFoundError: org/antlr/v4/runtime/misc/ParseCancellationException
at org.apache.spark.sql.internal.BaseSessionStateBuilder.sqlParser$lzycompute(BaseSessionStateBuilder.scala:117)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.sqlParser(BaseSessionStateBuilder.scala:116)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.catalog$lzycompute(BaseSessionStateBuilder.scala:138)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.catalog(BaseSessionStateBuilder.scala:131)
at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:289)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1050)
at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:130)
at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:130)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:129)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:126)
at org.apache.spark.sql.SparkSession.conf$lzycompute(SparkSession.scala:153)
at org.apache.spark.sql.SparkSession.conf(SparkSession.scala:153)
at org.apache.spark.sql.SQLContext.setConf(SQLContext.scala:125)
at local_project.testbigdata_0_1.TestBigData.tFileInputDelimited_1_HDFSInputFormatProcess(TestBigData.java:879)
at local_project.testbigdata_0_1.TestBigData.run(TestBigData.java:1184)
at local_project.testbigdata_0_1.TestBigData.runJobInTOS(TestBigData.java:1066)
at local_project.testbigdata_0_1.TestBigData.main(TestBigData.java:975)
Caused by: java.lang.ClassNotFoundException: org.antlr.v4.runtime.misc.ParseCancellationException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 18 more
[INFO ]: org.apache.spark.util.ShutdownHookManager - Shutdown hook called
[INFO ]: org.apache.spark.util.ShutdownHookManager - Deleting directory /tmp/spark-4f3354a7-70bb-482a-8282-96b74a80ece0
[DEBUG]: org.apache.hadoop.ipc.Client - stopping client from cache: org.apache.hadoop.ipc.Client@4d0b0fd4