Cannot call methods on SparkContext

DanM243 — Sat, 16 Nov 2024 04:58:51 GMT

Hi, I use Talend Real-time Big Data Platform and I encounter this error when I try to call a big data patch job with the tRunJob component. Can someone help me?
----------------------------------------------------------------------------------------------------------------------------------------
java.lang.RuntimeException: Child job returns 1. It doesn't terminate normally.
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
This stopped SparkContext was created at:

org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
keolis_maintenance.tf_tdb_maintenance_test_qualiac_0_1.TF_Tdb_Maintenance_Test_Qualiac.runJobInTOS(TF_Tdb_Maintenance_Test_Qualiac.java:1579)
keolis_maintenance.tf_tdb_maintenance_test_qualiac_0_1.TF_Tdb_Maintenance_Test_Qualiac.main(TF_Tdb_Maintenance_Test_Qualiac.java:1499)

The currently active SparkContext was created at:

(No active SparkContext.)

at org.apache.spark.SparkContext.assertNotStopped(SparkContext.scala:100)
at org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:2359)
at org.apache.spark.SparkContext.defaultMinPartitions(SparkContext.scala:2368)
at org.apache.spark.SparkContext.hadoopRDD$default$5(SparkContext.scala:995)
at org.apache.spark.api.java.JavaSparkContext.hadoopRDD(JavaSparkContext.scala:416)
at keolis_maintenance.tf_tdb_maintenance_test_qualiac_0_1.TF_Tdb_Maintenance_Test_Qualiac.tFileInputDelimited_2_HDFSInputFormatProcess(TF_Tdb_Maintenance_Test_Qualiac.java:1327)
at keolis_maintenance.tf_tdb_maintenance_test_qualiac_0_1.TF_Tdb_Maintenance_Test_Qualiac.run(TF_Tdb_Maintenance_Test_Qualiac.java:1646)
at keolis_maintenance.tf_tdb_maintenance_test_qualiac_0_1.TF_Tdb_Maintenance_Test_Qualiac.runJobInTOS(TF_Tdb_Maintenance_Test_Qualiac.java:1590)
at keolis_maintenance.tf_tdb_maintenance_test_qualiac_0_1.TF_Tdb_Maintenance_Test_Qualiac.main(TF_Tdb_Maintenance_Test_Qualiac.java:1499)
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
This stopped SparkContext was created at:

The currently active SparkContext was created at:

(No active SparkContext.)

at org.apache.spark.SparkContext.assertNotStopped(SparkContext.scala:100)
at org.apache.spark.SparkContext.cancelJobGroup(SparkContext.scala:2242)
at org.apache.spark.api.java.JavaSparkContext.cancelJobGroup(JavaSparkContext.scala:823)
at keolis_maintenance.tf_tdb_maintenance_test_qualiac_0_1.TF_Tdb_Maintenance_Test_Qualiac.run(TF_Tdb_Maintenance_Test_Qualiac.java:1653)
at keolis_maintenance.tf_tdb_maintenance_test_qualiac_0_1.TF_Tdb_Maintenance_Test_Qualiac.runJobInTOS(TF_Tdb_Maintenance_Test_Qualiac.java:1590)
at keolis_maintenance.tf_tdb_maintenance_test_qualiac_0_1.TF_Tdb_Maintenance_Test_Qualiac.main(TF_Tdb_Maintenance_Test_Qualiac.java:1499)

at keolis_maintenance.tf_tdb_maintenance_loop_0_1.TF_Tdb_Maintenance_Loop.tLoop_1Process(TF_Tdb_Maintenance_Loop.java:869)
at keolis_maintenance.tf_tdb_maintenance_loop_0_1.TF_Tdb_Maintenance_Loop.run(TF_Tdb_Maintenance_Loop.java:1103)
at keolis_maintenance.tf_tdb_maintenance_loop_0_1.TF_Tdb_Maintenance_Loop.runJobInTOS(TF_Tdb_Maintenance_Loop.java:1062)
at keolis_maintenance.tf_tdb_maintenance_loop_0_1.TF_Tdb_Maintenance_Loop.main(TF_Tdb_Maintenance_Loop.java:973)
[statistics] disconnected
java.lang.RuntimeException: Child job returns 1. It doesn't terminate normally.
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512m; support was removed in 8.0
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext.
This stopped SparkContext was created at:

The currently active SparkContext was created at:

Le Job TF_Tdb_Maintenance_Loop sest terminé à {1,date,HH:mm dd/MM/yyyy}. [code de sortie={2,number}]

Re: Cannot call methods on SparkContext

Anonymous — Fri, 16 Aug 2019 05:06:15 GMT

Hello,

Would you mind posting your big data batch job screenshots here and clarify in which Talend Real-time Big Data Platform version/edition you are?

Please mask your sensitive data.

Best regards

Sabrina

topic Re: Cannot call methods on SparkContext in Talend Studio

Cannot call methods on SparkContext

Re: Cannot call methods on SparkContext