topic Re: HDFS transformation error in Talend Studio https://community.qlik.com/t5/Talend-Studio/HDFS-transformation-error/m-p/2240510#M27979 Hi, <BR />On which build version you got this issue? Do you have any problem when upload the screenshots into forum? <BR />More information will be helpful for us to address your issue. <BR />Best regards <BR />Sabrina Fri, 04 Mar 2016 07:13:07 GMT Anonymous 2016-03-04T07:13:07Z HDFS transformation error https://community.qlik.com/t5/Talend-Studio/HDFS-transformation-error/m-p/2240509#M27978 Its a simple extraction job, no transformations but still getting error: <BR />Not able to upload images - its a Big Data batch job, and the same one runs for other files on hdfs but this one. <BR /> <PRE>java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to talendpoc.extract_loans_0_1.row1Struct<BR />at talendpoc.extract_loans_0_1.extract_loans$tFileOutputDelimited_1StructOutputFormat$HDFSRecordWriter.write(extract_loans.java:1)<BR />at org.apache.spark.SparkHadoopWriter.write(SparkHadoopWriter.scala:95)<BR />at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1073)<BR />at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)<BR />at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)<BR />at org.apache.spark.scheduler.Task.run(Task.scala:64)<BR />at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)<BR />at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<BR />at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<BR />at java.lang.Thread.run(Thread.java:745)<BR />: org.apache.spark.scheduler.TaskSetManager - Task 0 in stage 0.0 failed 4 times; aborting job<BR />org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, ip-10-9-1-197.ec2.internal): java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to talendpoc.extract_loans_0_1.row1Struct<BR />at talendpoc.extract_loans_0_1.extract_loans$tFileOutputDelimited_1StructOutputFormat$HDFSRecordWriter.write(extract_loans.java:1)<BR />at org.apache.spark.SparkHadoopWriter.write(SparkHadoopWriter.scala:95)<BR />at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1073)<BR />at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)<BR />at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)<BR />at org.apache.spark.scheduler.Task.run(Task.scala:64)<BR />at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)<BR />at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<BR />at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<BR />at java.lang.Thread.run(Thread.java:745)<BR />Driver stacktrace:<BR />at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1204)<BR />at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1193)<BR />at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)<BR />at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)<BR />at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)<BR />at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1192)<BR />at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)<BR />at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)<BR />at scala.Option.foreach(Option.scala:236)<BR />at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)<BR />at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)<BR />at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)<BR />at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)<BR />1|20160303071453_LSC0S|20160303071453_LSC0S|20160303071453_LSC0S|out|0|2|100.0<BR />1|20160303071453_LSC0S|20160303071453_LSC0S|20160303071453_LSC0S|row1|0|2|100.0<BR />1|20160303071453_LSC0S|20160303071453_LSC0S|20160303071453_LSC0S|row_tFileInputDelimited_1_HDFSInputFormat|0|2|100.0<BR />org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, ip-10-9-1-197.ec2.internal): java.lang.ClassCastException: org.apache.avro.generic.GenericData$Record cannot be cast to talendpoc.extract_loans_0_1.row1Struct<BR />at talendpoc.extract_loans_0_1.extract_loans$tFileOutputDelimited_1StructOutputFormat$HDFSRecordWriter.write(extract_loans.java:1)<BR />at org.apache.spark.SparkHadoopWriter.write(SparkHadoopWriter.scala:95)<BR />at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1073)<BR />at org.apache.spark.rdd.PairRDDFunctions$$anonfun$13.apply(PairRDDFunctions.scala:1059)<BR />at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)<BR />at org.apache.spark.scheduler.Task.run(Task.scala:64)<BR />at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)<BR />at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)<BR />at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)<BR />at java.lang.Thread.run(Thread.java:745)<BR />Driver stacktrace:<BR />at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1204)<BR />at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1193)<BR />at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)<BR />at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)<BR />at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)<BR />at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1192)<BR />at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)<BR />at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)<BR />at scala.Option.foreach(Option.scala:236)<BR />at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)<BR />at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)<BR />at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)<BR />at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)<BR /><I>Job extract_loans ended at 16:39 02/03/2016. </I><BR /></PRE> Sat, 16 Nov 2024 10:45:29 GMT https://community.qlik.com/t5/Talend-Studio/HDFS-transformation-error/m-p/2240509#M27978 Anonymous 2024-11-16T10:45:29Z Re: HDFS transformation error https://community.qlik.com/t5/Talend-Studio/HDFS-transformation-error/m-p/2240510#M27979 Hi, <BR />On which build version you got this issue? Do you have any problem when upload the screenshots into forum? <BR />More information will be helpful for us to address your issue. <BR />Best regards <BR />Sabrina Fri, 04 Mar 2016 07:13:07 GMT https://community.qlik.com/t5/Talend-Studio/HDFS-transformation-error/m-p/2240510#M27979 Anonymous 2016-03-04T07:13:07Z