Qlik Talend Big Data: Migrated Big Data Batch jobs throw "IndexOutOfBoundsException" error

[wei_guo]

Problem

Our Customer has over existing 200 big data spark jobs migrated from 7.3.1 to 8.0.1 and it gives "IndexOutOfBoundsException" after "use Dataset API in migrated components" checkbox is enabled in Spark Configuration.

Cause  

UsedatasetAPIinMigratedComponents

Error Log

!ENTRY org.eclipse.ui.workbench 4 2 2024-03-28 18:01:50.038 !MESSAGE Problems occurred when invoking code from plug-in: "org.eclipse.ui.workbench". !STACK 0 java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0 at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) at java.base/java.util.Objects.checkIndex(Objects.java:372) at java.base/java.util.ArrayList.get(ArrayList.java:459) at org.talend.designer.bigdata.avro.AvroRecordStructGenerator.firstSchemaFieldHasAnAsterisk(AvroRecordStructGenerator.java:256) at org.talend.designer.bigdata.avro.AvroRecordStructGenerator.generateRecordStructForDataset(AvroRecordStructGenerator.java:242) at org.talend.repository.spark.editor.SparkDataProcess.generateRecordStructForConnection(SparkDataProcess.java:207) at org.talend.designer.bigdata.common.BigDataDataProcess.buildRecordStructsForConnections(BigDataDataProcess.java:612) at org.talend.designer.bigdata.common.BigDataDataProcess.buildRecordStructsForConnections(BigDataDataProcess.java:588) at org.talend.repository.spark.editor.SparkDataProcess.buildFromGraphicalProcess(SparkDataProcess.java:156) at org.talend.designer.bigdata.common.BigDataProcess.getGeneratingNodes(BigDataProcess.java:93) at org.talend.designer.core.ui.editor.process.Process.getNodesOfType(Process.java:3949) at org.talend.designer.core.ui.editor.nodes.Node.checkParameters(Node.java:3029) at org.talend.designer.core.ui.editor.nodes.Node.checkNode(Node.java:4406) at org.talend.designer.core.ui.editor.process.Process.checkProblems(Process.java:3801) at org.talend.designer.core.ui.editor.process.Process.checkProcess(Process.java:3792) at org.talend.designer.core.ui.views.problems.Problems.initCurrentProblems(Problems.java:265) at org.talend.designer.core.ui.views.problems.Problems.addProcess(Problems.java:257) at org.talend.designer.core.ui.ActiveProcessTracker.addJobInProblemView(ActiveProcessTracker.java:169) at org.talend.designer.core.ui.ActiveProcessTracker.partOpened(ActiveProcessTracker.java:294)

After checked with R&D  and made some investigation and analysis on it,  this is a Plugin code generation logic bug from: plugins/org.talend.designer.bigdata/src/org/talend/designer/bigdata/avro/AvroRecordStructGenerator.java

Resolution

Fixed patch: V8.0.1-R2024-04 patch 

Internal Investigation ID(s)

Internal defect ID, TBD-16238

Environment

Talend Big Data

Talend Studio