topic Re: tExtractXMl component in a Spark job in Talend Studio

tExtractXMl component in a Spark job

badri-nair — Sat, 16 Nov 2024 06:40:17 GMT

The tExtractXMl compoent in a spark job is complaining with the following error for the below section of code .Unable to build the job.

Error message: "the code of method call (Tuple2<NullWritable,row9Struct>) is exceeding the 65535 byte limit"

public java.util.Iterator<scala.Tuple2<NullWritable, row3Struct>> call(
scala.Tuple2<NullWritable, row9Struct> data)
throws java.lang.Exception {
java.util.List<scala.Tuple2<NullWritable, row3Struct>> outputs = new java.util.ArrayList<scala.Tuple2<NullWritable, row3Struct>>();
row3Struct row3 = new row3Struct();
row9Struct row2 = data._2;

PLease help me understand this issue .

Re: tExtractXMl component in a Spark job

vapukov — Tue, 05 Feb 2019 10:08:54 GMT

hi,

65535 it is a limit for java method code (maybe I not 100% correct describe, but it is knowing error)

source of error could be a complicated structure (with long XPath and many columns)

there are no single solution, but often possible resolve it if:

exclude not used tags (if any)
split into several steps (if possible) - parse half, then next half, then join

Re: tExtractXMl component in a Spark job

badri-nair — Wed, 06 Feb 2019 09:21:03 GMT

Thank you very much. Your sugggestion worked. I had 498 columns with xpath. reduced it to 300 columns and that worked.

Thanks ONce again

Badri Nair