How to set the "org.apache.spark.serializer.KryoSerialize

badri-nair — Sat, 16 Nov 2024 06:42:05 GMT

HI,

I am using the Talend Cloud BigData platform Version 7.1.1

In the map there is a parquet file which is reading a field, which has xml value. its quite large (12kb) per field.

The code fails . with the below error.

How do i set the Customize Spark serialiser option "org.apache.spark.serializer.KryoSerialize" ??

what is the value that i need to put in the box to bump up the Memory ?

ERROR message:

#############################################################################################

Caused by: org.apache.spark.SparkException: Kryo serialization failed: Buffer overflow. Available: 0, required: 12264
Serialization trace:
xmldata (t_data.t_data_staging_flight_passenger_0_1.row1Struct). To avoid this, increase spark.kryoserializer.buffer.max value.
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:318)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:383)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.esotericsoftware.kryo.KryoException: Buffer overflow. Available: 0, required: 12264

#############################################################################################

Re: How to set the "org.apache.spark.serializer.KryoSerialize

badri-nair — Fri, 01 Feb 2019 11:51:31 GMT

Found the solution by myself.

Edit the hadoop cluster connection under metadata (values needs to be unexported)

Click on the use spark configuration button.

THere you can enter key value pairs . insert a row and ener the value as in the screen shot . It worked for me

topic How to set the "org.apache.spark.serializer.KryoSerialize in Talend Studio

How to set the "org.apache.spark.serializer.KryoSerialize

Re: How to set the "org.apache.spark.serializer.KryoSerialize