topic Re: tSqlRow - exception with bigdata on sun solaris in Talend Studio

tSqlRow - exception with bigdata on sun solaris

Anonymous — Sat, 16 Nov 2024 08:16:29 GMT

Hi,

We have talend server V6.3.1 running in linux with java jdk1.8.0_171 version. which runs tsqlrow fine but we have another env where we are running the same in SunSolaris and none of the tsqlrow components work . This is not OOM.

and exception we get is very confusing.

Checking ports...

Sending job 'SF_expo_ce_data' to server (acredit-etl21:8001)...

File transfer completed.

Deploying job 'SF_expo_ce_data' on server (158.137.74.35:8000)...

Running job 'SF_expo_ce_data'...
Starting job SF_expo_ce_data at 10:40 14/05/2018.

[statistics] connecting to socket on port 3954
[statistics] connected
[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.spark.SparkConf - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).

[Stage 0 (0 + 0) / 2]
[Stage 0 (0 + 2) / 2]
[Stage 0 (0 + 2) / 2][Stage 1 (0 + 2) / 2]
[Stage 1 (0 + 2) / 2]
[Stage 2 (0 + 0) / 200]
[Stage 2 (0 + 😎 / 200][thread 86 also had an error]
[thread 62 also had an error]
[thread 88 also had an error]
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0xffffffff6be8cb40, pid=5570, tid=0x0000000000000055
#
# JRE version: Java(TM) SE Runtime Environment (8.0_171-b11) (build 1.8.0_171-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.171-b11 mixed mode solaris-sparc compressed oops)
# Problematic frame:
# J 6141 C1 org.apache.spark.unsafe.bitset.BitSetMethods.isSet(Ljava/lang/Object;JI)Z (87 bytes) @ 0xffffffff6be8cb40 [0xffffffff6be8c9e0+0x160]
#
# Core dump written. Default location: /home/XXXX/talend/Talend-JobServer-20161216_1026-V6.3.1/TalendJobServersFiles/repository/SAMPLE2_SF_expo_ce_data_20180514_104022_CBvlX/SF_expo_ce_data/core or core.5570
#
# An error report file with more information is saved as:
# /home/XXXXX/talend/Talend-JobServer-20161216_1026-V6.3.1/TalendJobServersFiles/repository/SAMPLE2_SF_expo_ce_data_20180514_104022_CBvlX/SF_expo_ce_data/hs_err_pid5570.log
AHE@0x00000001002cfce0: 0xba000000 i2c: 0xffffffff6b83c920 c2i: 0xffffffff6b83c9c4 c2iUV: 0xffffffff6b83c93c
[thread 87 also had an error][thread 63 also had an error]

[thread 61 also had an error][thread 60 also had an error]

Compiled method (c1) 24578 6141 3 org.apache.spark.unsafe.bitset.BitSetMethods::isSet (87 bytes)
total in heap [0xffffffff6be8c850,0xffffffff6be8cd68] = 1304
relocation [0xffffffff6be8c978,0xffffffff6be8c9c8] = 80
main code [0xffffffff6be8c9e0,0xffffffff6be8cc40] = 608
stub code [0xffffffff6be8cc40,0xffffffff6be8cca8] = 104
oops [0xffffffff6be8cca8,0xffffffff6be8ccb0] = 8
metadata [0xffffffff6be8ccb0,0xffffffff6be8ccd0] = 32
scopes data [0xffffffff6be8ccd0,0xffffffff6be8cd20] = 80
scopes pcs [0xffffffff6be8cd20,0xffffffff6be8cd60] = 64
dependencies [0xffffffff6be8cd60,0xffffffff6be8cd68] = 8
Compiled method (c1) 24580 6138 3 org.apache.spark.sql.catalyst.expressions.UnsafeRow::isNullAt (18 bytes)
total in heap [0xffffffff6c501cd0,0xffffffff6c5021b8] = 1256
relocation [0xffffffff6c501df8,0xffffffff6c501e48] = 80
main code [0xffffffff6c501e60,0xffffffff6c502020] = 448
stub code [0xffffffff6c502020,0xffffffff6c502110] = 240
oops [0xffffffff6c502110,0xffffffff6c502118] = 8
metadata [0xffffffff6c502118,0xffffffff6c502138] = 32
scopes data [0xffffffff6c502138,0xffffffff6c502160] = 40
scopes pcs [0xffffffff6c502160,0xffffffff6c5021b0] = 80
dependencies [0xffffffff6c5021b0,0xffffffff6c5021b8] = 8
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
Job SF_expo_ce_data ended at 10:41 14/05/2018. [exit code=6]

Re: tSqlRow - exception with bigdata on sun solaris

Jesperrekuh — Mon, 14 May 2018 16:26:30 GMT

Interesting...
I recently had a problem ( not Talend related ) with openJDK vs oracleJDK, changed from jre -> to sdk and fixed this issue (compile, not Spark related).

But Im not quite sure about this one.
Im assuming (Im not a java developer) your talend - job/(spark)jars/environment isn't suitable for this specific environment due to java / platform diff.
Java engine still needs to talk to your OS and if native libraries (JNI) are used for Spark config ...

Did you have the SDK installed and not just the JRE on Sun?
Curious if (Oracle) sunSol is supported by Spark libs ???

Re: tSqlRow - exception with bigdata on sun solaris

Anonymous — Mon, 14 May 2018 16:45:13 GMT

Did you have the SDK installed and not just the JRE on Sun? --> I have JDK so that should include JRE.

Curious if (Oracle) sunSol is supported by Spark libs--> If the spark libraries are not compatible with solaris then other Spark components should also face problem i can solve same problem by using tmap and it works.

Re: tSqlRow - exception with bigdata on sun solaris

Jesperrekuh — Mon, 14 May 2018 17:07:29 GMT

--> Nice and for the sake of this discussion )

Depends on what classes are available and used in the Spark (any vendor) libs ... so not necessarily. Some errors are only raised when a specific class/sparkobject is 'constructed' / code/instructions are run/presented within the jvm, while other classes within the same lib run perfectly fine until a part of the code calls this class.

(a jar file contains all packaged shizzle, open with a 7zip / winrar / unzip )

Re: tSqlRow - exception with bigdata on sun solaris

Anonymous — Mon, 14 May 2018 18:42:54 GMT

ok yaa i see your point. I stll need to figure out What is causing this issue.

Re: tSqlRow - exception with bigdata on sun solaris

Anonymous — Mon, 14 May 2018 18:54:04 GMT

Ok I tested with straight extract and it works fine as soon i add a lookup file to tsqlrow it dosen't work and sql is very simple left join on 1 field. Agian this works in linux.

Re: tSqlRow - exception with bigdata on sun solaris

Jesperrekuh — Mon, 14 May 2018 21:42:02 GMT

Are you directly reading from a file stored on the filesystem or mounted to a network? .

Because I can imagine there's a small difference regarding filesystems and i/o... after some reading there's a difference in the way users/groups/acl is handled, also it's kernel functionality... maybe a little far fetched but I suggest asking a Solaris admin/expert on this Solaris mount options

Maybe check and install your Solaris db-drivers: what DB (Oracle, DB2, MSSQL, MySQL... ) ? Because you read from a file based on a db-client (driver) which open a file which contains db-inserts, am I right? you dont first opend the file and read line by line and inject them to the SQLRow?

Re: tSqlRow - exception with bigdata on sun solaris

Anonymous — Tue, 15 May 2018 16:31:11 GMT

I am reading a file and need to perform some SQLs on this . This is to avoid tmap in spark jobs for performance reasons. for now i am using a local file but this would be S3 in prod. Not sure how this is related to DB drivers. Also ACLs does not seem to be problem if there's only one file i am sourcing, as soon as i include a join it blows up.

Re: tSqlRow - exception with bigdata on sun solaris

Jesperrekuh — Tue, 15 May 2018 17:18:28 GMT

Ok .... please provide a screendump of your job flow. It helps to get an understaning on how you clicked it all together !?

0 - When you read or store things on disk, i/o and kernel functionality determines how you get data, blocksize fragments... in example think of NTFS vs FAT16 vs FAT32 on windows.

1 - Depending on your DB (engine tsql) with a mount to your solaris system... acl and UFS is relevant

INFILE '/somepathtosolarisfolder/loader.txt'
  INTO TABLE my_table
  FIELDS TERMINATED BY ','
  ( id         CHAR(10),
    author     CHAR(30),
    created    DATE "YYYY-MM-DD" ":created",
    fname      FILLER CHAR(80),
    text       LOBFILE(fname CHARACTERSET UTF8) TERMINATED BY EOF
  )

2 - Or : ... acl and UFS is relevant, but has tMap and !kaaabooooms!
tFileInputDelimted (contains some data) -> tmap (join with DBInput) -> tDBOutput
3 - Or :... acl and UFS is relevant

tSQLBulkExec component attribute : remoteFile (contains sql inserts) -> DBInput -> DB joins query -> DBoutput

Re: tSqlRow - exception with bigdata on sun solaris

Anonymous — Tue, 15 May 2018 18:32:54 GMT

So i created a prototype of my problem.

first flow from source file1 to tsql to tlogrow runs and dumps data.

The next sub process joining file1 with file2 throws the error

the SQL in 2nd process is

"SELECT
row3.dl_obj_id,
row5.enhc_prnt_obj_id
FROM row3 LEFT JOIN row5 ON
row3.enhc_obj_id=row5.enhc_prnt_obj_id
"

output

Pool|225103197.21|16882739.79|15906283.1|0.075|4||||

[Stage 3 (0 + 0) / 2]
[Stage 2 (0 + 2) / 2][Stage 3 (0 + 2) / 2]
[Stage 2 (0 + 2) / 2]
[Stage 4 (0 + 😎 / 200]#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0xa) at pc=0xffffffff6c96ba00, pid=17679, tid=0x000000000000005c
#
# JRE version: Java(TM) SE Runtime Environment (8.0_171-b11) (build 1.8.0_171-b11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.171-b11 mixed mode solaris-sparc compressed oops)
# Problematic frame:
# J 5485 C1

...... Rest of the error

Re: tSqlRow - exception with bigdata on sun solaris

Jesperrekuh — Sat, 19 May 2018 00:55:06 GMT

Thanks this is helpfull!!!
Could it be datatypes? Did you try a match with just two [ids], strings?
You build jobs local and export them to solaris machine?
JVM should handle this, but the spark driver is packaged and exported with your job, could it be the fact spark vs java and the solaris versions with spark?

Im out of ideas... last step is raise a bug!

Re: tSqlRow - exception with bigdata on sun solaris

Anonymous — Wed, 23 May 2018 16:15:40 GMT

Yes we build job locally and export it to Solaris.

It has to be OS issue .

Not sure who to raise the bug to. Spark or Talend ?