How to create big batch data job-with component cassandra,MSSQL,spark

Anonymous — Sat, 16 Nov 2024 10:23:04 GMT

Hi,
I'm new to talend.
I have to read data from Cassandra and sql server in talend big data batch jobs(spark code).There are component for MySQL but not for MSSQL.

Also while reading data from Cassandra in big data batch job I'm getting error:
Starting job POC_TEST_cassandra_component at 11:39 16/09/2016.
connecting to socket on port 3435
connected
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
SLF4J: Found binding in
SLF4J: See for an explanation.
SLF4J: Actual binding is of type
: org.apache.spark.SparkConf - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
: org.apache.hadoop.util.Shell - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
at org.apache.hadoop.security.Groups.<init>(Groups.java:77)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)
at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:232)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:718)
at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:703)
at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:605)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:1996)
at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:1996)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:1996)
at org.apache.spark.SecurityManager.<init>(SecurityManager.scala:207)
at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:163)
at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:267)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:270)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:61)
at rsa_dev.poc_test_cassandra_component_0_1.POC_TEST_cassandra_component.runJobInTOS(POC_TEST_cassandra_component.java:1336)
at rsa_dev.poc_test_cassandra_component_0_1.POC_TEST_cassandra_component.main(POC_TEST_cassandra_component.java:1234)
: org.apache.spark.SparkContext - Jar not found at ../lib/dom4j-1.6.1.jar
: org.apache.spark.SparkContext - Jar not found at ../lib/talend_file_enhanced_20070724.jar
java.sql.SQLException: null, message from server: "Host 'GLXY-DEV-TLD' is not allowed to connect to this MySQL server"
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:973)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1112)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2506)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2539)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2321)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at rsa_dev.poc_test_cassandra_component_0_1.POC_TEST_cassandra_component.tCassandraInput_1Process(POC_TEST_cassandra_component.java:1073)
at rsa_dev.poc_test_cassandra_component_0_1.POC_TEST_cassandra_component.run(POC_TEST_cassandra_component.java:1434)
at rsa_dev.poc_test_cassandra_component_0_1.POC_TEST_cassandra_component.runJobInTOS(POC_TEST_cassandra_component.java:1338)
at rsa_dev.poc_test_cassandra_component_0_1.POC_TEST_cassandra_component.main(POC_TEST_cassandra_component.java:1234)
java.sql.SQLException: null, message from server: "Host 'GLXY-DEV-TLD' is not allowed to connect to this MySQL server"
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:973)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1112)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2506)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2539)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2321)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:832)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
disconnected
: rsa_dev.poc_test_cassandra_component_0_1.POC_TEST_cassandra_component - TalendJob: 'POC_TEST_cassandra_component' - Failed with exit code: 1.
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:417)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
at java.sql.DriverManager.getConnection(Unknown Source)
at java.sql.DriverManager.getConnection(Unknown Source)
at rsa_dev.poc_test_cassandra_component_0_1.POC_TEST_cassandra_component.tCassandraInput_1Process(POC_TEST_cassandra_component.java:1073)
at rsa_dev.poc_test_cassandra_component_0_1.POC_TEST_cassandra_component.run(POC_TEST_cassandra_component.java:1434)
at rsa_dev.poc_test_cassandra_component_0_1.POC_TEST_cassandra_component.runJobInTOS(POC_TEST_cassandra_component.java:1338)
at rsa_dev.poc_test_cassandra_component_0_1.POC_TEST_cassandra_component.main(POC_TEST_cassandra_component.java:1234)
Job POC_TEST_cassandra_component ended at 11:39 16/09/2016.

Re: How to create big batch data job-with component cassandra,MSSQL,spark

Anonymous — Mon, 26 Sep 2016 09:58:20 GMT

Hi,
Could you please take a look at KB article about: TalendHelpCenter:The missing winutils.exe program in the Big Data Jobs?
Best regards
Sabrina

topic Re: How to create big batch data job-with component cassandra,MSSQL,spark in Talend Studio

How to create big batch data job-with component cassandra,MSSQL,spark

Re: How to create big batch data job-with component cassandra,MSSQL,spark