Re: TOS BD Error - package org.apache.hadoop.io do... - Qlik Community

Anonymous · ‎2018-10-09

Hello,
I am getting the error "package org.apache.hadoop.io does not exist" when trying to import using tsqoopimport from TOS for bigdata windows client to CDH 5.13 quickstart vm Hive database. Could you please help in resolving this.

I tried to set the environment variables in CDH but still the same issue continues. $JAVA_HOME is also set to JDK 1.7 path.
export HADOOP_HOME=/usr/lib/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_INSTALL=$HADOOP_HOME
export CLASSPATH=/usr/lib/hadoop/client-0.20:/usr/lib/hive/lib

Error Log below :
Starting job Sqoop_DB2_to_Hive at 15:05 09/10/2018.
[statistics] connecting to socket on port 3768
[statistics] connected
[WARN ]: org.ap

To see the whole post, download it here

Anonymous · ‎2018-10-17

Hello,

Could you please clarify in which Talend open studio for bigdata version/edition you are?

Ensure proper version of JDK is installed. For v6 of Talend products JDKv1.8 is required.

Best regards

Sabrina

Anonymous · ‎2018-10-24

Hello Sabrina

I am also getting same error . Below is the error detail.

Talend version is 7.0.1 (Real Time Big data edition)

Hadoop Cluster is Hortonworks-HDP 2.6.5

JDk Version is 1.8


Starting job HDFS_Test at 11:19 24/10/2018.

[statistics] connecting to socket on port 3348
[statistics] connected
[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.BytesWritable;
                           ^
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:8: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
                           ^
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:9: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Writable;
                           ^
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:37: error: cannot access Writable
public class T_SFO_USER extends SqoopRecord  implements DBWritable, Writable {
       ^
  class file for org.apache.hadoop.io.Writable not found
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:6348: error: cannot find symbol
  public void parse(Text __record) throws RecordParser.ParseError {
                    ^
  symbol:   class Text
  location: class T_SFO_USER
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:898: error: cannot find symbol
    this.Id = Id;
    ^          ^
  symbol:   variable this
  location: class T_SFO_USER
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:1239: error: cannot find symbol
    this.MobilePhone = MobilePhone;
    ^
  symbol:   variable this
  location: class T_SFO_USER
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:1242: error: cannot find symbol
    this.MobilePhone = MobilePhone;
    ^
  symbol:   variable this
  location: class T_SFO_USER
Note: \tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
100 errors
[ERROR]: org.apache.sqoop.tool.ImportTool - Encountered IOException running import job: java.io.IOException: Error returned by javac
	at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:217)
	at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:108)
	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.tSqoopImport_1Process(HDFS_Test.java:668)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.runJobInTOS(HDFS_Test.java:1004)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.main(HDFS_Test.java:830)

Exception in component tSqoopImport_1 (HDFS_Test)
java.lang.Exception: The Sqoop import job has failed. Please check the logs.
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.tSqoopImport_1Process(HDFS_Test.java:672)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.runJobInTOS(HDFS_Test.java:1004)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.main(HDFS_Test.java:830)
[FATAL]: hadoop_connection_project.hdfs_test_0_1.HDFS_Test - tSqoopImport_1 The Sqoop import job has failed. Please check the logs.
java.lang.Exception: The Sqoop import job has failed. Please check the logs.
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.tSqoopImport_1Process(HDFS_Test.java:672)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.runJobInTOS(HDFS_Test.java:1004)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.main(HDFS_Test.java:830)
[statistics] disconnected

Job HDFS_Test ended at 11:19 24/10/2018. [exit code=1]

Anonymous · ‎2018-11-02

Hello,

What's your RDBMS? Are you try to transfer data from Oracle into HDFS?

As indicated at https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_oracle_connection_reset_errors

A problem is related to the Oracle JDBC driver.

Additionally, please make sure that you don't install talend studio in path C:\Program Files (x86), since there is empty character,when job is running, can't find correct jar.

Best regards

Sabrina

roalro · ‎2019-05-03

Hi.

I have the same problem and I can't solve this. The problem is that SqoopImport component isn't able to solve de Query to insert the data in S3. Can't load the following classes

import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;

but load correctly others like

import org.apache.hadoop.mapred.lib.db.DBWritable;
import com.cloudera.sqoop.lib.JdbcWritableBridge;
import com.cloudera.sqoop.lib.DelimiterSet;
import com.cloudera.sqoop.lib.FieldFormatter;
import com.cloudera.sqoop.lib.RecordParser;
import com.cloudera.sqoop.lib.BooleanParser;
import com.cloudera.sqoop.lib.BlobRef;
import com.cloudera.sqoop.lib.ClobRef;
import com.cloudera.sqoop.lib.LargeObjectLoader;
import com.cloudera.sqoop.lib.SqoopRecord;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.sql.Date;
import java.sql.Time;
import java.sql.Timestamp;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.HashMap;

What is the problem? Did the other users solve this issue?

The version of talend is Talend Open Studio Big Data 7.1.1

java JDK is 1.8

Cloudera -- CDH5.14.4 Dymanic load in the repository.

Thanks.

roalro · ‎2019-05-07

Hi.

I have the same problem. I'm trying to import a table from PostgreSQL with component tSqoopImport and when run the job the log says:

[WARN ]: org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
[WARN ]: org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.BytesWritable;
^
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:8: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:9: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Writable;

Other classes like

import org.apache.hadoop.mapred.lib.db.DBWritable;
import com.cloudera.sqoop.lib.JdbcWritableBridge;
import com.cloudera.sqoop.lib.DelimiterSet;
import com.cloudera.sqoop.lib.FieldFormatter;
import com.cloudera.sqoop.lib.RecordParser;
import com.cloudera.sqoop.lib.BooleanParser;
import com.cloudera.sqoop.lib.BlobRef;
import com.cloudera.sqoop.lib.ClobRef;
import com.cloudera.sqoop.lib.LargeObjectLoader;
import com.cloudera.sqoop.lib.SqoopRecord;

are correctly load but not the others. Why does this problem happen? How did the others members solve this?

Version of the components:

TOS BD 7.1.1

CDH_5_14_4

JDK 1.8

Windows 10

PostgreSQL 9.4

Thanks.

roalro · ‎2019-05-07

Hi.

I have the same problem. I'm trying to import a table from PostgreSQL with component tSqoopImport and when run the job the log says:

[WARN ]: org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
[WARN ]: org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.BytesWritable;
^
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:8: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:9: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Writable;

Other classes like

import org.apache.hadoop.mapred.lib.db.DBWritable;
import com.cloudera.sqoop.lib.JdbcWritableBridge;
import com.cloudera.sqoop.lib.DelimiterSet;
import com.cloudera.sqoop.lib.FieldFormatter;
import com.cloudera.sqoop.lib.RecordParser;
import com.cloudera.sqoop.lib.BooleanParser;
import com.cloudera.sqoop.lib.BlobRef;
import com.cloudera.sqoop.lib.ClobRef;
import com.cloudera.sqoop.lib.LargeObjectLoader;
import com.cloudera.sqoop.lib.SqoopRecord;

are correctly load but not the others. Why does this problem happen? How did the others members solve this?

Version of the components:

TOS BD 7.1.1

CDH_5_14_4

JDK 1.8

Windows 10

PostgreSQL 9.4

Thanks.

roalro · ‎2019-05-07

Hi.

I have the same problem. I'm trying to import a table from PostgreSQL with component tSqoopImport and when run the job the log says:

[WARN ]: org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
[WARN ]: org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.BytesWritable;
^
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:8: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:9: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Writable;

Other classes like

import org.apache.hadoop.mapred.lib.db.DBWritable;
import com.cloudera.sqoop.lib.JdbcWritableBridge;
import com.cloudera.sqoop.lib.DelimiterSet;
import com.cloudera.sqoop.lib.FieldFormatter;
import com.cloudera.sqoop.lib.RecordParser;
import com.cloudera.sqoop.lib.BooleanParser;
import com.cloudera.sqoop.lib.BlobRef;
import com.cloudera.sqoop.lib.ClobRef;
import com.cloudera.sqoop.lib.LargeObjectLoader;
import com.cloudera.sqoop.lib.SqoopRecord;

are correctly load but not the others. Why does this problem happen? How did the others members solve this?

Version of the components:

TOS BD 7.1.1

CDH_5_14_4

JDK 1.8

Windows 10

PostgreSQL 9.4

Thanks.

Anonymous · ‎2019-05-17

Hello,

Where is your hadoop server? Did you install it in local? It looks like the problem is with the setting up the correct class path. What does your classpath look like?

Best regards

Sabrina

roalro · ‎2019-05-17

Hi.

Firstly thanks for your reply.

The hadoop server is not in local, it is in an another EC2 instance.

Precisely I have readed that the problem could be in the classpath, but how do i know what is the classpath wicht my Talend is using?

Thanks.

TOS BD Error - package org.apache.hadoop.io does not exist

Java

JDBC

Talend Big Data

v7.x