topic Re: TOS BD Error - package org.apache.hadoop.io does not exist in Talend Studio

TOS BD Error - package org.apache.hadoop.io does not exist

Anonymous — Sat, 16 Nov 2024 07:30:01 GMT

Hello,
I am getting the error "package org.apache.hadoop.io does not exist" when trying to import using tsqoopimport from TOS for bigdata windows client to CDH 5.13 quickstart vm Hive database. Could you please help in resolving this.

I tried to set the environment variables in CDH but still the same issue continues. $JAVA_HOME is also set to JDK 1.7 path.
export HADOOP_HOME=/usr/lib/hadoop
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_INSTALL=$HADOOP_HOME
export CLASSPATH=/usr/lib/hadoop/client-0.20:/usr/lib/hive/lib

Error Log below :
Starting job Sqoop_DB2_to_Hive at 15:05 09/10/2018.
[statistics] connecting to socket on port 3768
[statistics] connected
[WARN ]: org.ap

To see the whole post, download it here

Re: TOS BD Error - package org.apache.hadoop.io does not exist

Anonymous — Wed, 17 Oct 2018 08:11:02 GMT

Hello,

Could you please clarify in which Talend open studio for bigdata version/edition you are?

Ensure proper version of JDK is installed. For v6 of Talend products JDKv1.8 is required.

Best regards

Sabrina

Re: TOS BD Error - package org.apache.hadoop.io does not exist

Anonymous — Wed, 24 Oct 2018 10:49:34 GMT

Hello Sabrina

I am also getting same error . Below is the error detail.

Talend version is 7.0.1 (Real Time Big data edition)

Hadoop Cluster is Hortonworks-HDP 2.6.5

JDk Version is 1.8


Starting job HDFS_Test at 11:19 24/10/2018.

[statistics] connecting to socket on port 3348
[statistics] connected
[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.BytesWritable;
                           ^
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:8: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
                           ^
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:9: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Writable;
                           ^
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:37: error: cannot access Writable
public class T_SFO_USER extends SqoopRecord  implements DBWritable, Writable {
       ^
  class file for org.apache.hadoop.io.Writable not found
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:6348: error: cannot find symbol
  public void parse(Text __record) throws RecordParser.ParseError {
                    ^
  symbol:   class Text
  location: class T_SFO_USER
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:898: error: cannot find symbol
    this.Id = Id;
    ^          ^
  symbol:   variable this
  location: class T_SFO_USER
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:1239: error: cannot find symbol
    this.MobilePhone = MobilePhone;
    ^
  symbol:   variable this
  location: class T_SFO_USER
\tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java:1242: error: cannot find symbol
    this.MobilePhone = MobilePhone;
    ^
  symbol:   variable this
  location: class T_SFO_USER
Note: \tmp\sqoop-sharmak\compile\206cc178cefacea3ee8aec4e221c9238\T_SFO_USER.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
100 errors
[ERROR]: org.apache.sqoop.tool.ImportTool - Encountered IOException running import job: java.io.IOException: Error returned by javac
	at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:217)
	at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:108)
	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:488)
	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.tSqoopImport_1Process(HDFS_Test.java:668)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.runJobInTOS(HDFS_Test.java:1004)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.main(HDFS_Test.java:830)

Exception in component tSqoopImport_1 (HDFS_Test)
java.lang.Exception: The Sqoop import job has failed. Please check the logs.
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.tSqoopImport_1Process(HDFS_Test.java:672)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.runJobInTOS(HDFS_Test.java:1004)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.main(HDFS_Test.java:830)
[FATAL]: hadoop_connection_project.hdfs_test_0_1.HDFS_Test - tSqoopImport_1 The Sqoop import job has failed. Please check the logs.
java.lang.Exception: The Sqoop import job has failed. Please check the logs.
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.tSqoopImport_1Process(HDFS_Test.java:672)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.runJobInTOS(HDFS_Test.java:1004)
	at hadoop_connection_project.hdfs_test_0_1.HDFS_Test.main(HDFS_Test.java:830)
[statistics] disconnected

Job HDFS_Test ended at 11:19 24/10/2018. [exit code=1]

Re: TOS BD Error - package org.apache.hadoop.io does not exist

Anonymous — Fri, 02 Nov 2018 08:00:13 GMT

Hello,

What's your RDBMS? Are you try to transfer data from Oracle into HDFS?

As indicated at https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_oracle_connection_reset_errors

A problem is related to the Oracle JDBC driver.

Additionally, please make sure that you don't install talend studio in path C:\Program Files (x86), since there is empty character,when job is running, can't find correct jar.

Best regards

Sabrina

Re: TOS BD Error - package org.apache.hadoop.io does not exist

roalro — Fri, 03 May 2019 12:29:02 GMT

Hi.

I have the same problem and I can't solve this. The problem is that SqoopImport component isn't able to solve de Query to insert the data in S3. Can't load the following classes

import org.apache.hadoop.io.BytesWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.Writable;

but load correctly others like

import org.apache.hadoop.mapred.lib.db.DBWritable;
import com.cloudera.sqoop.lib.JdbcWritableBridge;
import com.cloudera.sqoop.lib.DelimiterSet;
import com.cloudera.sqoop.lib.FieldFormatter;
import com.cloudera.sqoop.lib.RecordParser;
import com.cloudera.sqoop.lib.BooleanParser;
import com.cloudera.sqoop.lib.BlobRef;
import com.cloudera.sqoop.lib.ClobRef;
import com.cloudera.sqoop.lib.LargeObjectLoader;
import com.cloudera.sqoop.lib.SqoopRecord;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.sql.Date;
import java.sql.Time;
import java.sql.Timestamp;
import java.util.Arrays;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.HashMap;

What is the problem? Did the other users solve this issue?

The version of talend is Talend Open Studio Big Data 7.1.1

java JDK is 1.8

Cloudera -- CDH5.14.4 Dymanic load in the repository.

Thanks.

Re: TOS BD Error - package org.apache.hadoop.io does not exist

roalro — Tue, 07 May 2019 08:56:27 GMT

Hi.

I have the same problem. I'm trying to import a table from PostgreSQL with component tSqoopImport and when run the job the log says:

[WARN ]: org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
[WARN ]: org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.BytesWritable;
^
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:8: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:9: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Writable;

Other classes like

import org.apache.hadoop.mapred.lib.db.DBWritable;
import com.cloudera.sqoop.lib.JdbcWritableBridge;
import com.cloudera.sqoop.lib.DelimiterSet;
import com.cloudera.sqoop.lib.FieldFormatter;
import com.cloudera.sqoop.lib.RecordParser;
import com.cloudera.sqoop.lib.BooleanParser;
import com.cloudera.sqoop.lib.BlobRef;
import com.cloudera.sqoop.lib.ClobRef;
import com.cloudera.sqoop.lib.LargeObjectLoader;
import com.cloudera.sqoop.lib.SqoopRecord;

are correctly load but not the others. Why does this problem happen? How did the others members solve this?

Version of the components:

TOS BD 7.1.1

CDH_5_14_4

JDK 1.8

Windows 10

PostgreSQL 9.4

Thanks.

Re: TOS BD Error - package org.apache.hadoop.io does not exist

roalro — Tue, 07 May 2019 08:59:01 GMT

Hi.

I have the same problem. I'm trying to import a table from PostgreSQL with component tSqoopImport and when run the job the log says:

[WARN ]: org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
[WARN ]: org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.BytesWritable;
^
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:8: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:9: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Writable;

Other classes like

are correctly load but not the others. Why does this problem happen? How did the others members solve this?

Version of the components:

TOS BD 7.1.1

CDH_5_14_4

JDK 1.8

Windows 10

PostgreSQL 9.4

Thanks.

Re: TOS BD Error - package org.apache.hadoop.io does not exist

roalro — Tue, 07 May 2019 09:18:16 GMT

Hi.

I have the same problem. I'm trying to import a table from PostgreSQL with component tSqoopImport and when run the job the log says:

[WARN ]: org.apache.hadoop.hdfs.shortcircuit.DomainSocketFactory - The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
[WARN ]: org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.BytesWritable;
^
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:8: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
\tmp\sqoop-Roberto Alvarez\compile\3c0fcc15297a1402205e0dcdc88c14e7\QueryResult.java:9: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Writable;

Other classes like

import org.apache.hadoop.mapred.lib.db.DBWritable;
import com.cloudera.sqoop.lib.JdbcWritableBridge;
import com.cloudera.sqoop.lib.DelimiterSet;
import com.cloudera.sqoop.lib.FieldFormatter;
import com.cloudera.sqoop.lib.RecordParser;
import com.cloudera.sqoop.lib.BooleanParser;
import com.cloudera.sqoop.lib.BlobRef;
import com.cloudera.sqoop.lib.ClobRef;
import com.cloudera.sqoop.lib.LargeObjectLoader;
import com.cloudera.sqoop.lib.SqoopRecord;

are correctly load but not the others. Why does this problem happen? How did the others members solve this?

Version of the components:

TOS BD 7.1.1

CDH_5_14_4

JDK 1.8

Windows 10

PostgreSQL 9.4

Thanks.

Re: TOS BD Error - package org.apache.hadoop.io does not exist

Anonymous — Fri, 17 May 2019 09:30:27 GMT

Hello,

Where is your hadoop server? Did you install it in local? It looks like the problem is with the setting up the correct class path. What does your classpath look like?

Best regards

Sabrina

Re: TOS BD Error - package org.apache.hadoop.io does not exist

roalro — Fri, 17 May 2019 10:03:50 GMT

Hi.

Firstly thanks for your reply.

The hadoop server is not in local, it is in an another EC2 instance.

Precisely I have readed that the problem could be in the classpath, but how do i know what is the classpath wicht my Talend is using?

Thanks.

Re: TOS BD Error - package org.apache.hadoop.io does not exist

Anonymous — Wed, 11 Sep 2019 03:21:07 GMT

Hi Sabrina,

I have exactly same problem here. Could you please provide step-by-step solution to check classpath that Talend & Hadoop use? My Hadoop server is in another instance, not local. I'm use Talend Big Data & I have installed JDK1.8

[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration.
\tmp\sqoop-monica.baniwijaya\compile\f39416108dab4711b50081d45ac7281d\STG1_KN_F_PRODUCT.java:7: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.BytesWritable;
^
\tmp\sqoop-monica.baniwijaya\compile\f39416108dab4711b50081d45ac7281d\STG1_KN_F_PRODUCT.java:8: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Text;
^
\tmp\sqoop-monica.baniwijaya\compile\f39416108dab4711b50081d45ac7281d\STG1_KN_F_PRODUCT.java:9: error: package org.apache.hadoop.io does not exist
import org.apache.hadoop.io.Writable;
^
\tmp\sqoop-monica.baniwijaya\compile\f39416108dab4711b50081d45ac7281d\STG1_KN_F_PRODUCT.java:37: error: cannot access Writable
public class STG1_KN_F_PRODUCT extends SqoopRecord implements DBWritable, Writable {
^
class file for org.apache.hadoop.io.Writable not found
\tmp\sqoop-monica.baniwijaya\compile\f39416108dab4711b50081d45ac7281d\STG1_KN_F_PRODUCT.java:645: error: cannot find symbol
public void parse(Text __record) throws RecordParser.ParseError {
^
symbol: class Text
location: class STG1_KN_F_PRODUCT
\tmp\sqoop-monica.baniwijaya\compile\f39416108dab4711b50081d45ac7281d\STG1_KN_F_PRODUCT.java:124: error: cannot find symbol
this.LOB = LOB;
^
symbol: variable this
location: class STG1_KN_F_PRODUCT
\tmp\sqoop-monica.baniwijaya\compile\f39416108dab4711b50081d45ac7281d\STG1_KN_F_PRODUCT.java:127: error: cannot find symbol
this.LOB = LOB;
^
symbol: variable this
location: class STG1_KN_F_PRODUCT
\tmp\sqoop-monica.baniwijaya\compile\f39416108dab4711b50081d45ac7281d\STG1_KN_F_PRODUCT.java:128: error: cannot find symbol
return this;

Thank you

Re: TOS BD Error - package org.apache.hadoop.io does not exist

roalro — Tue, 24 Sep 2019 13:04:59 GMT

Hi fransiskakd.

Could you solve this problem?

Thanks.

Re: TOS BD Error - package org.apache.hadoop.io does not exist

Anonymous — Wed, 25 Sep 2019 10:51:28 GMT

Hello,

We don't find any related jira issue on talend bug tracker.

We met this issue before when using Sqoop to import data from a local database to HDFS.

package org.apache.hadoop.io does not exist

It seems Talend Studio is installed in the C:\Program Files (x86) path, so there is an empty character, and the Job can't find the correct JAR files.

Are you using Talend in Kerberos Environment?

Best regards

Sabrina