Error whit metadata when i try to execute a job: tsqoopimport

[Anonymous]

Hi guys.

I'm trying to execute a job . This job is very simple: is a component tsqoopimport.

I have to export data from a SQL Server to Hive. But when a execute the job, i have the follown error:

[ERROR]: org.apache.sqoop.Sqoop - Got exception running Sqoop: org.kitesdk.data.DatasetIOException: Cannot access descriptor location: hdfs://SERVER/.metadata
at org.apache.sqoop.manager.SQLServerManager.importQuery(SQLServerManager.java:405)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:509)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:615)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at local_project.prueba_dos_0_1.prueba_dos.tSqoopImport_2Process(prueba_dos.java:422)
at local_project.prueba_dos_0_1.prueba_dos.runJobInTOS(prueba_dos.java:701)
at local_project.prueba_dos_0_1.prueba_dos.main(prueba_dos.java:558)
Caused by: java.net.ConnectException: Call From DESKTOP-ALQ/ to SERVER:8020 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[statistics] disconnected
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1408)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy7.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:762)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy8.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2102)
at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1215)
at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1211)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1412)
at org.kitesdk.data.spi.filesystem.FileSystemMetadataProvider.checkExists(FileSystemMetadataProvider.java:561)
... 18 more
Caused by: java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:713)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1524)
at org.apache.hadoop.ipc.Client.call(Client.java:1447)
... 36 more
Exception in component tSqoopImport_2
java.lang.Exception: The Sqoop import job has failed. Please check the logs.
at local_project.prueba_dos_0_1.prueba_dos.tSqoopImport_2Process(prueba_dos.java:427)
at local_project.prueba_dos_0_1.prueba_dos.runJobInTOS(prueba_dos.java:701)
at local_project.prueba_dos_0_1.prueba_dos.main(prueba_dos.java:558)

I understand is a error with the metadata, but i don't know how to resolve. Thank you for your help!

[Anonymous]

Hello,

tSqoopImport calls Sqoop to transfer data from a relational database management system (RDBMS) such as MySQL or Oracle into the Hadoop Distributed File System (HDFS).

For your hive server, you can create an external table into hive which points out a HDFS location or use thiveload, thiverow component.

Best regards

Sabrina

[Anonymous]

Thank you Sabrina.

I was trying with a thiveload component but i have a problem: I loaded data in csv format , i need to load data in parquet format. The thiveload component does not have this option. Any ideas that how load data in parquet format with Talend?

load.JPG

[Anonymous]

HELLO,

It seems issue with your Hadoop and Sqoop's configuration.Sqoop component uses MapReduce,so you need to specify classpath variable HADOOP_HOME and HADOOP_CONF_DIR and SQOOP_HOME and SQOOP_CONF_DIR(in your .bashrc file in ubuntu).
Please try it and inform if resolved.

hope this was helpfull!

[Anonymous · Accepted solution]

Hello,

Are you able to load files into the HDFS first? And then you can use Hive to interpret such files as database files.

Best regards

Sabrina

[Anonymous]

Firts i loaded files into HDFS and then i used the thiveload to insert in the hive's database . Thank you Sabrina