Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
i'm trying to create very simple job to insert data from file to HDFS as below design
but i'm receiving error as below
Starting job fix at 07:27 14/12/2022.
[statistics] connecting to socket on port 3571
[statistics] connected
[WARN ] 07:27:05 org.apache.hadoop.util.Shell- Did not find winutils.exe: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/C:/Program%20Files%20(x86)/Talend-Studio/studio/configuration/.m2/repository/org/apache/hadoop/hadoop-auth/3.0.0-cdh6.1.1/hadoop-auth-3.0.0-cdh6.1.1.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[WARN ] 07:27:05 org.apache.hadoop.util.NativeCodeLoader- Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Abandoning BP-967283304-172.16.162.11-1668247158932:blk_1073789078_48377
[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Excluding datanode DatanodeInfoWithStorage[172.16.162.14:9866,DS-c57dcd1d-a671-4b85-9668-113523f3fddf,DISK]
[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Abandoning BP-967283304-172.16.162.11-1668247158932:blk_1073789079_48378
[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Excluding datanode DatanodeInfoWithStorage[172.16.162.18:9866,DS-8abb14a9-a3e7-4694-813e-0e6e7e51ffbd,DISK]
[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Abandoning BP-967283304-172.16.162.11-1668247158932:blk_1073789080_48379
[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Excluding datanode DatanodeInfoWithStorage[172.16.162.15:9866,DS-bffe88ba-dfbc-47b3-b212-591179a13f5b,DISK]
[WARN ] 07:27:07 org.apache.hadoop.hdfs.DataStreamer- Abandoning BP-967283304-172.16.162.11-1668247158932:blk_1073789081_48380
[WARN ] 07:27:07 org.apache.hadoop.hdfs.DataStreamer- Excluding datanode DatanodeInfoWithStorage[172.16.162.17:9866,DS-9bd0b5ea-dcdc-4f3b-a20a-c59cb0eef5c8,DISK]
Exception in component tHDFSOutput_1 (fix)
java.io.IOException: Could not get block locations. Source file "/user/devtest/talendoutput/2.txt" - Aborting...block==null
at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1477)
at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)
[WARN ] 07:27:07 org.apache.hadoop.hdfs.DataStreamer- DataStreamer Exception
java.io.IOException: Unable to create new block.
at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1694) ~[hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716) [hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]
[WARN ] 07:27:07 org.apache.hadoop.hdfs.DataStreamer- Could not get block locations. Source file "/user/devtest/talendoutput/2.txt" - Aborting...block==null
[FATAL] 07:27:07 uatproj.fix_0_1.fix- tHDFSOutput_1 Could not get block locations. Source file "/user/devtest/talendoutput/2.txt" - Aborting...block==null
java.io.IOException: Could not get block locations. Source file "/user/devtest/talendoutput/2.txt" - Aborting...block==null
at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1477) ~[hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]
at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256) ~[hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667) ~[hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]
[statistics] disconnected
Have you installed the WinUtils? The answer to this post may help you....
https://stackoverflow.com/questions/35652665/java-io-ioexception-could-not-locate-executable-null-bin-winutils-exe-in-the-ha/41584008
i'm using linux machine , is that matter ?
This looks like a configuration issue then. This is in your error.....That is a Windows path.
yes it's windows path as jump machine , hadoop is hosted on linux machine and i'm connecting it using kerberos as below connection
You are running that job on your Linux machine, right? It (the machine) will need to be set up to do this. Your keytab value is pointing to a Windows path. This file will need to be on the machine where you are running the job.
I also notice messages that suggest that access to files is not available. That could be because the files are not in the stated location OR (often more likely) that the user running the Studio does not have permission to access the locations that the files are in. In this case, a simple test is to stop the Studio and run it as root (or another user with enhanced access).
already running with admin access , i can access the files
however file is created successfully but cannot insert the data in it and file permission is chmod 777 but shows error unable to create new block
Have you corrected your keytab settings? It is currently showing a path for a Windows system. This path needs to be available to the machine you are running this on.