java.io.ioexception unable to create new block talend

[mshafeeq]

i'm trying to create very simple job to insert data from file to HDFS as below design

but i'm receiving error as below

Starting job fix at 07:27 14/12/2022.

[statistics] connecting to socket on port 3571

[statistics] connected

[WARN ] 07:27:05 org.apache.hadoop.util.Shell- Did not find winutils.exe: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/C:/Program%20Files%20(x86)/Talend-Studio/studio/configuration/.m2/repository/org/apache/hadoop/hadoop-auth/3.0.0-cdh6.1.1/hadoop-auth-3.0.0-cdh6.1.1.jar) to method sun.security.krb5.Config.getInstance()

WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

[WARN ] 07:27:05 org.apache.hadoop.util.NativeCodeLoader- Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Abandoning BP-967283304-172.16.162.11-1668247158932:blk_1073789078_48377

[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Excluding datanode DatanodeInfoWithStorage[172.16.162.14:9866,DS-c57dcd1d-a671-4b85-9668-113523f3fddf,DISK]

[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Abandoning BP-967283304-172.16.162.11-1668247158932:blk_1073789079_48378

[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Excluding datanode DatanodeInfoWithStorage[172.16.162.18:9866,DS-8abb14a9-a3e7-4694-813e-0e6e7e51ffbd,DISK]

[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Abandoning BP-967283304-172.16.162.11-1668247158932:blk_1073789080_48379

[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Excluding datanode DatanodeInfoWithStorage[172.16.162.15:9866,DS-bffe88ba-dfbc-47b3-b212-591179a13f5b,DISK]

[WARN ] 07:27:07 org.apache.hadoop.hdfs.DataStreamer- Abandoning BP-967283304-172.16.162.11-1668247158932:blk_1073789081_48380

[WARN ] 07:27:07 org.apache.hadoop.hdfs.DataStreamer- Excluding datanode DatanodeInfoWithStorage[172.16.162.17:9866,DS-9bd0b5ea-dcdc-4f3b-a20a-c59cb0eef5c8,DISK]

Exception in component tHDFSOutput_1 (fix)

java.io.IOException: Could not get block locations. Source file "/user/devtest/talendoutput/2.txt" - Aborting...block==null

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1477)

at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)

at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)

[WARN ] 07:27:07 org.apache.hadoop.hdfs.DataStreamer- DataStreamer Exception

java.io.IOException: Unable to create new block.

at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1694) ~[hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]

at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716) [hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]

[WARN ] 07:27:07 org.apache.hadoop.hdfs.DataStreamer- Could not get block locations. Source file "/user/devtest/talendoutput/2.txt" - Aborting...block==null

[FATAL] 07:27:07 uatproj.fix_0_1.fix- tHDFSOutput_1 Could not get block locations. Source file "/user/devtest/talendoutput/2.txt" - Aborting...block==null

java.io.IOException: Could not get block locations. Source file "/user/devtest/talendoutput/2.txt" - Aborting...block==null

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1477) ~[hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]

at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256) ~[hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]

at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667) ~[hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]

[statistics] disconnected

[Anonymous]

Have you installed the WinUtils? The answer to this post may help you....

https://stackoverflow.com/questions/35652665/java-io-ioexception-could-not-locate-executable-null-bin-winutils-exe-in-the-ha/41584008

[mshafeeq]

i'm using linux machine , is that matter ?

[Anonymous]

This looks like a configuration issue then. This is in your error.....That is a Windows path.

[mshafeeq]

yes it's windows path as jump machine , hadoop is hosted on linux machine and i'm connecting it using kerberos as below connection

[Anonymous]

You are running that job on your Linux machine, right? It (the machine) will need to be set up to do this. Your keytab value is pointing to a Windows path. This file will need to be on the machine where you are running the job.

[Anonymous]

I also notice messages that suggest that access to files is not available. That could be because the files are not in the stated location OR (often more likely) that the user running the Studio does not have permission to access the locations that the files are in. In this case, a simple test is to stop the Studio and run it as root (or another user with enhanced access).

[mshafeeq]

already running with admin access , i can access the files

however file is created successfully but cannot insert the data in it and file permission is chmod 777 but shows error unable to create new block

[Anonymous]

Have you corrected your keytab settings? It is currently showing a path for a Windows system. This path needs to be available to the machine you are running this on.