topic Re: java.io.ioexception unable to create new block talend in Talend Studio

java.io.ioexception unable to create new block talend

mshafeeq — Fri, 15 Nov 2024 22:16:28 GMT

i'm trying to create very simple job to insert data from file to HDFS as below design

but i'm receiving error as below

Starting job fix at 07:27 14/12/2022.

[statistics] connecting to socket on port 3571

[statistics] connected

[WARN ] 07:27:05 org.apache.hadoop.util.Shell- Did not find winutils.exe: java.io.FileNotFoundException: java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset. -see https://wiki.apache.org/hadoop/WindowsProblems

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/C:/Program%20Files%20(x86)/Talend-Studio/studio/configuration/.m2/repository/org/apache/hadoop/hadoop-auth/3.0.0-cdh6.1.1/hadoop-auth-3.0.0-cdh6.1.1.jar) to method sun.security.krb5.Config.getInstance()

WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

[WARN ] 07:27:05 org.apache.hadoop.util.NativeCodeLoader- Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Abandoning BP-967283304-172.16.162.11-1668247158932:blk_1073789078_48377

[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Excluding datanode DatanodeInfoWithStorage[172.16.162.14:9866,DS-c57dcd1d-a671-4b85-9668-113523f3fddf,DISK]

[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Abandoning BP-967283304-172.16.162.11-1668247158932:blk_1073789079_48378

[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Excluding datanode DatanodeInfoWithStorage[172.16.162.18:9866,DS-8abb14a9-a3e7-4694-813e-0e6e7e51ffbd,DISK]

[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Abandoning BP-967283304-172.16.162.11-1668247158932:blk_1073789080_48379

[WARN ] 07:27:06 org.apache.hadoop.hdfs.DataStreamer- Excluding datanode DatanodeInfoWithStorage[172.16.162.15:9866,DS-bffe88ba-dfbc-47b3-b212-591179a13f5b,DISK]

[WARN ] 07:27:07 org.apache.hadoop.hdfs.DataStreamer- Abandoning BP-967283304-172.16.162.11-1668247158932:blk_1073789081_48380

[WARN ] 07:27:07 org.apache.hadoop.hdfs.DataStreamer- Excluding datanode DatanodeInfoWithStorage[172.16.162.17:9866,DS-9bd0b5ea-dcdc-4f3b-a20a-c59cb0eef5c8,DISK]

Exception in component tHDFSOutput_1 (fix)

java.io.IOException: Could not get block locations. Source file "/user/devtest/talendoutput/2.txt" - Aborting...block==null

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1477)

at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256)

at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667)

[WARN ] 07:27:07 org.apache.hadoop.hdfs.DataStreamer- DataStreamer Exception

java.io.IOException: Unable to create new block.

at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1694) ~[hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]

at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716) [hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]

[WARN ] 07:27:07 org.apache.hadoop.hdfs.DataStreamer- Could not get block locations. Source file "/user/devtest/talendoutput/2.txt" - Aborting...block==null

[FATAL] 07:27:07 uatproj.fix_0_1.fix- tHDFSOutput_1 Could not get block locations. Source file "/user/devtest/talendoutput/2.txt" - Aborting...block==null

java.io.IOException: Could not get block locations. Source file "/user/devtest/talendoutput/2.txt" - Aborting...block==null

at org.apache.hadoop.hdfs.DataStreamer.setupPipelineForAppendOrRecovery(DataStreamer.java:1477) ~[hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]

at org.apache.hadoop.hdfs.DataStreamer.processDatanodeOrExternalError(DataStreamer.java:1256) ~[hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]

at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:667) ~[hadoop-hdfs-client-3.0.0-cdh6.1.1.jar:?]

[statistics] disconnected

Re: java.io.ioexception unable to create new block talend

Anonymous — Wed, 14 Dec 2022 18:14:40 GMT

Have you installed the WinUtils? The answer to this post may help you....

https://stackoverflow.com/questions/35652665/java-io-ioexception-could-not-locate-executable-null-bin-winutils-exe-in-the-ha/41584008

Re: java.io.ioexception unable to create new block talend

mshafeeq — Thu, 15 Dec 2022 09:40:56 GMT

i'm using linux machine , is that matter ?

Re: java.io.ioexception unable to create new block talend

Anonymous — Thu, 15 Dec 2022 11:37:56 GMT

This looks like a configuration issue then. This is in your error.....That is a Windows path.

Re: java.io.ioexception unable to create new block talend

mshafeeq — Thu, 15 Dec 2022 13:32:26 GMT

yes it's windows path as jump machine , hadoop is hosted on linux machine and i'm connecting it using kerberos as below connection

Re: java.io.ioexception unable to create new block talend

Anonymous — Thu, 15 Dec 2022 14:23:01 GMT

You are running that job on your Linux machine, right? It (the machine) will need to be set up to do this. Your keytab value is pointing to a Windows path. This file will need to be on the machine where you are running the job.

Re: java.io.ioexception unable to create new block talend

Anonymous — Thu, 15 Dec 2022 14:24:50 GMT

I also notice messages that suggest that access to files is not available. That could be because the files are not in the stated location OR (often more likely) that the user running the Studio does not have permission to access the locations that the files are in. In this case, a simple test is to stop the Studio and run it as root (or another user with enhanced access).

Re: java.io.ioexception unable to create new block talend

mshafeeq — Thu, 15 Dec 2022 18:48:38 GMT

already running with admin access , i can access the files

however file is created successfully but cannot insert the data in it and file permission is chmod 777 but shows error unable to create new block

Re: java.io.ioexception unable to create new block talend

Anonymous — Thu, 15 Dec 2022 19:46:15 GMT

Have you corrected your keytab settings? It is currently showing a path for a Windows system. This path needs to be available to the machine you are running this on.