tPigLoad cannot load any file from HDFS, it is getting the null value
Hi, Guys
I am using your talend big data open studio to testing for big data, and running into an issue
When I tried to use tPigLoad to load file in HDFS, it gives the error please see the screen shot.
I have tried every solution in google, but none does work.
this is my testing schema in hdfs
please help me. Thank you!
I don't know why the screen shot is missed
please see the error
Starting job T_Pigload_Test at 03:08 10/08/2015. connecting to socket on port 3587
connected
connecting to socket on port 4700
connected
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
objc: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home/bin/java and /Library/Java/JavaVirtualMachines/jdk1.8.0_11.jdk/Contents/Home/jre/lib/libinstrument.dylib. One of the two will be used. Which one is undefined.
: org.apache.hadoop.mapreduce.JobSubmitter - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
: org.apache.hadoop.mapreduce.JobSubmitter - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
: org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed!
disconnected
disconnected
Job T_Pigload_Test ended at 03:09 10/08/2015.
FYI, in the document from talend open big studio, I can successfully run the first 3 example A_HCatalog_Create B_HCatalog_Load C_HCatalog_Read I have no issue about connect to HCatalog, HDFS and namenode. I setup my own vm cluster and can run the pig command inside the pig shell (grunt) in the cluster. Just cannot trigger the tPigLoad component in talend to the cluster, I believe the connection to namenode from talend to hadoop cluster is good. Can someone help me? Thank you so much.
I'm having the same issue - same stack trace. I don't seem able to post screenshots, but my flow is essentially: tRowGenerator -> tPigLoad --> tPigCode --> tPigStoreResult If I disable the StoreResult component, I get no errors...
Talend version 6.1.1
TPigload- tpigstoreresult
Reading a file from hdfs in mapreduce mode and storing the results in a different directory in hdfs
Can you please help? It is urgentt
Hi kswamina,
Are you able to connect to HDFS? What's your OS and hadoop version?
Is your Talend Studio Java version is the same as the cluster? 1.8? More information will be preferred.
Best regards
Sabrina
Hi Sabrina,
Yes, I am able to connect to HDFS. I am using
windows 7 enterprise. Hadoop version is
2.7.1. Java version
1.8.0_65 in both talend and cluster.
Namenode URI "hdfs://10.56.70.93:9000"
Resource Manager "10.56.70.93:8032"
Resource Manager Scheduler Address: "10.56.70.93:8030"
If I remove the tPigStoreResult component from the job, i don't get any error.
Please helpp. If you need more information, please let me know
thanks
Krithika