[resolved] Error Running PIG in Talend Sandbox

Anonymous — Sat, 16 Nov 2024 10:27:27 GMT

Hi
I am getting the below error when trying to run a simple PIG Job. This just loads and stores the files
Starting job Test_Pig1 at 22:01 15/08/2016.
connecting to socket on port 3912
connected
: org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
: org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
: org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox:8020
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
: org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
: org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
: org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
: org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=}
: org.apache.hadoop.conf.Configuration.deprecation - mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
: org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at sandbox/192.168.255.136:8050
: org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
: org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
: org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job2768210825357464261.jar
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job2768210825357464261.jar created
: org.apache.hadoop.conf.Configuration.deprecation - mapred.jar is deprecated. Instead, use mapreduce.job.jar
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
: org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.http.address is deprecated. Instead, use mapreduce.jobtracker.http.address
: org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at sandbox/192.168.255.136:8050
: org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
: org.apache.hadoop.mapreduce.JobSubmitter - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
: org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 4
: org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 4
: org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
: org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
: org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1471297391392_0002
: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1471297391392_0002
: org.apache.hadoop.mapreduce.Job - The url to track the job:
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1471297391392_0002
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases tPigLoad_1_row1_RESULT
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: tPigLoad_1_row1_RESULT C: R:
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_1471297391392_0002 has failed! Stop running all dependent jobs
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
: org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
: org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.4.0.2.1.1.0-385 0.12.1.2.1.1.0-385 talend 2016-08-15 22:01:14 2016-08-15 22:01:33 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
job_1471297391392_0002 tPigLoad_1_row1_RESULT MAP_ONLY Message: Job failed! hdfs://sandbox:8020/user/talend/airplanes_cp1,
Input(s):
Failed to read data from "hdfs://sandbox:8020/user/talend/clickstream_demo/input"
Output(s):
Failed to produce result in "hdfs://sandbox:8020/user/talend/airplanes_cp1"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1471297391392_0002

: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
disconnected
Job Test_Pig1 ended at 22:01 15/08/2016.

Re: [resolved] Error Running PIG in Talend Sandbox

Anonymous — Tue, 16 Aug 2016 07:28:32 GMT

Hi,
Could you please post your job setting screenshot into forum which will be helpful for us to address your issue quickly?
Best regards
Sabrina

Re: [resolved] Error Running PIG in Talend Sandbox

Anonymous — Tue, 16 Aug 2016 17:12:57 GMT

Hi Sabrina
Thanks for looking into this, Please find attached the screenshots. First 2 are for Load and 3rd is for store
Regards
Raj

Re: [resolved] Error Running PIG in Talend Sandbox

Anonymous — Tue, 16 Aug 2016 17:15:06 GMT

screenshots
screenshots.pdf

Re: [resolved] Error Running PIG in Talend Sandbox

Anonymous — Tue, 16 Aug 2016 18:07:27 GMT

Hi
This is resolved. I have added the below Hadoop Property and it got fixed
Property : "yarn.application.classpath"
value : "/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*,/usr/hdp/current/hadoop-mapreduce-client/lib/*,/usr/hdp/current/hadoop-mapreduce-client/*"

Re: [resolved] Error Running PIG in Talend Sandbox

Anonymous — Thu, 18 Aug 2016 09:39:15 GMT

Hi,
Thanks for your prompt reply and sharing your solution with us.
Best regards
Sabrina

topic Re: [resolved] Error Running PIG in Talend Sandbox in Talend Studio

[resolved] Error Running PIG in Talend Sandbox

Re: [resolved] Error Running PIG in Talend Sandbox

Re: [resolved] Error Running PIG in Talend Sandbox

Re: [resolved] Error Running PIG in Talend Sandbox

Re: [resolved] Error Running PIG in Talend Sandbox

Re: [resolved] Error Running PIG in Talend Sandbox