[resolved] Error Running PIG in Talend Sandbox

One Star

[resolved] Error Running PIG in Talend Sandbox

Hi
I am getting the below error when trying to run a simple PIG Job. This just loads and stores the files
Starting job Test_Pig1 at 22:01 15/08/2016.
connecting to socket on port 3912
connected
: org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
: org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
: org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://sandbox:8020
: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
: org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
: org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
: org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: UNKNOWN
: org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=}
: org.apache.hadoop.conf.Configuration.deprecation - mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
: org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at sandbox/192.168.255.136:8050
: org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job
: org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
: org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - creating jar file Job2768210825357464261.jar
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - jar file Job2768210825357464261.jar created
: org.apache.hadoop.conf.Configuration.deprecation - mapred.jar is deprecated. Instead, use mapreduce.job.jar
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
: org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker.http.address is deprecated. Instead, use mapreduce.jobtracker.http.address
: org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at sandbox/192.168.255.136:8050
: org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
: org.apache.hadoop.mapreduce.JobSubmitter - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
: org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 4
: org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 4
: org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
: org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
: org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_1471297391392_0002
: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1471297391392_0002
: org.apache.hadoop.mapreduce.Job - The url to track the job:
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1471297391392_0002
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases tPigLoad_1_row1_RESULT
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: tPigLoad_1_row1_RESULT C:  R: 
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_1471297391392_0002 has failed! Stop running all dependent jobs
: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
: org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
: org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics: 
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.4.0.2.1.1.0-385 0.12.1.2.1.1.0-385 talend 2016-08-15 22:01:14 2016-08-15 22:01:33 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
job_1471297391392_0002 tPigLoad_1_row1_RESULT MAP_ONLY Message: Job failed! hdfs://sandbox:8020/user/talend/airplanes_cp1,
Input(s):
Failed to read data from "hdfs://sandbox:8020/user/talend/clickstream_demo/input"
Output(s):
Failed to produce result in "hdfs://sandbox:8020/user/talend/airplanes_cp1"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1471297391392_0002

: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
disconnected
Job Test_Pig1 ended at 22:01 15/08/2016.

Accepted Solutions
One Star

Re: [resolved] Error Running PIG in Talend Sandbox

Hi
This is resolved. I have added the below Hadoop Property and it got fixed
Property : "yarn.application.classpath" 
value : "/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*,/usr/hdp/current/hadoop-mapreduce-client/lib/*,/usr/hdp/current/hadoop-mapreduce-client/*"

All Replies
Moderator

Re: [resolved] Error Running PIG in Talend Sandbox

Hi,
Could you please post your job setting screenshot into forum which will be helpful for us to address your issue quickly?
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: [resolved] Error Running PIG in Talend Sandbox

Hi Sabrina
Thanks for looking into this, Please find attached the screenshots. First 2 are for Load and 3rd is for store
Regards
Raj
One Star

Re: [resolved] Error Running PIG in Talend Sandbox

screenshots
screenshots.pdf
One Star

Re: [resolved] Error Running PIG in Talend Sandbox

Hi
This is resolved. I have added the below Hadoop Property and it got fixed
Property : "yarn.application.classpath" 
value : "/usr/hdp/current/hadoop-client/*,/usr/hdp/current/hadoop-client/lib/*,/usr/hdp/current/hadoop-hdfs-client/*,/usr/hdp/current/hadoop-hdfs-client/lib/*,/usr/hdp/current/hadoop-yarn-client/*,/usr/hdp/current/hadoop-yarn-client/lib/*,/usr/hdp/current/hadoop-mapreduce-client/lib/*,/usr/hdp/current/hadoop-mapreduce-client/*"
Moderator

Re: [resolved] Error Running PIG in Talend Sandbox

Hi,
Thanks for your prompt reply and sharing your solution with us.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.