Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.

Anonymous — Sat, 16 Nov 2024 09:50:18 GMT

Hi,

I encountered the exception below when I'm running a a pig component on Talend. Any ideas on what's the cause of exception?

============================================================================================

Starting job aggregate_movie_director at 16:19 03/05/2017.

[statistics] connecting to socket on port 3572
[statistics] connected
[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.pig.PigServer - Empty string specified for jar path
[WARN ]: org.apache.hadoop.mapreduce.JobResourceUploader - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
[WARN ]: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
[WARN ]: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - There is no log file to write to.
[ERROR]: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher - Backend error message
Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
   at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
   at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
   at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:87)
   at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initialize(PigRecordReader.java:192)
   at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found
   at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
   at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
   ... 11 more

[ERROR]: org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate exception from backed error: Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
   at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
   at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
   at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:87)
   at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader.initialize(PigRecordReader.java:192)
   at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
   at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
   at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
   at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:415)
   at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
   at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found
   at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
   at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
   ... 11 more

[ERROR]: org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
[statistics] disconnected
Job aggregate_movie_director ended at 16:20 03/05/2017. [exit code=0]

============================================================================================

Please help me to investigate.

Re: Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.

Anonymous — Tue, 09 May 2017 08:45:13 GMT

Hi,

Have you already installed all required jar files for your pig job? Which talend build version you are using?

Here is a related jira issue:https://jira.talendforge.org/browse/TBD-3527.

Best regards

Sabrina

Re: Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.

Anonymous — Mon, 15 May 2017 03:55:56 GMT

Hi Sabrina,

First of all, thank you so much for your suggestion.

By the way the talend build version that I use is "TOS_BD-20161216_1026-V6.3.1".

And please tell me, how could I know what's the required jar files for this version ?

Best Regards.

Thank you.

Re: Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.

Anonymous — Mon, 15 May 2017 04:39:44 GMT

Hello,

The Modules view lists all the modules required to use the components embedded in the Studio, including those Java libraries and drivers that you must install to get the relevant components or Metadata connection working.

Could you please check if there is any required jar files for your job?

Best regards

Sabrina

1.png

topic Re: Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found. in Talend Studio

Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.

Re: Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.

Re: Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.

Re: Error: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.