Pig UDF java.lang.NoSuchMethodError for writing custom UDFs.

Anonymous — Sat, 16 Nov 2024 11:26:13 GMT

Hi TalendForge,
I'm using 5.4.
I can successfully use the UPPER Pig UDF in a tMap like the following:

However, if I make my own Pig UDF using Talend's Eval template, which simply sets a string to uppercase, like:

public class TestUDF extends EvalFunc<String> {
    public String exec(Tuple input) throws IOException {
        if (input == null || input.size() == 0) {
            return null;
        }
        try {
            String str = (String) input.get(0);
            return str.toUpperCase();
        } catch (Exception e) {
            throw new IOException("Caught exception processing input row ", e);
        }
    }
}

And change the tMap to use pigudf.TestUDF(row1.newColumn):

I then get the following error:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:261)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:180)
	at org.apache.pig.PigServer.launchPlan(PigServer.java:1270)
	at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1255)
	at org.apache.pig.PigServer.execute(PigServer.java:1245)
	at org.apache.pig.PigServer.executeBatch(PigServer.java:362)
	at primehome.pigudf2_0_1.PigUDF2.tPigLoad_1Process(PigUDF2.java:757)
	at primehome.pigudf2_0_1.PigUDF2.runJobInTOS(PigUDF2.java:1071)
	at primehome.pigudf2_0_1.PigUDF2.main(PigUDF2.java:936)
 disconnected
Job PigUDF2 ended at 13:29 04/11/2014.

In fact, if I change the UDF back to UPPER which originally worked, the job will still fail with the same error. I have to create a new job to get rid of this error.
Regards,
Matthew

Re: Pig UDF java.lang.NoSuchMethodError for writing custom UDFs.

Anonymous — Sun, 09 Nov 2014 12:48:17 GMT

Hello,
Can you please create a JIRA issue in our issue tracker ? https://jira.talendforge.org
You can create it in the TBD project.
Thanks in advance,
Regards,
Rémy.

Re: Pig UDF java.lang.NoSuchMethodError for writing custom UDFs.

Anonymous — Tue, 18 Nov 2014 09:32:47 GMT

Hi,
It seems this problem comes from a classloading issue. If you right click on the UDF you created in the repository, you can see the UDF dependencies. By default, Talend uses pig-0.10.jar in order to compile the UDF. The problem is that the cluster you work on uses a more recent version of Pig. You should update it with the correct Pig version.
HTH,
Rémy.

topic Re: Pig UDF java.lang.NoSuchMethodError for writing custom UDFs. in Talend Studio

Pig UDF java.lang.NoSuchMethodError for writing custom UDFs.

Re: Pig UDF java.lang.NoSuchMethodError for writing custom UDFs.

Re: Pig UDF java.lang.NoSuchMethodError for writing custom UDFs.