topic Add Dependencies for Routine in Talend Studio

Add Dependencies for Routine

ankushdeshpande — Sat, 16 Nov 2024 03:54:46 GMT

Hello All,

I have written a routine to run a Java code to read a file and store as RDD. i have added few Spark & Scala dependencies using "Edit Routine Libraries"

When i run the component tJava with call to routine, it fails with classNotFound Exception. once i add that particular Dependency, the error goes away but job fails for another ClassNotFound error.

I already have all the JARS present in "\.m2\repository" path, so not sure how a routine can use all existing JARS.

Routine :

package routines;

import org.apache.spark.SparkConf;
import org.apache.spark.api.java.JavaRDD;
import org.apache.spark.api.java.JavaSparkContext;

public class file_to_rdd {

public static void main(String filepath) {
// configure spark
SparkConf sparkConf = new SparkConf();

// start a spark context
JavaSparkContext sc = new JavaSparkContext(sparkConf);

// provide path to input text file
String path = filepath;

// read text file to RDD
JavaRDD<String> lines = sc.textFile(path);

// collect RDD for printing
for(String line:lines.collect()){
System.out.println(line);
}
}

}

Re: Add Dependencies for Routine

Anonymous — Mon, 09 Dec 2019 06:58:32 GMT

Hello,

In studio, right click on routine and select 'Edit routine libraries' option to add external libraries to your routine.

With talend subscription solution, you can open the configuration page in TAC and define the User libraries parameters to specify a repository URL for storing all libraries. In this way, your jar files should be loaded automatically to Nexus after you configure the parameters and the dependency libraries are exported also when deploy the job to remote Job server.

Let us know if it helps.

Best regards

Sabrina

Re: Add Dependencies for Routine

Anonymous — Wed, 11 Dec 2019 04:36:58 GMT

Hi,

An example about how to add jars to routines can be referred from below link.

https://community.talend.com/t5/Design-and-Development/Talend-and-Amazon-Comprehend-Integration/ta-p/144048#toc-hId-1940395967

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved