Add maven dependency in Talend studio

[selimseddiki]

Hi,

I raised one more time the issue reported in :

https://community.talend.com/s/feed/0D53p00007vCmNWCA0?language=en_US

back in 2019.

and in my own ticket back in 2021 :

https://community.talend.com/s/feed/0D55b00006Xz3VJCAZ

I must add a maven dependency :

<dependency>

<groupId>com.files</groupId>

<artifactId>files-java</artifactId>

<version>$VERSION</version>

</dependency>

to use the SDK provided by the Files.com platform :

https://developers.files.com/# sdks (java SDK)

https://github.com/Files-com/files-sdk-java

I did it in the project setting, in a subfolder, according to :

https://help.talend.com/r/QrWkwPvKmKybs~JNLeBt3Q/zPYTS4PLhRoEK_Mj8ztaJw

I applied and closed. I then created a job in this subfolder and try to "

import com.files.FilesClient"

with a tJava component

but the dependency can not be resolved.

It is quite critical for our project, and I don't find any solution of the community.

Can we have a call to review this please ?

Best regards

[Anonymous]

Hello @selim seddiki​ ,

I tried the below steps:

1. download the files-sdk-1.0.225.jar and import it by Windows->show view ->Talend->Modules view in studio
2. add the below maven dependency in project settings -> Maven->default->Standalone job

3 create a new job , add tLibraryLoad component to load the jar files-sdk-1.0.225.jar, and add tJavaRow component, input the below line " import com.files.FilesClient;" in it's advanced settings, it's working fine. no compile error,

could you please try the above steps to see if it work in your side?

Thanks

Aiming

[Anonymous]

Hello,

On the long run you'll be better to add these dependencies to a routine. (I'd suggest to use the new Routine jar option: https://help.talend.com/r/en-US/8.0/studio-user-guide/creating-custom-routine-jars also available for 7.3 )

Download the pom.xml for this jar and add that xml to your routine (needs Talend 8 enterprise) if the dependencies for that component is properly registered in the pom.xml Studio will add its dependencies. If one of those dependencies also have dependencies, that still have to be added. ( https://help.talend.com/r/en-US/8.0/studio-user-guide/editing-user-routine-libraries )

That way you can minimize the Java code in your tJava* components.

+ routines have the abilty to do caching / can be shared between jobs (Just define a private static java.util.Map in your routine and that Map will have 1 single copy inside the JVM)

So you can store the connections, tokens, etc.

Then these routines can be easily upgraded / bugfixed as you can have a simple "contract" from the Talend job(s).

tJavaFlex:

for(com.files.files-java.File file : filesComRoutine.listFiles(context.user , context.password , "folderID" ) {

output_row.fileName = file.getName();

....

}

if later you have to change something then you can do it centrally in the routine.

Alternatively you can also add this jar as a tLibraryLoad.

[selimseddiki]

Hi Aiming,

Thank you for your feedback ! I can try that.

But to me, let me know if I misunderstood something, the ability to add maven dependencies in project settings would be precisely to avoid having to manually import the .jar from a local folder. This manual import + local reference to the Files.com SDK library would have to be made on every new studios that Marc Jacobs will ever have to install for the purpose of potentially large team work.

Could you please comment on that ? In the meantime, I can try the steps, and put the result here for other readers.

Thank you very much for your support.

Best regards, Sélim

[selimseddiki]

Hi,

Thank you so much for your support too !

We are currently using Talend studio 7.3.1.20200219_1130.

The solution you describe would certainly better fits Marc Jacobs requirements though. Having a remote reference to the Files.com SDK, via dependency definition in a POM file, makes the development process way more industrializable, especially in case of a potentially large team working on the Talend project.

My first question would then be : is there a workaround with Talend studio 7.3.1.20200219_1130 to achieve the same result ?

Best regards, Sélim

[Anonymous]

Make sure you apply the monthly patch (even for 7.3)

Based on your other response, I'd also consider:

Create a reference project.

Add the routine jar + libraries there. (Create some test jobs / test cases so you can validate those in-place.)

Add this reference project to your main projects and you'll have a central single place to maintain your routine and its dependencies.

I think this is the best you can have.

(You can also create custom components, we have Talend Component Kit framework available for that: https://talend.github.io/component-runtime/main/latest/index.html )

[selimseddiki]

Hi,

we will migrate Talend studio from version 7.3.1.20200219_1130 to version 8.0.1 this week. I will try the method you describe starting next week on so. In the meantime, could please answer a few more questions :

1/ "Download the pom.xml for this jar and add that xml to your routine" => I have to add maven dependency as described here : https://github.com/Files-com/files-sdk-java, using the method described here : https://help.talend.com/r/en-US/8.0/open-studio-user-guide/customizing-global-build-script-templates. Shall I add to the routine the Talend POM complemented with Files.com maven dependency ?

2/ "If one of those dependencies also have dependencies, that still have to be added." => Does this mean, if Files.com JDK have POM files with their own dependencies, these have to be added as well in the Talend routine ?

3/ Could you please provide a code example for defining a private static java.util.Map in routine to share connections, tokens ?

4/ Could you please provide a code example for the routines' method filesComRoutine.listFiles ?

5/ "Alternatively you can also add this jar as a tLibraryLoad"

 => but adding the jar, instead of defining dependencies via POM file direct dependencies, would then mean having to manually put the .jar either on a local or remote repository, right ? 

Thank you so much for your support. Having these questions answered we can start testing and let you know here the result for other potential interested Talend users.

Best regards, Sélim

[Anonymous]

I did a quick test:

Download the pom from here: https://mvnrepository.com/artifact/com.files/files-sdk/1.0.225

Add it to a routine:

Hit okay and:

 

As Talend brings some of these API-s we can uncheck the Required for a few.

Require checkbox basically tells studio whether a dependency is required for the job execution. Unchecked means it is needed for the routine compilation but the final execution will include a jar from somewhere else.

So if we want to avoid CVE-s (I'm sure we do. ) I'd say log4j-core is to uncheck and let talend bring its own.

I think okhttp3 also has some of its own dependencies so we might want to re-add that to our routine (and that way it brings its dependencies)

As you can see it's not perfect but much faster than browsing for all those jars.

[selimseddiki]

Hi,

I have started testing. I downloaded the POM file files-sdk-1.0.225.pom from the link, created a custom routine jar, add the POM in the custom routine jar, created a routine inside and started my class. But the com.files library is not resolved.

Please find below the libraries imported (log4j-core unchecked as you advised) + the import statement in my class :

I can find the direct dependencies defined in the POM file in the module view. But not the com.files library itself. Do I have to do something more ?

[Anonymous]

Hmm, that's weird. Maybe you discovered a bug? (Feel free to report this to support so they can engage R&D to investigate.)

You're right that com.files should be there. I didn't catch that.

For now you can add the com.files jar again (as all the dependencies are there) it should start to work.

The jar itself has the pom.xml embedded so maybe by adding the jar we get the jar + dependencies but by adding the pom we only get the dependencies?