Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Wanted to understand talend modules.
In talend, while using tLibraryLoad or while managing library for routines, can see options to add jar wherein, its either already available under platform list or we can select 3rd party jar and install that as a module. Then classes from these modules can be used within the job. Snap Below:
Wanted to understand below regarding the talend modules:
I'm trying to use an external jar for one of my routine. But once this goes to Production, in a long run, when we upgrade talend verision or java version, will we explicitly have to remember to upgrade the external jar referenced in the routine, or since its under Platform/ or installed as module in talend, talend studio will notify on this when upgrading talend version.
could someone please advise on above.
Hello,
I'm not sure how zip4j versioning works. Maybe 1.x and 2.x can coexist just like log4j and slf4j. Lets have a few hypotetical scenarios:
I personally would try to align myself to the libs that Talend provides. (Even if this means adding some components that I never trigger) By staying on the latest patch one would automatically benefit from security upgrades. Even if this means that time to time my routine has to be adjusted.
Maybe via having dependabot checking the routines pom.xml or setting up a security scan on the final build inside CICD one could be notified about risky jars. Our jenkins pipeline can be easily expanded: https://help.talend.com/r/en-US/8.0/software-dev-lifecycle-best-practices-guide/ci-jenkins
hope this helps
Hello,
I'm glad you asked this question. The modules available in Talend can be viewed via the Moduels view: https://help.talend.com/r/en-US/Cloud/studio-user-guide-api-services-platform/installing-external-modules-manually-using-modules-view
For the Routine dependencies you have 2 options.
How does this Required checkbox work?
When the Required checkbox is enabled that means the dependency will be included to the final build. If it's unchecked then it will only be used to compile the routine. (Routines are complied separately and before the jobs built.) So how could one use such?
Lets say your routine is using AWS secret manager to retrieve secrets. As the S3 component brings the aws-java-sdk-bundle one could develop the routine by specifying the aws-java-sdk platform jar. If the Required checkbox is enabled then in the beginning there's no visible difference.
Lets fast forward 12 months. Talend S3 component now uses a newer jar. This jar might be only a security fix or might be conflicting with the one required by the routine. If the Required checkbox is unchecked that would mean your final job build will include the jar that as upgraded by the Talend patch.
Another option is to use the custom mvn uri mapping and override the MVN versions before the pom generation / build. https://help.talend.com/r/en-US/Cloud/studio-user-guide-api-services-platform/overriding-external-modules-by-customizing-mvn-uri
If your jar is not used by any of the components then you're left with no choice but to maintain your jar manually. (Detect when it has new versions / security fixes, and take appropriate action.)
Hope this helps.
Regards,
Balázs
Hi Balazs,
Great, this is what I wanted to understand.
Follow up question on the details you provided for better clarification:
Yes, it is correct. Hopefully the 2.11 is backward compatible with 1.3.3
Routine will look for the class from the classpath which depends on the jars your JVM loads. (which can be a problem when you have multiple jars providing the same class as in that case if the 1.3 is loaded sooner than the CVE / performance fixes of 2.x won't be visible.
For this external jar use-case I'd suggest to set up a reference project, and maintain it there. That way you'll be able to manage it centrally. And the rest of your projects will inherit it. We have custom routine jars that will generate a dedicated jar file instead of altering routines.jar, see: https://help.talend.com/r/en-US/Cloud/studio-user-guide-api-services-platform/creating-custom-routine-jars
Thanks for the explanation makes sense.
Now, in my case, zip4j version 1.3.3 needs to be used in a separate job, that job will only include class path to this jar version. There are other jobs which will use tFileArchive and those won't use the custom routine, so will only include the platform jar in the class path. With this, if in the future say Talend 8 if it uses zip4j - 2.10.0 , then that will only affect the jobs using the tFileArchive right, and the job which uses the routine could still use zip4j 1.3.3 ( keeping the required checked in routine library config ) - I'm writing this because for zip4j all versions > 2.0.0 is a major rewrite so the code using 1.3.3 won't work with > 2.0.0 without the changes in the routine. Is there anything that needs to be considered with this approach.
On the central management using referenced project and custom routine jar, this looks nice. Thanks for this.
Is there a way to update the maven setting in talend projects, so that it would notify availability of newer versions of dependent jar, so that I don't have to keep track of all the external dependencies outside the platform jars.
Hello,
I'm not sure how zip4j versioning works. Maybe 1.x and 2.x can coexist just like log4j and slf4j. Lets have a few hypotetical scenarios:
I personally would try to align myself to the libs that Talend provides. (Even if this means adding some components that I never trigger) By staying on the latest patch one would automatically benefit from security upgrades. Even if this means that time to time my routine has to be adjusted.
Maybe via having dependabot checking the routines pom.xml or setting up a security scan on the final build inside CICD one could be notified about risky jars. Our jenkins pipeline can be easily expanded: https://help.talend.com/r/en-US/8.0/software-dev-lifecycle-best-practices-guide/ci-jenkins
hope this helps
Thanks a lot, really helpful.
We're also trying to build a CI pipeline with Jenkins, probably will get back on that if need any assistance.