Talend, Jenkins, and Nexus snapshot to release best practice
We are currently upgrading from Talend 6.x to 7.3 and I'm setting up continuous integration. After significant effort for a CI novice, I successfully set up a Jenkins pipeline to compile and deploy all project jobs to Nexus snapshot repository. Now we are trying to figure out our process to deploy artifacts to the releases repository.
Talend documentation seems to be scattered, conflicting, outdated, and/or incomplete throughout my experience when installing and setting up 7.3. I've looked over many different community forum postings, community documents, and Talend help documents (namely the Talend Software Development Life Cycle Best Practices Guide 7.3). Nothing seems to give a full picture of how we should handle promotion from snapshot to releases.
I created a pipeline that pointed to the release repo and received the error that it violated the version policy (because it should not have -SNAPSHOT at the end). Am I supposed to modify the project settings within Talend Studio (Build > Maven > Deployment Versioning) for all jobs and uncheck "Use snapshot" before deploying to release? I read some help.talend.com articles that -SNAPSHOT is supposed to automatically add/remove itself but it didn't really explain further. Am I supposed to modify job versions individually within Studio and deploy/Publish within Studio one job at a time? I've seen some community docs that say that's a no-no and some help docs that suggest to do it.
What I would like to do is what one article suggested: It is recommended that you publish directly to the Nexus SNAPSHOT folder when in development, and move to the RELEASE folder once the Job is stable. However, it didn't tell me how to do that.
This is frustrating for a novice to say the least. Any help, direction, or advice would be greatly appreciated and thank you in advance.
Here are some of the web pages I have used for research:
After talking with support, there wasn't a real clear best practice, which is understandable since it depends on the project. However deploying from Studio is not a no-no, contrary to the best practices article I read.
For our project we are going to be using Studio to push individually run jobs to a Nexus snapshot. If there are many dependent jobs/items, we will use a Jenkins pipeline to build the whole project to a Nexus snapshot.
We will be creating a release branch and a release Jenkins pipeline that builds from it. At certain milestones, we will 1) tag the master branch, 2) merge the master into the release branch, 3) modify all jobs to not be a snapshot via Project Setting's Deployment Versioning, 4) run the release pipeline, and 5) up the version for all jobs in master branch via Deployment Versioning.
I will leave the posting open for comments and/or advice.
After talking with support, there wasn't a real clear best practice, which is understandable since it depends on the project. However deploying from Studio is not a no-no, contrary to the best practices article I read.
For our project we are going to be using Studio to push individually run jobs to a Nexus snapshot. If there are many dependent jobs/items, we will use a Jenkins pipeline to build the whole project to a Nexus snapshot.
We will be creating a release branch and a release Jenkins pipeline that builds from it. At certain milestones, we will 1) tag the master branch, 2) merge the master into the release branch, 3) modify all jobs to not be a snapshot via Project Setting's Deployment Versioning, 4) run the release pipeline, and 5) up the version for all jobs in master branch via Deployment Versioning.
I will leave the posting open for comments and/or advice.
I'm in the same position, but the other way round:
I managed to get a jenkins pipeline going for building and deploying release artifacts to Nexus, but struggle to do the same for snapshots.
It took me a while to get my head around the basic concepts, so I'll share what i have found out so far, hoping to get a discussion going:
By default, the artifact version in the pom for a route is set to release (e.g. <version> 1.0.0 </version>).
You can change this manually in the studio under Project Settings → Maven → Deployment Versioning
If you change all or a single route to "use snapshot", then Talend studio modfies the atifact version in the route pom (amongst other things) as follows: <version> 1.0.0-SNAPSHOT</version> and commits it.
Unfortunately there is no "build a snapshot version" parameter for the Maven CI builder plugin. During the Jenkins build, it uses the version in the pom that gets checked out from the project.
But with half a dozen developers working on 250 routes, we don't want to manually change the version in Talend Studio each time.
After a long, long struggle, Talend came up with potential solutions (untested of course) to programmatically modify the poms during the Jenkins build:
A) Use the command line API to set the route version to -SNAPSHOT before building the route
Unfortunately, we cannot use the commandline downloaded by the Maven CI builder plugin, because that's a "light" version and cannot be started in server or script mode.
So we would need a seperate installation of the full commandline somewhere to do this, which totally defeats the purpose of the "Zero install" workflow.
B) Prior to calling the CI Builder Plugin, Use another maven plugin to modify the pom as needed, such as:
Maven version plugin
Maven Replacer plugin
Maven XML plugin
C) Modify the Maven POM Templates in Talend Studio under Project Settings -> Maven -> Deployment Versioning
I'm in the process of implementing B) or C) and will post my findings.
If anyone out there has done this and can share some wisdom on how to do this, it would be apreciated.
Then the helper script (jenkins_talend_helper.groovy) includes a function called:
/**
* setMvnVersion(version, snapshot)
* Calls Studio in scripted mode through Maven to change the Version of the project
* Parameters:
* version the version to be set
* snapshot true/false
*/
def setMvnVersion(version, snapshot) {
This will set the maven version and snapshot/release, and as its done via commandline it will automatically update the poms as well. (Similar to generateAllPoms)
