Merge git Development to Master Branch via Talend Studio

[nmodi]

We are using bitBucket as our source code repository in talend. We are trying to follow below best practice suggested by Talend & I have one question on Merging

Below diagram shows that we should have multiple branches in Git as below: 

• Master (to keep final production source code after development & testing is done on development branch)
• Development (this git branch is used for actual development)
  • Feature (local branches that only exist on local developer’s machine, not on Git – correct?)

 

Reference : https://help.talend.com/reader/GDDhoNyPTrERkrITzE6qPQ/QhAXF8ijSOHq~lhrVCoLKQ

Now we want to follow this scenario however we have question on merging Development branch code to master branch – how do we do that?

• Can we merge Development branch to Master branch using Talend studio?
• Or Do we have to do this outside of talend studio that is use bitBucket to merge branches (may be using create pull request)
  • If we have to follow this scenario then How do we handle Merge conflicts outside of Talend studio something like below, I mean this is impossible to manage since most of the files are like this?

Could you please guide us in right direction, we are using Talend enterprise version 6.4.1 with TAC.

Thanks,

Nimesh

[nmodi]

@xdshi Could you please help us with this issue & guide us in right direction?

I would really appreciate your response on this.

Thanks,

Nimesh

[mayo · Accepted solution]

Not sure how much help I can be since our team is new to this also.  We are starting minimal by just having a master and using tagging for certain milestones, branching when necessary for larger features and then merging it back to the master.

In the scenario below I think you would be able to merge Development with Master within Talend Studio by

• switching to the local Master branch
• execute a Pull and Merge Branch
• select the Development branch (within the popup that allows you to select a branch to merge with)
• do conflict resolution, etc
• execute a Push to update the remote Master

[Anonymous]

Hi,

Which is talend subscription solution you are using? Have you already created a case on talend support portal?

Best regards

Sabrina

[nmodi]

Thanks @mayo, this solution works for us.

@xdshi, We are using 6.4 enterprise version.

[n999]

Hello

Apologies for hijacking this thread -  I've done this as all Posters may already have an Answer to the below.

I have a question relating to the Software Development Life Cycle (SDLC) and CI in Talend.

We are just starting out on our subscriber Talend 6.4 Journey and are interested in using GIT and following a good workflow - one that we can move to CI asap.

My question revolves around Source Control and Versions, looking down from the SCM Repository Level all the way down to the Job Level.

Dale Anderson ( https://www.talend.com/blog/2016/03/30/talend-job-design-patterns-best-practices-part-2/) seems to say that Job versioning should not be done using the Talend Studio Job major and minor buttons, but we should "use the native SCC branching and tagging mechanisms." instead.

I'm confused how this would work:

1) Talend puts ALL projects in a single Git repository.

So If you branch/tag your GIT repository then ALL  projects are branched or tagged. - Q: Can you not configure a repository per project?

2) Even though the projects are versioned, it is jobs THEMSELVES that are published to nexus at a particular version number. - Q: How do you tie a job version to a branch/tag? (Part of me feels each job should have it's own git repo.... esp good for READMEs, etc) 

3) I did find this https://help.talend.com/reader/ElruncSKqadnY2jVuXhG8g/orkrMC2OzmiRk151LjYeVw which seems to  "... allow you to release your whole project with the same fixed version"

So this could tie the job version to the project version which could be tied to a branch/tag. (even though the branch/tag would exist in other projects!) - Q:is anyone doing this?

4) In this scenario what happens if:

* You have a project at branch/tag 1.1 with 2 jobs published to nexus and deployed to your runtime at version 1.1
* you work on jobA but not jobB and branch/tag the project (actually whole repo!) at 1.2.
* you change the version of all Jobs to 1.2 and publish and deploy? (this means jobB has incremented in version although it has not changed?
* you work on jobB but not jobA and branch/tag the project (actually whole repo!) at 1.3.
* you change the version of all Jobs to 1.3 and publish and deploy? (this means here jobA has incremented in version although it has not changed?

Q:  what if you need to rollback jobA to 1.2. If you rollback all jobs to 1.2 you will have lost changes to jobB? Do you never rollback but only move forward so you would need to create 1.4 with a fix to jobA? (which would be overwriting  jobA 1.3 with jobA 1.2 somehow?)

Apologies if these are newbie questions and I have missed something obvious?

Cheers


(ps - It seems that the diagram at https://help.talend.com/reader/ElruncSKqadnY2jVuXhG8g/oJ3DTkOPqBjmkQw69p24dg lists a

"Gitflow Workflow" as described at https://www.talend.com/blog/2017/08/14/continuous-integration-talend-ci-builder/.

https://www.talend.com/blog/2017/08/14/continuous-integration-talend-ci-builder/ also lists a "Feature Branch Workflow" which I think is what mayo describes above?

We are unsure which one we will opt for ourselves......)

[mayo]

Sorry for the delayed response.

I did some testing of the version within Talend.  I also read about not using this functionality.  I think I found that it does not give any extra functionality that Git does not already include.  Using that versioning with Git may make things extra confusing and hard to manage.

1)  I did not know Talend puts all the projects into 1 Git repository.  I am also confused how that would work.  We have 1 Git repository for each project.  Reference project gets its own Git repository.  This has worked well for us so far.

2) I am still new to this and may not be the best source for advice.  However, I can tell you that basically for now we have just a master branch and some tags to capture a significant point in history.  Tags are useful because you can quickly switch to that tag from within Talend to see how all jobs within the project were at that milestone.  You configure the POM with the version number in Jenkins.  When we run a build and deploy to Nexus (all jobs in the project), Jenkins will automatically add on a timestamp and a running number to distinguish the different builds.  We haven not yet needed to do branching (other than tags) to later merge back to the master branch.  Also we only build off the master branch, so I am not sure how building off of other branches would work.  I think you can configure in Jenkins what branch you build off of though, so I think you have the abilities to distinguish in this situation.

From what I experienced, if each job had its own repo it would be very hard to manage.  We don't have many jobs, but it would still be a TON of overhead if it were 1 job per repo.

3)  Yes when we do builds, it builds all jobs at once, fails the build if ANY job doesn't compile or pass unit tests, and deploys them all to Nexus with a particular build number (version + timestamp + running ID for that version).  

4) I think I follow you in this scenario.  We look at a project as a whole suite of jobs, harmonious even if unrelated.  So if one of the jobs needs to roll back, we would roll back just that one job but still continue to move forward with the same instance of that version (with different timestamp and running number).  We would increase versions for milestones or new functionality or whatever floats your boat.  I don't think we would ever rollback to a prevoius tag/branch version because this would imply you are rolling back ALL jobs in the project, which is what I think you realize as well.

Hopefully this helps.  I am new to continuous integration, so can only talk from our experience.

[n999]

Hiya - please don't apologise - I'm v grateful for the advice!

1) Yes - since posting I've discovered that you can have one project per repo...I think I was confused by the root config in the tac where you can add Git config.

Yes one job per project per repo sounds like something that won't work with the way talend works - If only you could include ref projects or joblets or metadata or connection details  from outside your own project as injected dependencies, and version these injected dependencies in your project. This would be how you would build a regular maven based java project by referencing injected dependencies in your project pom.

2) an 3)  - sorry some quick fire questions

So you change the project version number ( I think this might be what talend calls 'the published version number...') manually in the pom in jenkins?

Have you tried changing the project version number as per https://help.talend.com/reader/ElruncSKqadnY2jVuXhG8g/orkrMC2OzmiRk151LjYeVw?_ga=2.139841203.1208745...?

I presume this does the same thing?

What version number do you use? Is it just incremental numerical versioning? 

You don't use the git tag as the pom version number?

So this deploys all jobs in that project to nexus at the SAME version?

So you have some jobs in that project incrementing in version number in nexus even though they have no code change?

If so, does that mean you redeploy all these new unchanged jobs to a job server?

---

I did find this https://www.teschglobal.com/resources-all/talend-devops-continuous-integration :

The video is all interesting (and the blog post mentions continuous deployment using metaservlet) showing the use of a script (around 29:40) to deploy to nexus and changing what he calls the publish version for all jobs. This is opposed to editing the pom in jenkins or using the method in the help.talend.com link above....

4) " So if one of the jobs needs to roll back, we would roll back just that one job but still continue to move forward ..." Just to clarify , so you would roll back job A on the TAC/Job Server but not in the code base? so you would make an incremental fix to job A in Master to represent the rollback and then release the project all at a new incremented version? And again deploying all new versioned jobs in that project on the Tac/job server?

Sorry for all the questions  and thanks again for your time

cheers