Pcloud-publisher - deploy to TMC fails for job with child jobs

[irina_s]

Hello,

I have a Jenkins pipeline for job deployement to TMC.

• generation of pom files for the jobs I want to depoly :

mvn -f $PROJECT_GIT_NAME/poms/pom.xml -s ${SETTINGS_MAVEN} -e -fae -am org.talend.ci:builder-maven-plugin:8.0.3:generateAllPoms

• publish to TMC :

mvn -f $PROJECT_GIT_NAME/poms/pom.xml -s ${SETTINGS_MAVEN} -e -fae -Pcloud-publisher clean deploy

All works fine when the job does not contain child job(s).

When I try to deploy a main job calling one or several child jobs the second step (cloud publisher) fails :

[INFO] 

[INFO] --< fr.abc.INIT_PROJECT.my_group_project:MAIN_JOB_TEST >--

[INFO] Building INIT_PROJECT MAIN_JOB_TEST-0.1.0 (0.1,Jobs) 0.1.0 [11/11]

[INFO] --------------------------------[ jar ]---------------------------------

[WARNING] The POM for fr.abc.INIT_PROJECT.my_group_project:CHILD_JOB_TEST:jar:0.1.0 is missing, no dependency information available

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO] 

[INFO] INIT_PROJECT Codes Master 8.0.1 ...................... SUCCESS [04:28 min]

[INFO] INIT_PROJECT Routines ................................ SUCCESS [ 14.814 s]

[INFO] INIT_PROJECT Beans ................................... SUCCESS [ 0.231 s]

[INFO] INIT_PROJECT MAIN_JOB_TEST-0.1.0 (0.1,Jobs) 0.1.0 FAILURE [ 0.078 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 04:45 min

[INFO] Finished at: 2022-05-13T14:30:31+02:00

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal on project MAIN_JOB_TEST: Could not resolve dependencies for project fr.abc.INIT_PROJECT.my_group_project:MAIN_JOB_TEST:jar:0.1.0: Failure to find fr.abc.INIT_PROJECT.my_group_project:CHILD_JOB_TEST:jar:0.1.0 in http://nexus.abc.fr/repository/talend_libs_releases/ was cached in the local repository, resolution will not be reattempted until the update interval of talend-custom-libs-release has elapsed or updates are forced -> [Help 1]

org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project MAIN_JOB_TEST: Could not resolve dependencies for project fr.abc.INIT_PROJECT.my_group_project:MAIN_JOB_TEST:jar:0.1.0: Failure to find fr.abc.INIT_PROJECT.my_group_project:CHILD_JOB_TEST:jar:0.1.0 in http://nexus.abc.fr/repository/talend_libs_releases/ was cached in the local repository, resolution will not be reattempted until the update interval of talend-custom-libs-release has elapsed or updates are forced

Can you please help ?

[Anonymous]

Hello,

Talend main pom is a multi-module project, (I.e. the jobs you can build is under the <modules> section in poms/pom.xml)

Once you decided about the list of jobs you want to build you list them via -pl parameter. (i..e -pl jobs/process/myfirstmaster_0.1,jobs/process/myothermaster_0.1 ) better copy-paste from the pom.xml, the new CI scripts for Jenkins / Azure has dedicated steps that take care of this. https://help.talend.com/r/en-US/8.0/software-dev-lifecycle-best-practices-guide/filter-execution-of-your-project-on-selected-artifacts

But how will this care of of my child jobs? Well: Maven have this "-am" a.k.a --also-make parameter that will compile the dependencies. In this case the child jobs.

For ESB / Microservices you need the -amd so also the -feature is compiled.

I hope this helps.

[irina_s]

hello,

Thank you for your input, ended up by generating the pom for all the jobs in my project ( it takes some time to generate...) and in my second step I deploy only the Main jobs (loop on the list of jobs to deploy) and call the Pcloud publisher with the -pl option.

It works but I find it heavy and time consuming as opposed to the first method where I generate the pom only for the jobs that I want to deploy and then deploy them all.

[Anonymous]

In the second step you could list all your master jobs and let the CI build the common dependencies once. (This should be the fastest approach.)

For the POM generation the problem is that we don't have -pl -am -amd filters / flags implemented. If we'd then we could simply build the same master jobs + dependencies like we'd do the builds later.

If you're interested in such a feature then please raise a support case with the feature / use-case and support will log a Feature Request with R&D .

[irina_s]

Is there any way of indicating in the -pl option to look in all the folders of the project ?

eg : -pl jobs/process/*/${JOB_NAME}_${JOB_VERSION} -am

[Anonymous]

Unfortunately you need to specify everything precisely. (It's case senstive) but you can create a comma separated list

(which can be generated using a previous step via find or some other parameters)

The old CI scripts were mostly targeting single job builds, the new ones allow you to specify multiple one out of the box.

This -pl not able to support * is a limitation from maven actually.