Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47823

withMaven fails to inject Maven settings files with docker-pipeline 1.14

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • pipeline-maven-plugin
    • None
    • pipeline-maven-plugin 2.x+
      docker-pipeline plugin 1.14

      Problem Description

      The docker-pipeline changed its behavior with version 1.14 and is no longer allowing plugins to contribute the "PATH" environment variable.

      The Pipeline Maven Plugin contributes to the PATH environment variable to specify the "mvn" command line arguments such as the settings file.

      Due to this change of behavior, the Pipeline Maven Plugin is no longer able to inject specify Maven settings file.

      Workaround

      If you need to specify the Maven settings file in a dockerized build agent using the Docker Pipeline Plugin,

      References

      https://github.com/jenkinsci/docker-workflow-plugin/commit/f55c122fe10f28bf31ccfca66f0587facf508148#diff-4d69c729a160747eba5c12ea83f7753cR248

      https://github.com/jenkinsci/pipeline-maven-plugin/blob/pipeline-maven-3.0.2/jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java#L279

          [JENKINS-47823] withMaven fails to inject Maven settings files with docker-pipeline 1.14

          joshtrow I understand what you say.

          The problem that we discovered with docker-pipeline up to 1.13 is that it overwrites the "PATH" of the Docker container by the "PATH" of the underlying agent. There could have a huge mess with a RHEL/Centos agent launching a debian based Docker image...

          We are still brainstorming to reduce the impact of the fix brought by docker-pipeline:1.14. In the meantime, I would recommend to stick to docker-pipeline

          I have a few questions please

          1. pipeline-maven:3.0.3 transparently integrates with Takari's Maven wrapper 'mvnw' (v0.1.5+), you don't have to use the workaround of "MVN_CMD_DIR" and it works with docker-pipeline 1.13 and 1.14 (sample below). Would this be interesting in your context?
          2. Can you please tell us which OS you use for the host of the agent and which type of Docker image you like to use (the official "maven" image that inherits from "debian"...)?
          3. Could you test maven-pipeline:3.0.3-beta-2 with the "MVN_CMD_DIR" workaround?
          4. Could you give us a rough number of how many pipelines using withMaven you have?
          node("linux-agent-with-docker") {
             docker.image('maven').inside {
                  git 'https://github.com/takari/maven-wrapper.git'
                  withMaven (...) {
                      // begin the sh step with "env &&" for troubleshooting and display "MAVEN_CONFIG" , no need in real life
                      sh 'env && ./mvnw effective-settings'
                  }
              }
          }
          

          Cyrille Le Clerc added a comment - joshtrow I understand what you say. The problem that we discovered with docker-pipeline up to 1.13 is that it overwrites the "PATH" of the Docker container by the "PATH" of the underlying agent. There could have a huge mess with a RHEL/Centos agent launching a debian based Docker image... We are still brainstorming to reduce the impact of the fix brought by docker-pipeline:1.14. In the meantime, I would recommend to stick to docker-pipeline I have a few questions please pipeline-maven:3.0.3 transparently integrates with Takari's Maven wrapper 'mvnw' (v0.1.5+), you don't have to use the workaround of "MVN_CMD_DIR" and it works with docker-pipeline 1.13 and 1.14 (sample below). Would this be interesting in your context? Can you please tell us which OS you use for the host of the agent and which type of Docker image you like to use (the official "maven" image that inherits from "debian"...)? Could you test maven-pipeline:3.0.3-beta-2 with the "MVN_CMD_DIR" workaround? Could you give us a rough number of how many pipelines using withMaven you have? node( "linux-agent-with-docker" ) { docker.image( 'maven' ).inside { git 'https: //github.com/takari/maven-wrapper.git' withMaven (...) { // begin the sh step with "env &&" for troubleshooting and display "MAVEN_CONFIG" , no need in real life sh 'env && ./mvnw effective-settings' } } }

          Josh Trow added a comment -

          Hi cleclerc

          I'll take a look at the details of the some of the questions later when i'm free for longer, but the short answer set I can give is this:

          1. mvnw looks to be an easier way to make the change, if we had to across the board certainly
          2. Unless someone has changed it underneath me, we are using Alpine both for master and slave machines generally
          3. I'll test beta-2 later on today
          4. Direct calls to withMaven: 171 files found when I search our entire repository, and that's just the public ones I know about, there could be more

          Thanks for explaining the root issue better though - it has never really been clear to me why the issue that 1.14 'fixed' was really an issue, and that does help to clarify the situation (Debian vs RHEL paths) somewhat. I'll admit to not being an expert in the differences between them so I don't still fully comprehend the full details of the issue, but I might be able to pick up the conversations in other issues better now.

          Josh Trow added a comment - Hi cleclerc I'll take a look at the details of the some of the questions later when i'm free for longer, but the short answer set I can give is this: mvnw looks to be an easier way to make the change, if we had to across the board certainly Unless someone has changed it underneath me, we are using Alpine both for master and slave machines generally I'll test beta-2 later on today Direct calls to withMaven: 171 files found when I search our entire repository, and that's just the public ones I know about, there could be more Thanks for explaining the root issue better though - it has never really been clear to me why the issue that 1.14 'fixed' was really an issue, and that does help to clarify the situation (Debian vs RHEL paths) somewhat. I'll admit to not being an expert in the differences between them so I don't still fully comprehend the full details of the issue, but I might be able to pick up the conversations in other issues better now.

          Tim Downey added a comment -

          cleclerc – I've tested with beta 2 and can confirm that it works.  Following on joshtrow comments:

          • Updating our pipelines would be a real headache.  I have in the neighborhood of 100 to deal with when you look across repos and branches.
          • This headache is worse since I'd need to change pipelines on Production and Hotfix branches which is enough to really scare management typically.  In other words, we can't start a change like this on develop and roll it through.  There's no real way around this since Jenkins is building across branches.
          • I'm using the "official" Maven image just about exclusively.  Typically `maven:3-jdk8-alpine`.
          • Takari Wrapper may be something in the future, but we'd still have to update all of our pipelines to use `mvnw` instead of `mvn`, right?  If so, that is certainly more aesthetically pleasing, but doesn't really change the work.
          • Could MAVEN_CONFIG be set inside the pipeline?  It looks like that may help.  This is from the mvn shell script (3.5.2.1).  I'm not sure how far back it goes.
          MAVEN_CMD_LINE_ARGS="$MAVEN_CONFIG $@"
          export MAVEN_CMD_LINE_ARGS
          

          But if you could set MAVEN_CONFIG and pass in the --settings or other options there may be a solution.  A quick Google doesn't turn up much on MAVEN_CONFIG so it may not be reliable across versions though...

          • Perhaps failing all of this, Docker Pipeline could keep working as in 1.13 but provide an arg in 1.14 to prevent the PATH from being overwritten?  At least in this case compatibility would be preserved and those with problems would have a flag to throw to not overwrite the PATH.

          Tim

           

          Tim Downey added a comment - cleclerc – I've tested with beta 2 and can confirm that it works.  Following on joshtrow comments: Updating our pipelines would be a real headache.  I have in the neighborhood of 100 to deal with when you look across repos and branches. This headache is worse since I'd need to change pipelines on Production and Hotfix branches which is enough to really scare management typically.  In other words, we can't start a change like this on develop and roll it through.  There's no real way around this since Jenkins is building across branches. I'm using the "official" Maven image just about exclusively.  Typically `maven:3-jdk8-alpine`. Takari Wrapper may be something in the future, but we'd still have to update all of our pipelines to use `mvnw` instead of `mvn`, right?  If so, that is certainly more aesthetically pleasing, but doesn't really change the work. Could MAVEN_CONFIG be set inside the pipeline?  It looks like that may help.  This is from the mvn shell script (3.5.2.1).  I'm not sure how far back it goes. MAVEN_CMD_LINE_ARGS= "$MAVEN_CONFIG $@" export MAVEN_CMD_LINE_ARGS But if you could set MAVEN_CONFIG and pass in the --settings or other options there may be a solution.  A quick Google doesn't turn up much on MAVEN_CONFIG so it may not be reliable across versions though... Perhaps failing all of this, Docker Pipeline could keep working as in 1.13 but provide an arg in 1.14 to prevent the PATH from being overwritten?  At least in this case compatibility would be preserved and those with problems would have a flag to throw to not overwrite the PATH. Tim  

          timdowney we start exposing MAVEN_CONFIG with pipeline-maven 3.0.3-beta but it does not get picked by the "mvn" 3.5.2 executable as it gets picked by Takari's "mvnw". My understanding is that "MAVEN_CMD_LINE_ARGS" is not used anywhere in Maven.

          https://github.com/apache/maven/blob/maven-3.5.2/apache-maven/src/bin/mvn#L188

          Cyrille Le Clerc added a comment - timdowney we start exposing MAVEN_CONFIG with pipeline-maven 3.0.3-beta but it does not get picked by the "mvn" 3.5.2 executable as it gets picked by Takari's "mvnw". My understanding is that "MAVEN_CMD_LINE_ARGS" is not used anywhere in Maven. https://github.com/apache/maven/blob/maven-3.5.2/apache-maven/src/bin/mvn#L188

          Tim Downey added a comment -

          Hmm...I should have looked further down.  You're right.  It does not appear to be used.

          Tim Downey added a comment - Hmm...I should have looked further down.  You're right.  It does not appear to be used.

          Nuno Costa added a comment -

          cleclerc, using the example you mentioned above works ok but I noticed some other issues. Not sure if this issues are only happening because this is just a beta:

          1. When invoking mvn command, uname is not found
            ----- withMaven Wrapper script -----
            /opt/apache-maven-3.1.1/bin/mvn: 1: /opt/apache-maven-3.1.1/bin/mvn: uname: not found
            ...
          1. We use maven plugins that triggers cmake, libboost, etc, and this commands/libraries are only found if I add its paths to the export command.
            ...
            sh "export PATH=$MVN_CMD_DIR:$PATH:opt/bin && mvn clean test"
            ...

            All those commands are already set in PATH but for some reason are not picked by the export command.
            Below more information about paths set inside a ubuntu 12.04 container. Build server on CentOS7.

            ...
            sh "echo $PATH"
            sh "which mvn"
            sh "which cmake"
            sh "echo $MVN_CMD_DIR"
            sh "env | grep PATH"
            sh "uname -r"
            sh "export PATH=$MVN_CMD_DIR:$PATH && which mvn && mvn -v"
            sh "echo $PATH"
            sh "which mvn"
            sh "which cmake"
            sh "env | grep PATH"
            ...
            

            Console output:

            Running shell script
            + echo /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin
            /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin
            sh
            Running shell script
            + which mvn
            /opt/apache-maven-3.1.1/bin/mvn
            sh
            Running shell script
            + which cmake
            /opt/bin/cmake
            sh
            Running shell script
            + echo /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4
            /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4
            sh
            Running shell script
            + env
            + grep PATH
            LIBRARY_PATH=/opt/qt/lib:/opt/lib:
            LD_LIBRARY_PATH=/opt/lib:
            PATH=/opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
            CLASSPATH=
            sh
            Running shell script
            + uname -r
            4.4.83-1.el7.elrepo.x86_64
            sh
            Running shell script
            + export PATH=/var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin
            + which mvn
            /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4/mvn
            + mvn -v
            ----- withMaven Wrapper script -----
            /opt/apache-maven-3.1.1/bin/mvn: 1: /opt/apache-maven-3.1.1/bin/mvn: uname: not found
            Warning: JAVA_HOME environment variable is not set.
            Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4"
            Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17 16:22:22+0100)
            Maven home: /opt/apache-maven-3.1.1
            Java version: 1.8.0_111, vendor: Oracle Corporation
            Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
            Default locale: en_US, platform encoding: ANSI_X3.4-1968
            OS name: "linux", version: "4.4.83-1.el7.elrepo.x86_64", arch: "amd64", family: "unix"
            sh
            Running shell script
            + echo /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin
            /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin
            sh
            Running shell script
            + which mvn
            /opt/apache-maven-3.1.1/bin/mvn
            sh
            Running shell script
            + which cmake
            /opt/bin/cmake
            sh
            Running shell script
            + env
            + grep PATH
            LIBRARY_PATH=/opt/qt/lib:/opt/lib:
            LD_LIBRARY_PATH=/opt/lib:
            PATH=/opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
            CLASSPATH=
            

          Nuno Costa added a comment - cleclerc , using the example you mentioned above works ok but I noticed some other issues. Not sure if this issues are only happening because this is just a beta: When invoking mvn command, uname is not found ----- withMaven Wrapper script ----- /opt/apache-maven-3.1.1/bin/mvn: 1: /opt/apache-maven-3.1.1/bin/mvn: uname: not found ... We use maven plugins that triggers cmake, libboost, etc, and this commands/libraries are only found if I add its paths to the export command. ... sh "export PATH=$MVN_CMD_DIR:$PATH:opt/bin && mvn clean test" ... All those commands are already set in PATH but for some reason are not picked by the export command. Below more information about paths set inside a ubuntu 12.04 container. Build server on CentOS7. ... sh "echo $PATH" sh "which mvn" sh "which cmake" sh "echo $MVN_CMD_DIR" sh "env | grep PATH" sh "uname -r" sh "export PATH=$MVN_CMD_DIR:$PATH && which mvn && mvn -v" sh "echo $PATH" sh "which mvn" sh "which cmake" sh "env | grep PATH" ... Console output: Running shell script + echo / var /lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin / var /lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin sh Running shell script + which mvn /opt/apache-maven-3.1.1/bin/mvn sh Running shell script + which cmake /opt/bin/cmake sh Running shell script + echo / var /lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4 / var /lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4 sh Running shell script + env + grep PATH LIBRARY_PATH=/opt/qt/lib:/opt/lib: LD_LIBRARY_PATH=/opt/lib: PATH=/opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin CLASSPATH= sh Running shell script + uname -r 4.4.83-1.el7.elrepo.x86_64 sh Running shell script + export PATH=/ var /lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/ var /lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin + which mvn / var /lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4/mvn + mvn -v ----- withMaven Wrapper script ----- /opt/apache-maven-3.1.1/bin/mvn: 1: /opt/apache-maven-3.1.1/bin/mvn: uname: not found Warning: JAVA_HOME environment variable is not set. Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext. class. path= "/ var /lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder= "/ var /lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4" Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17 16:22:22+0100) Maven home: /opt/apache-maven-3.1.1 Java version: 1.8.0_111, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre Default locale: en_US, platform encoding: ANSI_X3.4-1968 OS name: "linux" , version: "4.4.83-1.el7.elrepo.x86_64" , arch: "amd64" , family: "unix" sh Running shell script + echo / var /lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin / var /lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin sh Running shell script + which mvn /opt/apache-maven-3.1.1/bin/mvn sh Running shell script + which cmake /opt/bin/cmake sh Running shell script + env + grep PATH LIBRARY_PATH=/opt/qt/lib:/opt/lib: LD_LIBRARY_PATH=/opt/lib: PATH=/opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin CLASSPATH=

          Code changed in jenkins
          User: Cyrille Le Clerc
          Path:
          jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java
          http://jenkins-ci.org/commit/pipeline-maven-plugin/52141ec07401eb73b94963e414f4075a2903e1b1
          Log:
          JENKINS-47823 better help in build logs

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java http://jenkins-ci.org/commit/pipeline-maven-plugin/52141ec07401eb73b94963e414f4075a2903e1b1 Log: JENKINS-47823 better help in build logs

          Cyrille Le Clerc added a comment - - edited

          ncosta can you please open a different bug if the problem is not fixed. It may be a problem with the way docker-pipeline handles the $PATH or a problem with the way the $PATH is defined in your Docker image (please keep in mind that docker-pipeline no longer honors the entrypoint).

          Closing this defect as pipeline-maven-plugin 3.0.3 offers a solution to work with docker-pipeline 1.14: https://wiki.jenkins.io/display/JENKINS/Pipeline+Maven+Plugin#PipelineMavenPlugin-withMavenAndDockerPipeline

          Cyrille Le Clerc added a comment - - edited ncosta can you please open a different bug if the problem is not fixed. It may be a problem with the way docker-pipeline handles the $PATH or a problem with the way the $PATH is defined in your Docker image (please keep in mind that docker-pipeline no longer honors the entrypoint). Closing this defect as pipeline-maven-plugin 3.0.3 offers a solution to work with docker-pipeline 1.14: https://wiki.jenkins.io/display/JENKINS/Pipeline+Maven+Plugin#PipelineMavenPlugin-withMavenAndDockerPipeline

          Nuno Costa added a comment -

          cleclerc, just created JENKINS-48284 with some details.

          Nuno Costa added a comment - cleclerc , just created JENKINS-48284 with some details.

          Joshua Noble added a comment -

          Joshua Noble added a comment - joshtrow I know it's been a year, but I came up with a workaround for not having to update every mvn call here:  https://issues.jenkins-ci.org/browse/JENKINS-40484?focusedCommentId=352635&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-352635

            cleclerc Cyrille Le Clerc
            cleclerc Cyrille Le Clerc
            Votes:
            2 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated:
              Resolved: