Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40484

Unable to use withMaven() step inside docker container for old versions of Docker

      I'm not able to use withMaven step inside docker container.

      [Pipeline] withMaven
      $ docker exec ffff env printenv MAVEN_HOME
      $ docker exec ffff env printenv M2_HOME
      $ docker exec ffff env /bin/sh -c "which mvn"
      Using maven exec: /opt/apache-maven-3.3.9/bin/mvn
      Using global settings config with name MavenGlobal
      Replacing all maven server entries not found in credentials list is false
      [Pipeline] {
      [Pipeline] sh
      
      [test-XXX] Running shell script
      nohup: failed to run command `sh': No such file or directory
      
      [Pipeline] }
      [Pipeline] // withMaven

      My Jenkinsfile pipeline:

      node('docker') {
          stage('Checkout') {
              checkout scm
          }
          buildInDocker('linux') {
              stage('Maven') {
                  withMaven(globalMavenSettingsConfig: '11111111-2222-3333-4444-555555555555') {
                          sh 'mvn clean test'
                  }
              }
          }
      }

      It looks like PATH env gets overwritten.

          [JENKINS-40484] Unable to use withMaven() step inside docker container for old versions of Docker

          dan tran added a comment -

          Love to see this fixed

          dan tran added a comment - Love to see this fixed

          Same here, with this simple use case:

          node('master') {
              docker.image('maven').inside {
                  withMaven {
                      sh 'mvn -version'
                  }
              }
          }
          

          Bruno Meneguello added a comment - Same here, with this simple use case: node( 'master' ) { docker.image( 'maven' ).inside { withMaven { sh 'mvn -version' } } }

          Alvaro Lobato added a comment -

          Does this work for you?

          node('master') {
              docker.image('maven').inside {
                      sh 'mvn -version'
              }
          }
          

          Alvaro Lobato added a comment - Does this work for you? node( 'master' ) { docker.image( 'maven' ).inside { sh 'mvn -version' } }

          Yes, it does

          Bruno Meneguello added a comment - Yes, it does

          Test User added a comment -

          Simple Jenkinsfile to show how env are changed:

          node('docker') {
              docker.image('linux').inside {
                  echo 'env.PATH=' + env.PATH
                  sh 'echo $PATH > path'
                  path = readFile('path')
                  echo 'PATH=' + path
                  withMaven {
                      echo 'env.PATH.mvn=' + env.PATH
                      env.PATH = env.PATH + ':/bin'
                      sh 'echo $PATH > path_mvn'
                      path_mvn = readFile('path_mvn')
                      echo 'PATH.mvn=' + path_mvn
                      }
              }
          }
          env.PATH=/usr/local/bin:/usr/bin
          PATH=/opt/apache-maven-3.3.9/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
          env.PATH.mvn=/var/lib/jenkins/workspace/repository-test_develop-ISA36EXDSAND7KY74323XF3CUR2ZV65I43UWYEDK2UMP3I5UQYMQ@tmp/withMavenaf31c1e4:/usr/local/bin:/usr/bin
          PATH.mvn=/var/lib/jenkins/workspace/repository-test_develop-ISA36EXDSAND7KY74323XF3CUR2ZV65I43UWYEDK2UMP3I5UQYMQ@tmp/withMavenaf31c1e4:/var/lib/jenkins/workspace/repository-test_develop-ISA36EXDSAND7KY74323XF3CUR2ZV65I43UWYEDK2UMP3I5UQYMQ@tmp/withMavenaf31c1e4:/usr/local/bin:/usr/bin:/bin

          Test User added a comment - Simple Jenkinsfile to show how env are changed: node( 'docker' ) { docker.image( 'linux' ).inside { echo 'env.PATH=' + env.PATH sh 'echo $PATH > path' path = readFile( 'path' ) echo 'PATH=' + path withMaven { echo 'env.PATH.mvn=' + env.PATH env.PATH = env.PATH + ':/bin' sh 'echo $PATH > path_mvn' path_mvn = readFile( 'path_mvn' ) echo 'PATH.mvn=' + path_mvn } } } env.PATH=/usr/local/bin:/usr/bin PATH=/opt/apache-maven-3.3.9/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin env.PATH.mvn=/ var /lib/jenkins/workspace/repository-test_develop-ISA36EXDSAND7KY74323XF3CUR2ZV65I43UWYEDK2UMP3I5UQYMQ@tmp/withMavenaf31c1e4:/usr/local/bin:/usr/bin PATH.mvn=/ var /lib/jenkins/workspace/repository-test_develop-ISA36EXDSAND7KY74323XF3CUR2ZV65I43UWYEDK2UMP3I5UQYMQ@tmp/withMavenaf31c1e4:/ var /lib/jenkins/workspace/repository-test_develop-ISA36EXDSAND7KY74323XF3CUR2ZV65I43UWYEDK2UMP3I5UQYMQ@tmp/withMavenaf31c1e4:/usr/local/bin:/usr/bin:/bin

          dan tran added a comment -

          relate to JENKINS-41339 ??

          dan tran added a comment - relate to JENKINS-41339 ??

          Nuno Costa added a comment - - edited

          The workaround to pass the variables from outside withMaven block, mentioned by Test User here works on a Jenkins 2.32.2 but not on Jenkins 2.46.

          Ubuntu 12.04
          Jenkins 2.32.2
          Pipeline Maven Plugin 0.5
          Durable Task Plugin 1.13

          CentOS 7
          Jenkins 2.46
          Pipeline Maven Plugin 0.5
          Durable Task Plugin 1.13

          Buildserver
          Ubuntu 14.04 + docker 1.13.1

          Buildserver
          CentOS7 + docker 1.13.1

          Docker image:
          Ubuntu 12.04
          Maven 3.1.1

          Nuno Costa added a comment - - edited The workaround to pass the variables from outside withMaven block, mentioned by Test User here works on a Jenkins 2.32.2 but not on Jenkins 2.46. Ubuntu 12.04 Jenkins 2.32.2 Pipeline Maven Plugin 0.5 Durable Task Plugin 1.13 CentOS 7 Jenkins 2.46 Pipeline Maven Plugin 0.5 Durable Task Plugin 1.13 Buildserver Ubuntu 14.04 + docker 1.13.1 Buildserver CentOS7 + docker 1.13.1 Docker image: Ubuntu 12.04 Maven 3.1.1

          Nuno Costa added a comment - - edited

          This can be replicated with below code. CentOS node build will fail with

          [number1] nohup: failed to run command `sh': No such file or directory
          

          Commented is some troubleshoot steps.
          Can be resolved by getting the container path into a file, pass it into a variable and finally pass that to withEnv.

          Be aware that echo adds a new line to the file and in my case, this was also triggering the same issue. Use trim() to remove the extra line.
          The use of printf instead of echo will avoid the trim() usage.

          CentOS 7
          Jenkins 2.46

          parallel(
              number1: {
                  node('centos-buildserver') {
                      buildenv = docker.image("custom-ubuntu12.04-maven")
                      buildenv.inside("-v /var/lib/jenkins/.m2/repository/${EXECUTOR_NUMBER}:/home/jenkins/.m2/repository"){
                          stage("test") {
                              /*sh 'echo $PATH > path'
                              sh 'printf $PATH > path'
                              path = readFile('path').trim()
                              path = readFile('path')
                              pathtrim = readFile('path').trim()
                              echo '#######'
                              echo 'no trim: #'+path+'#'
                              echo '#######'
                              echo 'with trim: #'+pathtrim+'#'
                              echo '#######'*/
                              sh 'ls -l'
                              //withEnv(['PATH+workaround='+path]) {
                                  withMaven(mavenSettingsConfig: 'settings-id'){
                                      echo 'inside withmaven'
                                      sh 'ls -l'
                                  }
                              //}
                          }
                      }
                  }
              },
              number2: {
                  node('ubuntu-buildserver') {
                      buildenv = docker.image("custom-ubuntu12.04-maven")
                      buildenv.inside("-v /var/lib/jenkins/.m2/repository/${EXECUTOR_NUMBER}:/home/jenkins/.m2/repository"){
                          stage("test") {
                              /*sh 'echo $PATH > path'
                              sh 'printf $PATH > path'
                              path = readFile('path').trim()
                              path = readFile('path')
                              pathtrim = readFile('path').trim()
                              echo '#######'
                              echo '#'+path+'#'
                              echo '#######'
                              echo '#'+pathtrim+'#'
                              echo '#######'*/
                              sh 'ls -l'
                              //withEnv(['PATH+workaround='+path]) {
                                  withMaven(mavenSettingsConfig: 'settings-id'){
                                      echo 'inside withmaven'
                                      sh 'ls -l'
                                  }
                              //}
                          }
                      }
                  }
              }
          )
          

          Nuno Costa added a comment - - edited This can be replicated with below code. CentOS node build will fail with [number1] nohup: failed to run command `sh': No such file or directory Commented is some troubleshoot steps. Can be resolved by getting the container path into a file, pass it into a variable and finally pass that to withEnv. Be aware that echo adds a new line to the file and in my case, this was also triggering the same issue. Use trim() to remove the extra line. The use of printf instead of echo will avoid the trim() usage. CentOS 7 Jenkins 2.46 parallel( number1: { node('centos-buildserver') { buildenv = docker.image("custom-ubuntu12.04-maven") buildenv.inside("-v /var/lib/jenkins/.m2/repository/${EXECUTOR_NUMBER}:/home/jenkins/.m2/repository"){ stage("test") { /*sh 'echo $PATH > path' sh 'printf $PATH > path' path = readFile('path').trim() path = readFile('path') pathtrim = readFile('path').trim() echo '#######' echo 'no trim: #'+path+'#' echo '#######' echo 'with trim: #'+pathtrim+'#' echo '#######'*/ sh 'ls -l' //withEnv(['PATH+workaround='+path]) { withMaven(mavenSettingsConfig: 'settings-id'){ echo 'inside withmaven' sh 'ls -l' } //} } } } }, number2: { node('ubuntu-buildserver') { buildenv = docker.image("custom-ubuntu12.04-maven") buildenv.inside("-v /var/lib/jenkins/.m2/repository/${EXECUTOR_NUMBER}:/home/jenkins/.m2/repository"){ stage("test") { /*sh 'echo $PATH > path' sh 'printf $PATH > path' path = readFile('path').trim() path = readFile('path') pathtrim = readFile('path').trim() echo '#######' echo '#'+path+'#' echo '#######' echo '#'+pathtrim+'#' echo '#######'*/ sh 'ls -l' //withEnv(['PATH+workaround='+path]) { withMaven(mavenSettingsConfig: 'settings-id'){ echo 'inside withmaven' sh 'ls -l' } //} } } } } )

          Alvaro Lobato added a comment -

          I've done multiple tests and I can't reproduce this issue. Can you give me the exact versions of the pipeline plugins you are running? The easiest way to help me reproduce this is to install the Support Plugin and generate a support bundle which will contain all the envirornment information.

          Did you check if this is related to JENKINS-41339?

          Alvaro Lobato added a comment - I've done multiple tests and I can't reproduce this issue. Can you give me the exact versions of the pipeline plugins you are running? The easiest way to help me reproduce this is to install the Support Plugin and generate a support bundle which will contain all the envirornment information. Did you check if this is related to JENKINS-41339 ?

          dan tran added a comment - - edited

          here is my simple pipeline script running against 2.46.1. This time it has a different error 
          [pipeline-test] Running shell script
          env: nohup: No such file or directory

           

          {code}

          pipeline {

            agent {
              docker {
              image "maven"
              }
            }

            stages {

              stage("run") {
                steps {
                  withMaven( mavenLocalRepo: '.repository') {
                     sh "echo hello"

                   }
              }
            }

            }

           

          I now attached the support bundle

            }

           

           

          dan tran added a comment - - edited here is my simple pipeline script running against 2.46.1. This time it has a different error  [pipeline-test] Running shell script env: nohup: No such file or directory   {code} pipeline {   agent {     docker {     image "maven"     }   }   stages {     stage("run") {       steps {         withMaven( mavenLocalRepo: '.repository') {            sh "echo hello"          }     }   }   }   I now attached the support bundle   }    

          Cyrille Le Clerc added a comment - - edited

          Cannot reproduce. If your job fails with "nohup" not found, I suspect that the host linux instance is missing "nohup" or maybe you have used a "weird" Docker image "maven" with missing "nohup".

          Please re open a ticket if you are still facing issues.

           My test is based on

          • pipeline-maven-spy-2.1.1-beta-1.hpi (should behave the same on older versions)
          • "maven:3.5.0-jdk-8" and executed
          • Xenial 16:04 ubuntu build agent.

          https://gist.github.com/cyrille-leclerc/20af6c45ce04f6a3c727342bdf474159

          node {
              stage('build') {
                  docker.image('maven:3.5.0-jdk-8').inside {
                      git 'https://github.com/spring-projects/spring-petclinic.git'
                      withMaven() {
                          sh 'mvn clean package'
                      }
                  }
              }
          }
           

          Cyrille Le Clerc added a comment - - edited Cannot reproduce. If your job fails with "nohup" not found, I suspect that the host linux instance is missing "nohup" or maybe you have used a "weird" Docker image "maven" with missing "nohup". Please re open a ticket if you are still facing issues.  My test is based on pipeline-maven-spy-2.1.1-beta-1.hpi (should behave the same on older versions) "maven:3.5.0-jdk-8" and executed Xenial 16:04 ubuntu build agent. https://gist.github.com/cyrille-leclerc/20af6c45ce04f6a3c727342bdf474159 node { stage( 'build' ) { docker.image( 'maven:3.5.0-jdk-8' ).inside { git 'https: //github.com/spring-projects/spring-petclinic.git' withMaven() { sh 'mvn clean package ' } } } }  

          dan tran added a comment -

          cleclerc could you try the my provided declarative pipeline?

          dan tran added a comment - cleclerc could you try the my provided declarative pipeline?

          dan tran added a comment -

          ran your example 

          node {
          stage('build') {
          docker.image('maven:3.5.0-jdk-8').inside {
          git 'https://github.com/spring-projects/spring-petclinic.git'
          withMaven( mavenLocalRepo: '.repository', globalMavenSettingsConfig : 'maven-global', mavenSettingsConfig : 'maven-user')

          { sh 'mvn clean package' }

          }
          }
          }

          need to add my maven settings to get it going. Still see 'nohup' issue
           

           

           

          dan tran added a comment - ran your example  node { stage('build') { docker.image('maven:3.5.0-jdk-8').inside { git 'https://github.com/spring-projects/spring-petclinic.git' withMaven( mavenLocalRepo: '.repository', globalMavenSettingsConfig : 'maven-global', mavenSettingsConfig : 'maven-user') { sh 'mvn clean package' } } } } need to add my maven settings to get it going. Still see 'nohup' issue      

          dan tran added a comment -

          It turns out my jenkins slave/node configuration has
          Prepare jobs environment checked and Unset System Environment Variables checked

          This is common settings at all of our jenkins slave that I have no control

          I think it is still a bug since it works with using configFileProvider directly

          configFileProvider([
          configFile(fileId: 'maven-user', variable: 'MVN_USER_SETTINGS'),
          configFile(fileId: 'maven-global', variable: 'MVN_GLOBAL_SETTINGS')
          ])

          { sh " mvn ..." }

          dan tran added a comment - It turns out my jenkins slave/node configuration has Prepare jobs environment checked and Unset System Environment Variables checked This is common settings at all of our jenkins slave that I have no control I think it is still a bug since it works with using configFileProvider directly configFileProvider([ configFile(fileId: 'maven-user', variable: 'MVN_USER_SETTINGS'), configFile(fileId: 'maven-global', variable: 'MVN_GLOBAL_SETTINGS') ]) { sh " mvn ..." }

          Nuno Costa added a comment -

          alobato, can you replicate the issue with this steps?

          cleclerc , I tried your test case on a ubuntu 14.04 build node and it ran ok.
          Running it on a pretty simple Centos7 build node, failed with the nohup error.

          No global environment variables are set on Jenkins "Configure System - Global properties"

          Connecting to the centos buildserver and switch to the jenkins user, I get this results:

          [jenkins@centos7-buildserver-02 ~]$ ll /usr/bin/bash
          -rwxr-xr-x. 1 root root 960472 Dec 7 00:19 /usr/bin/bash
          [jenkins@centos7-buildserver-02 ~]$ ll /usr/bin/sh
          lrwxrwxrwx. 1 root root 4 Dec 15 12:10 /usr/bin/sh -> bash
          [jenkins@centos7-buildserver-02 ~]$ ll /bin/sh
          lrwxrwxrwx. 1 root root 4 Dec 15 12:10 /bin/sh -> bash
          [jenkins@centos7-buildserver-02 ~]$ ll /bin/bash
          -rwxr-xr-x. 1 root root 960472 Dec 7 00:19 /bin/bash
          ...
          [jenkins@centos7-buildserver-02 ~]$ which nohup
          /bin/nohup
          [jenkins@centos7-buildserver-02 ~]$ whereis nohup
          nohup: /usr/bin/nohup /usr/share/man/man1/nohup.1.gz /usr/share/man/man1p/nohup.1p.gz
          [jenkins@centos7-buildserver-02 ~]$ echo $PATH
          /usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/var/lib/jenkins/.local/bin:/var/lib/jenkins/bin
          ...
          [jenkins@centos7-buildserver-02 ~]$ which sh
          /bin/sh
          [jenkins@centos7-buildserver-02 ~]$ whereis sh
          sh: /usr/bin/sh /usr/share/man/man1/sh.1.gz /usr/share/man/man1p/sh.1p.gz
          ...
          bash_profile
          PATH=$PATH:$HOME/.local/bin:$HOME/bin

          Adding an echo $PATH inside the build stage but outside the container, I noticed that the /bin was missing from there.
          This PATH value seems to be configured by the Jenkins ssh agent connection and the PATH variable set on jenkins user bash_profile is not used.

          node('centos-buildserver-02') {
          stage('build') {
                  sh 'echo $PATH'
                  sh 'which nohup'
                  sh 'which sh'
                  docker.image('maven:3.5.0-jdk-8').inside {
                      git 'https://github.com/spring-projects/spring-petclinic.git'
                      sh 'echo $PATH'
                      sh 'which nohup'
                      sh 'which sh'
                      withMaven() {
                          sh 'mvn clean package'
                      }
                  }
              }
          }
          

          node agent config from the connection log
          PATH=/usr/local/bin:/usr/bin

          After adding PATH with /usr/local/bin:/usr/bin:/bin as a environment variable on that specific node, on "Node Properties - Environment variables", the build run sucessfully.

          What is strange is that the agent node connection log still misses the added /bin and only shows PATH=/usr/local/bin:/usr/bin after disconnect and reconnect, although the sh'echo $PATH' shows "/usr/local/bin:/usr/bin:/bin"

          [Pipeline] stage
          [Pipeline] { (build)
          [Pipeline] sh
          [withMaven-test2] Running shell script
          + echo /usr/local/bin:/usr/bin:/bin
          /usr/local/bin:/usr/bin:/bin
          [Pipeline] sh
          

          On Ubuntu 14.04, sh is only installed on /bin/sh

          node agent config
          PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

          jenkins@ub14-buildserver-02:~$ echo $PATH
           /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games

          Is the behaviour shown on CentOS7 "working as designed" or something that needs to be corrected in Jenkins/Pipeline/ssh agent itself?

          Nuno Costa added a comment - alobato , can you replicate the issue with this steps? cleclerc , I tried your test case on a ubuntu 14.04 build node and it ran ok. Running it on a pretty simple Centos7 build node, failed with the nohup error. No global environment variables are set on Jenkins "Configure System - Global properties" Connecting to the centos buildserver and switch to the jenkins user, I get this results: [jenkins@centos7-buildserver-02 ~]$ ll /usr/bin/bash -rwxr-xr-x. 1 root root 960472 Dec 7 00:19 /usr/bin/bash [jenkins@centos7-buildserver-02 ~]$ ll /usr/bin/sh lrwxrwxrwx. 1 root root 4 Dec 15 12:10 /usr/bin/sh -> bash [jenkins@centos7-buildserver-02 ~]$ ll /bin/sh lrwxrwxrwx. 1 root root 4 Dec 15 12:10 /bin/sh -> bash [jenkins@centos7-buildserver-02 ~]$ ll /bin/bash -rwxr-xr-x. 1 root root 960472 Dec 7 00:19 /bin/bash ... [jenkins@centos7-buildserver-02 ~]$ which nohup /bin/nohup [jenkins@centos7-buildserver-02 ~]$ whereis nohup nohup: /usr/bin/nohup /usr/share/man/man1/nohup.1.gz /usr/share/man/man1p/nohup.1p.gz [jenkins@centos7-buildserver-02 ~]$ echo $PATH /usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/ var /lib/jenkins/.local/bin:/ var /lib/jenkins/bin ... [jenkins@centos7-buildserver-02 ~]$ which sh /bin/sh [jenkins@centos7-buildserver-02 ~]$ whereis sh sh: /usr/bin/sh /usr/share/man/man1/sh.1.gz /usr/share/man/man1p/sh.1p.gz ... bash_profile PATH=$PATH:$HOME/.local/bin:$HOME/bin Adding an echo $PATH inside the build stage but outside the container, I noticed that the /bin was missing from there. This PATH value seems to be configured by the Jenkins ssh agent connection and the PATH variable set on jenkins user bash_profile is not used. node( 'centos-buildserver-02' ) { stage( 'build' ) { sh 'echo $PATH' sh 'which nohup' sh 'which sh' docker.image( 'maven:3.5.0-jdk-8' ).inside { git 'https: //github.com/spring-projects/spring-petclinic.git' sh 'echo $PATH' sh 'which nohup' sh 'which sh' withMaven() { sh 'mvn clean package ' } } } } node agent config from the connection log PATH=/usr/local/bin:/usr/bin After adding PATH with /usr/local/bin:/usr/bin:/bin as a environment variable on that specific node, on "Node Properties - Environment variables" , the build run sucessfully. What is strange is that the agent node connection log still misses the added /bin and only shows PATH=/usr/local/bin:/usr/bin after disconnect and reconnect, although the sh'echo $PATH' shows "/usr/local/bin:/usr/bin:/bin" [Pipeline] stage [Pipeline] { (build) [Pipeline] sh [withMaven-test2] Running shell script + echo /usr/local/bin:/usr/bin:/bin /usr/local/bin:/usr/bin:/bin [Pipeline] sh On Ubuntu 14.04, sh is only installed on /bin/sh node agent config PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games jenkins@ub14-buildserver-02:~$ echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games Is the behaviour shown on CentOS7 "working as designed" or something that needs to be corrected in Jenkins/Pipeline/ssh agent itself?

          Stéphane Tournié added a comment - - edited

          I'm getting this error too when I configure a "provided settings.xml" and/or a "provided global settings.xml" in the Maven configuration in tools global configuration page.
          When I change it to "use default Maven settings" and call the withMaven() method without any parameter I no more get the nohup: failed to run command `sh': No such file or directory error.

          But then I'm getting another error that I cannot solve so far: script returned exit code -2.

          Stéphane Tournié added a comment - - edited I'm getting this error too when I configure a "provided settings.xml" and/or a "provided global settings.xml" in the Maven configuration in tools global configuration page. When I change it to "use default Maven settings" and call the withMaven() method without any parameter I no more get the nohup: failed to run command `sh': No such file or directory error. But then I'm getting another error that I cannot solve so far: script returned exit code -2 .

          stour can you please provide more details with the pipeline script, the version of the plugin and the logs? This problem seem to be specific to some hosts (centos / rhel) and does not seem to happen on Ubuntu.

          Cyrille Le Clerc added a comment - stour can you please provide more details with the pipeline script, the version of the plugin and the logs? This problem seem to be specific to some hosts (centos / rhel) and does not seem to happen on Ubuntu.

          Here is my pipeline script:

          node('docker-installed') {
            docker.image('maven:3.3.9-jdk-8-alpine').inside() {
              git branch: "${branch}", url: "$url"
              withMaven() {
                sh "mvn -f workspace/project/pom.xml clean install -s $settingsXml"
              }
            }
          }
          

          I'm running Jenkins 2.64 and pipeline-maven-plugin 2.2.0. I'm running Jenkins in a Docker container and I use a remote SSH slave that has Docker installed to execute my pipelines that require Docker.

          Here are the logs of an execution of this stage.

          Stéphane Tournié added a comment - Here is my pipeline script: node( 'docker-installed' ) { docker.image( 'maven:3.3.9-jdk-8-alpine' ).inside() { git branch: "${branch}" , url: "$url" withMaven() { sh "mvn -f workspace/project/pom.xml clean install -s $settingsXml" } } } I'm running Jenkins 2.64 and pipeline-maven-plugin 2.2.0. I'm running Jenkins in a Docker container and I use a remote SSH slave that has Docker installed to execute my pipelines that require Docker. Here are the logs of an execution of this stage.

          dantran, ncosta, stour, testuser7 can you please confirm the detailed version of the Operating System of your build agent and the version of the docker engine on this agent? The problem seem to only happen on RHEL/CentOS 6 and 7 and on some versions of the docker engine

           

          ncosta I found above that you mentioned "CentOS7 + docker 1.13.1", can you please confirm?

          Cyrille Le Clerc added a comment - dantran , ncosta , stour , testuser7 can you please confirm the detailed version of the Operating System of your build agent and the version of the docker engine on this agent? The problem seem to only happen on RHEL/CentOS 6 and 7 and on some versions of the docker engine   ncosta I found above that you mentioned "CentOS7 + docker 1.13.1", can you please confirm?

          dan tran added a comment - - edited

          Mine is sles12sp2/docker 1.12.6. Only happens if I unset system environment variable at node configuration, see

          dan tran added a comment - - edited Mine is sles12sp2/docker 1.12.6. Only happens if I unset system environment variable at node configuration, see

          My build agent is Centos OS 7 + Docker 17.05.0-ce.

          Stéphane Tournié added a comment - My build agent is Centos OS 7 + Docker 17.05.0-ce.

          Cyrille Le Clerc added a comment - - edited

          Another platform where the problem is reported: 

          Master: Red Hat Enterprise Linux Server release 6.7 (Santiago)

          Build agent: Red Hat Enterprise Linux Server release 7.3 (Maipo) + docker 1.12.6

           problem does not seem to happen with docker 1.11

           

          Cyrille Le Clerc added a comment - - edited Another platform where the problem is reported:  Master: Red Hat Enterprise Linux Server release 6.7 (Santiago) Build agent: Red Hat Enterprise Linux Server release 7.3 (Maipo) + docker 1.12.6  problem does not seem to happen with docker 1.11  

          Nuno Costa added a comment - - edited

          cleclerc, On Production, we are on CentOS7 with below package versions. We are using kernel 4.x because we need it to use the docker overlay driver.

          docker-engine.x86_64 >> 1.13.1-1.el7.centos >> @docker-main
          kernel-lt.x86_64 >> 4.4.50-1.el7.elrepo >> @elrepo-kernel
          

          I also have the same issue on my Test Build Server VM with CentOS7 with:

          docker-ce.x86_64 >> 17.06.0.ce-1.el7.centos >> @docker-ce-stable
          kernel.x86_64 >> 3.10.0-514.26.1.el7 >> @updates
          kernel-lt.x86_64 >> 4.4.75-1.el7.elrepo >> @elrepo-kernel

          Starting the Test Build Server with kernel 3 or 4, provides the same results, when building:

          nohup: failed to run command ‘sh’: No such file or directory

           

          Nuno Costa added a comment - - edited cleclerc , On Production, we are on CentOS7 with below package versions. We are using kernel 4.x because we need it to use the docker overlay driver. docker-engine.x86_64 >> 1.13.1-1.el7.centos >> @docker-main kernel-lt.x86_64 >> 4.4.50-1.el7.elrepo >> @elrepo-kernel I also have the same issue on my Test Build Server VM with CentOS7 with: docker-ce.x86_64 >> 17.06.0.ce-1.el7.centos >> @docker-ce-stable kernel.x86_64 >> 3.10.0-514.26.1.el7 >> @updates kernel-lt.x86_64 >> 4.4.75-1.el7.elrepo >> @elrepo-kernel Starting the Test Build Server with kernel 3 or 4, provides the same results, when building: nohup: failed to run command ‘sh’: No such file or directory  

          Code changed in jenkins
          User: Cyrille Le Clerc
          Path:
          jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java
          http://jenkins-ci.org/commit/pipeline-maven-plugin/0193d7c1b1be95024d35b149d37f050b415fcb20
          Log:
          JENKINS-40484 Better troubleshooting messages

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java http://jenkins-ci.org/commit/pipeline-maven-plugin/0193d7c1b1be95024d35b149d37f050b415fcb20 Log: JENKINS-40484 Better troubleshooting messages

          Code changed in jenkins
          User: Cyrille Le Clerc
          Path:
          jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java
          http://jenkins-ci.org/commit/pipeline-maven-plugin/59702c5b9631b80522cbad939a9931d08b05f04a
          Log:
          Merge pull request #65 from jenkinsci/JENKINS-40484

          JENKINS-40484 Better troubleshooting messages

          Compare: https://github.com/jenkinsci/pipeline-maven-plugin/compare/b9bee3c497cc...59702c5b9631

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Cyrille Le Clerc Path: jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java http://jenkins-ci.org/commit/pipeline-maven-plugin/59702c5b9631b80522cbad939a9931d08b05f04a Log: Merge pull request #65 from jenkinsci/ JENKINS-40484 JENKINS-40484 Better troubleshooting messages Compare: https://github.com/jenkinsci/pipeline-maven-plugin/compare/b9bee3c497cc...59702c5b9631

          I have added some log messages as part of 2.5.1 to better understand the value of the PATH environment variable.

          Could you please enable logs at level FINER on org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution and attach the logs generated by a build execution to this ticket.

          The new log messages will help us understand the value of the PATH environment variable:

          Jul 17, 2017 4:26:54 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Options: [...]
          
          Jul 17, 2017 4:26:54 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          env.PATH: /usr/local/...
          
          ...
          
          Jul 17, 2017 4:27:01 PM FINER org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          ExpanderImpl(overrides: {GLOBAL_MVN_SETTINGS=/Users/cyrilleleclerc/...
          
          Jul 17, 2017 4:27:01 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          envOverride: {GLOBAL_MVN_SETTINGS=/Users/cyrilleleclerc/...
          
          Jul 17, 2017 4:27:11 PM FINER org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          ExpanderImpl.expand - env before expand: {__CF_USER_TEXT_ENCODING=...
          
          Jul 17, 2017 4:27:11 PM FINER org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          ExpanderImpl.expand - env after expand: {__CF_USER_TEXT_ENCODING=...
          

          Cyrille Le Clerc added a comment - I have added some log messages as part of 2.5.1 to better understand the value of the PATH environment variable. Could you please enable logs at level FINER on org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution and attach the logs generated by a build execution to this ticket. The new log messages will help us understand the value of the PATH environment variable: Jul 17, 2017 4:26:54 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Options: [...] Jul 17, 2017 4:26:54 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution env.PATH: /usr/local/... ... Jul 17, 2017 4:27:01 PM FINER org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution ExpanderImpl(overrides: {GLOBAL_MVN_SETTINGS=/Users/cyrilleleclerc/... Jul 17, 2017 4:27:01 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution envOverride: {GLOBAL_MVN_SETTINGS=/Users/cyrilleleclerc/... Jul 17, 2017 4:27:11 PM FINER org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution ExpanderImpl.expand - env before expand: {__CF_USER_TEXT_ENCODING=... Jul 17, 2017 4:27:11 PM FINER org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution ExpanderImpl.expand - env after expand: {__CF_USER_TEXT_ENCODING=...

          Test User added a comment -

          Logs from version 2.5.2:

          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Maven: null
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Jdk: null
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          MavenOpts: 
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Settings Config: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Settings FilePath: 
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Global settings Config: null
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Global settings FilePath: 
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Options: []
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          env.PATH: /usr/local/bin:/usr/bin
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Computer: slave1
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Env: {_=/usr/bin/java, HOME=/var/lib/jenkins, LOGNAME=jenkins, MAIL=/var/mail/jenkins, NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat, OLDPWD=/var/lib/jenkins, PATH=/usr/local/bin:/usr/bin, PWD=/var/lib/jenkins, SELINUX_LEVEL_REQUESTED=, SELINUX_ROLE_REQUESTED=, SELINUX_USE_CURRENT_RANGE=, SHELL=/bin/bash, SHLVL=1, SSH_CLIENT=*.*.*.* * *, SSH_CONNECTION=*.*.*.* * *.*.*.* *, USER=jenkins, XDG_RUNTIME_DIR=/run/user/*, XDG_SESSION_ID=*, XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt}
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Step running within docker.image()
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Setting up maven: null
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Searching for Maven through MAVEN_HOME and M2_HOME environment variables...
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          Calling printenv on docker container...
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          NO maven installation discovered on docker container through MAVEN_HOME and M2_HOME environment variables
          
          Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution
          No Maven Installation or MAVEN_HOME found, looking for mvn executable by using which/where command
          

          Test User added a comment - Logs from version 2.5.2: Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Maven: null Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Jdk: null Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution MavenOpts: Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Settings Config: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Settings FilePath: Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Global settings Config: null Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Global settings FilePath: Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Options: [] Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution env.PATH: /usr/local/bin:/usr/bin Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Computer: slave1 Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Env: {_=/usr/bin/java, HOME=/var/lib/jenkins, LOGNAME=jenkins, MAIL=/var/mail/jenkins, NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat, OLDPWD=/var/lib/jenkins, PATH=/usr/local/bin:/usr/bin, PWD=/var/lib/jenkins, SELINUX_LEVEL_REQUESTED=, SELINUX_ROLE_REQUESTED=, SELINUX_USE_CURRENT_RANGE=, SHELL=/bin/bash, SHLVL=1, SSH_CLIENT=*.*.*.* * *, SSH_CONNECTION=*.*.*.* * *.*.*.* *, USER=jenkins, XDG_RUNTIME_DIR=/run/user/*, XDG_SESSION_ID=*, XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt} Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Step running within docker.image() Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Setting up maven: null Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Searching for Maven through MAVEN_HOME and M2_HOME environment variables... Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution Calling printenv on docker container... Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution NO maven installation discovered on docker container through MAVEN_HOME and M2_HOME environment variables Aug 27, 2017 3:12:23 PM FINE org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution No Maven Installation or MAVEN_HOME found, looking for mvn executable by using which/where command

          Cyrille Le Clerc added a comment - - edited

          Detailed logs of an happy path: https://gist.github.com/cyrille-leclerc/c23d403761ac802f7a8955df40631906

          Sep 06, 2017 10:04:23 PM FINEST org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution$ExpanderImpl expand
          ...
          PATH=/home/ubuntu/jenkins-aws-home/workspace/plugins/pipeline-maven-plugin/spring-petclinic/spring-petclinic-in-docker-pipeline-plugin@tmp/withMavene56239cc:
          /usr/local/sbin:
          /usr/local/bin:
          /usr/sbin:
          /usr/bin:
          /sbin:
          /bin:
          /usr/games:
          /usr/local/games
          
          sh 'env'
          ...
          PATH=/usr/local/sbin:
          /usr/local/bin:
          /usr/sbin:
          /usr/bin:
          /sbin:
          /bin
          

          Cyrille Le Clerc added a comment - - edited Detailed logs of an happy path: https://gist.github.com/cyrille-leclerc/c23d403761ac802f7a8955df40631906 Sep 06, 2017 10:04:23 PM FINEST org.jenkinsci.plugins.pipeline.maven.WithMavenStepExecution$ExpanderImpl expand ... PATH=/home/ubuntu/jenkins-aws-home/workspace/plugins/pipeline-maven-plugin/spring-petclinic/spring-petclinic-in-docker-pipeline-plugin@tmp/withMavene56239cc: /usr/local/sbin: /usr/local/bin: /usr/sbin: /usr/bin: /sbin: /bin: /usr/games: /usr/local/games sh 'env' ... PATH=/usr/local/sbin: /usr/local/bin: /usr/sbin: /usr/bin: /sbin: /bin

          Our problem seem to be the same as JENKINS-41339 and JENKINS-41204. I didn't find a path to move forward, the troubleshooting message we have added don't help. I'll ping jglick to see if he can help us as this seem to be more related to "pipeline core" and "docker-pipeline" with old Docker engines than to be a withMaven specific issue.

          Cyrille Le Clerc added a comment - Our problem seem to be the same as JENKINS-41339 and JENKINS-41204 . I didn't find a path to move forward, the troubleshooting message we have added don't help. I'll ping jglick to see if he can help us as this seem to be more related to "pipeline core" and "docker-pipeline" with old Docker engines than to be a withMaven specific issue.

          Cyrille Le Clerc added a comment - - edited

           I have added a "known limitation" to the withMaven plugin page: "withMaven() not supported in "docker.image('xxx').inside{...}" with old versions of the Docker engine

          Cyrille Le Clerc added a comment - - edited  I have added a "known limitation" to the withMaven plugin page: " withMaven() not supported in "docker.image('xxx').inside{...}" with old versions of the Docker engine " 

          dan tran added a comment -

          for my case(sles12sp2), same issue found with latest jenkins plugin + docker 17.4

          dan tran added a comment - for my case(sles12sp2), same issue found with latest jenkins plugin + docker 17.4

          Jesse Glick added a comment -

          cleclerc I am not sure what the relation is to the issues you mention. Most likely a bug in docker-workflow, or perhaps JENKINS-39307 or similar. At any rate I am not working on this any more, and generally recommend stopping use of Image.inside at the first sign of trouble.

          Jesse Glick added a comment - cleclerc I am not sure what the relation is to the issues you mention. Most likely a bug in docker-workflow , or perhaps  JENKINS-39307 or similar. At any rate I am not working on this any more, and generally recommend stopping use of Image.inside at the first sign of trouble.

          Nuno Costa added a comment -

          I've seen this issue using both docker 1.x and 17.x versions.
          At the moment all buildservers are using 17.x versions.

          Noticed that the the build runs OK on a specific version (container ID 88714384d642) of maven:3.5.0-jdk-8 image.
          According to docker images command, it was created 7 weeks ago.

          Also noticed that with latest image (ID cbf114925530 created ~31 hours ago) and a older one (ID 66091267e43d created 4 months ago), the build fails with the sh error.

          Compared the logs created by Jenkins Logger (as cleclerc mentioned in this comment but I didn't notice anything that could indicate any issue. Both Sucess and Failure builds had env.PATH: /usr/local/bin:/usr/bin

          Attaching console and logger logs from failed and sucessfull builds:
          20170915-v.2.78-JENKINS-40484-CONSOLE-LOG-NOK.txt
          20170915-v.2.78-JENKINS-40484-JENKINS-LOG-NOK.txt

          20170915-v.2.78-JENKINS-40484-CONSOLE-LOG-OK.txt
          20170915-v.2.78-JENKINS-40484-JENKINS-LOG-OK.txt

          What I could see from Console log was the sh command being used from a different location.
          Build finishes OK when sh is called from /usr/bin/sh (image ID 88714384d642) but fails when sh is called from /bin/sh (image ID cbf114925530).
          Used the which sh and whereis sh commands to check.

          Retested today with a fully patched centos7 node and centos7 jenkins 2.78 server(fully updated plugins as well).

          Additionally, tested on a fully patched ubuntu 14.04 server and I don't have any build issues using image ID cbf114925530.

          NOTE: Not sure how we can pull a specific ID of a docker image.
          In my case, I was manually tagging the latest version, adding the ID, since I'm following this issue.
          This way I could check if a new update was made on the image, every time I pulled the image.

          Jenkinsfile used for testing:

          node('mydocker') {
          deleteDir()
          stage('build') {
                  sh 'echo $PATH'
                  sh 'which nohup'
                  sh 'which sh'
                  docker.image('maven:3.5.0-jdk-8-cbf114925530').inside {
                      git 'https://github.com/spring-projects/spring-petclinic.git'
                      sh 'echo $PATH'
                      sh 'which nohup'
                      sh 'which sh'
                      withMaven() {
                          sh 'mvn clean package'
                      }
                  }
              }
          deleteDir()
          }
          

          Nuno Costa added a comment - I've seen this issue using both docker 1.x and 17.x versions. At the moment all buildservers are using 17.x versions. Noticed that the the build runs OK on a specific version (container ID 88714384d642) of maven:3.5.0-jdk-8 image. According to docker images command, it was created 7 weeks ago. Also noticed that with latest image (ID cbf114925530 created ~31 hours ago) and a older one (ID 66091267e43d created 4 months ago), the build fails with the sh error. Compared the logs created by Jenkins Logger (as cleclerc mentioned in this comment but I didn't notice anything that could indicate any issue. Both Sucess and Failure builds had env.PATH: /usr/local/bin:/usr/bin Attaching console and logger logs from failed and sucessfull builds: 20170915-v.2.78-JENKINS-40484-CONSOLE-LOG-NOK.txt 20170915-v.2.78-JENKINS-40484-JENKINS-LOG-NOK.txt 20170915-v.2.78-JENKINS-40484-CONSOLE-LOG-OK.txt 20170915-v.2.78-JENKINS-40484-JENKINS-LOG-OK.txt What I could see from Console log was the sh command being used from a different location. Build finishes OK when sh is called from /usr/bin/sh (image ID 88714384d642) but fails when sh is called from /bin/sh (image ID cbf114925530). Used the which sh and whereis sh commands to check. Retested today with a fully patched centos7 node and centos7 jenkins 2.78 server(fully updated plugins as well). Additionally, tested on a fully patched ubuntu 14.04 server and I don't have any build issues using image ID cbf114925530. NOTE: Not sure how we can pull a specific ID of a docker image. In my case, I was manually tagging the latest version, adding the ID, since I'm following this issue. This way I could check if a new update was made on the image, every time I pulled the image. Jenkinsfile used for testing: node( 'mydocker' ) { deleteDir() stage( 'build' ) { sh 'echo $PATH' sh 'which nohup' sh 'which sh' docker.image( 'maven:3.5.0-jdk-8-cbf114925530' ).inside { git 'https: //github.com/spring-projects/spring-petclinic.git' sh 'echo $PATH' sh 'which nohup' sh 'which sh' withMaven() { sh 'mvn clean package ' } } } deleteDir() }

          jglick can you please point us to documentation for an alternative to docker.image().inside{}?

          Cyrille Le Clerc added a comment - jglick can you please point us to documentation for an alternative to docker.image().inside{}?

          Jesse Glick added a comment -

          Jesse Glick added a comment - https://github.com/jenkinsci/docker-workflow-plugin/pull/105/files  for example.

          Jesse Glick added a comment -

          Jesse Glick added a comment - Or: https://github.com/jglick/ephemeral-docker-agent-plugin  (CC abayer , ndeloof )

          Andrew Bayer added a comment -

          Andrew Bayer added a comment - FYI, https://github.com/jenkinsci/docker-workflow-plugin/pull/97 does seem to resolve this.

          Andrew Bayer added a comment -

          Fixed in docker-workflow 1.14, which I'm releasing as we speak.

          Andrew Bayer added a comment - Fixed in docker-workflow 1.14, which I'm releasing as we speak.

          dan tran added a comment -

          awesome!! thank you very much for this long waited fix

          dan tran added a comment - awesome!! thank you very much for this long waited fix

          Tim Downey added a comment -

          Hello, the fix for this in Docker Pipeline 1.14 seems to have broken things.  I've been using withMaven & Docker successfully for a while now and things have been working up until 1.13.

          Here's what I've been doing in the builds.  In this case, I'm using a generated settings.xml, but I have another build that just passes in an identifier for a managed settings.xml and I get the same results.

           

          // I'm generating this elsewhere
          def settingsFilePath = 'target/generated-settings.xml'
          
          // simplified for brevity
          withDockerContainer('maven:3.5.0-jdk-8-alpine') {
              withMaven(mavenSettingsFilePath: settingsFilePath) {
                 sh 'mvn -U -X clean install'
              }
          }

          Since upgrading to 1.14, attempts to set the settings.xml fail because of what appears to be changes to the user home.

          Here's what I'm seeing.  Notice that the debug logic shows in the target/generated-settings.xml as in use, but before below in the true Maven debug, you can see that user settings got set to ?./m2/settings.xml instead.

          This did not happen with 1.13.

           

          $ docker run -t -d -u 1000:482 -w /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy --volumes-from e24fc4ecfb9e762918e653126582a67aa6c907086b279d581d1b998f4915aa3d -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat maven:3.5.0-jdk-8-alpine
          [Pipeline] {
          [Pipeline] withMaven
          [withMaven] Options: [org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher[disabled=true], org.jenkinsci.plugins.pipeline.maven.publishers.TasksScannerPublisher[disabled=true]]
          [withMaven] Available options: 
          [withMaven] use JDK installation provided by the build agent
          $ docker exec 145d411e24f49e2140dd3a1a1fb9db26dfe0fa5e7d61553528bcd946e9478914 env printenv MAVEN_HOME
          [withMaven] use Maven installation provided by the build agent with the environment variable MAVEN_HOME=/usr/share/maven
          [withMaven] use Maven settings provided on the build agent 'target/generated-settings.xml' 
          [Pipeline] {
          [Pipeline] echo
          Downloading com.oneadp.taas.apigateway:taas-apigateway-payroll:$VERSION to temp folder
          [Pipeline] sh
          [deploy] Running shell script
          + mvn -U -X org.apache.maven.plugins:maven-dependency-plugin:3.0.1:copy -Dartifact=com.oneadp.taas.apigateway:taas-apigateway-payroll:1.0.2-SNAPSHOT:zip -DoutputDirectory=target -Dmdep.useBaseVersion=true
          Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e" 
          Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T19:39:06Z)
          Maven home: /usr/share/maven
          Java version: 1.8.0_131, vendor: Oracle Corporation
          Java home: /usr/lib/jvm/java-1.8-openjdk/jre
          Default locale: en_US, platform encoding: UTF-8
          OS name: "linux", version: "3.10.0-514.21.2.el7.x86_64", arch: "amd64", family: "unix"
          [DEBUG]   Included /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/pipeline-maven-spy.jar
          [DEBUG] Populating class realm maven.ext
          [DEBUG]   Included /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/pipeline-maven-spy.jar
          [jenkins-maven-event-spy] INFO generate /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/maven-spy-20171106-211420-4734323727761713157281.log.tmp ...
          [DEBUG] Created new class realm maven.api
          [DEBUG] Importing foreign packages into class realm maven.api
          [DEBUG]   Imported: javax.enterprise.inject.* < maven.ext
          [DEBUG]   Imported: javax.enterprise.util.* < maven.ext
          
          *** lots clipped ***
          
          [DEBUG]   Imported: org.slf4j.* < maven.ext
          [DEBUG]   Imported: org.slf4j.helpers.* < maven.ext
          [DEBUG]   Imported: org.slf4j.spi.* < maven.ext
          [DEBUG] Populating class realm maven.api
          [INFO] Error stacktraces are turned on.
          [DEBUG] Message scheme: color
          [DEBUG] Message styles: debug info warning error success failure strong mojo project
          [DEBUG] Reading global settings from /usr/share/maven/conf/settings.xml
          [DEBUG] Reading user settings from ?/.m2/settings.xml
          [DEBUG] Reading global toolchains from /usr/share/maven/conf/toolchains.xml
          [DEBUG] Reading user toolchains from ?/.m2/toolchains.xml
          [DEBUG] Using local repository at /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy/?/.m2/repository
          [DEBUG] Using manager EnhancedLocalRepositoryManager with priority 10.0 for /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy/?/.m2/repository
          [INFO] Scanning for projects...
          [DEBUG] Extension realms for project org.apache.maven:standalone-pom:pom:1: (none)
          [DEBUG] Looking up lifecycle mappings for packaging pom from ClassRealm[maven.ext, parent: ClassRealm[plexus.core, parent: null]]
          [DEBUG] Using transporter WagonTransporter with priority -1.0 for https://repo.maven.apache.org/maven2
          [DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://repo.maven.apache.org/maven2
          Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/3.0.1/maven-dependency-plugin-3.0.1.pom
          [DEBUG] Writing tracking file /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy/?/.m2/repository/org/apache/maven/plugins/maven-dependency-plugin/3.0.1/maven-dependency-plugin-3.0.1.pom.lastUpdated
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD FAILURE
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 02:07 min
          

          This issue may be related

          https://stackoverflow.com/questions/12465186/why-does-maven-use-as-my-home-directory

          Thanks for any suggestions.

           

           

           

          Tim Downey added a comment - Hello, the fix for this in Docker Pipeline 1.14 seems to have broken things.  I've been using withMaven & Docker successfully for a while now and things have been working up until 1.13. Here's what I've been doing in the builds.  In this case, I'm using a generated settings.xml, but I have another build that just passes in an identifier for a managed settings.xml and I get the same results.   // I'm generating this elsewhere def settingsFilePath = 'target/generated-settings.xml' // simplified for brevity withDockerContainer( 'maven:3.5.0-jdk-8-alpine' ) { withMaven(mavenSettingsFilePath: settingsFilePath) { sh 'mvn -U -X clean install' } } Since upgrading to 1.14, attempts to set the settings.xml fail because of what appears to be changes to the user home. Here's what I'm seeing.  Notice that the debug logic shows in the target/generated-settings.xml as in use, but before below in the true Maven debug, you can see that user settings got set to ?./m2/settings.xml instead. This did not happen with 1.13.   $ docker run -t -d -u 1000:482 -w /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy --volumes-from e24fc4ecfb9e762918e653126582a67aa6c907086b279d581d1b998f4915aa3d -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat maven:3.5.0-jdk-8-alpine [Pipeline] { [Pipeline] withMaven [withMaven] Options: [org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher[disabled=true], org.jenkinsci.plugins.pipeline.maven.publishers.TasksScannerPublisher[disabled=true]] [withMaven] Available options: [withMaven] use JDK installation provided by the build agent $ docker exec 145d411e24f49e2140dd3a1a1fb9db26dfe0fa5e7d61553528bcd946e9478914 env printenv MAVEN_HOME [withMaven] use Maven installation provided by the build agent with the environment variable MAVEN_HOME=/usr/share/maven [withMaven] use Maven settings provided on the build agent 'target/generated-settings.xml' [Pipeline] { [Pipeline] echo Downloading com.oneadp.taas.apigateway:taas-apigateway-payroll:$VERSION to temp folder [Pipeline] sh [deploy] Running shell script + mvn -U -X org.apache.maven.plugins:maven-dependency-plugin:3.0.1:copy -Dartifact=com.oneadp.taas.apigateway:taas-apigateway-payroll:1.0.2-SNAPSHOT:zip -DoutputDirectory=target -Dmdep.useBaseVersion=true Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e" Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T19:39:06Z) Maven home: /usr/share/maven Java version: 1.8.0_131, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-1.8-openjdk/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-514.21.2.el7.x86_64", arch: "amd64", family: "unix" [DEBUG] Included /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/pipeline-maven-spy.jar [DEBUG] Populating class realm maven.ext [DEBUG] Included /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/pipeline-maven-spy.jar [jenkins-maven-event-spy] INFO generate /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy@tmp/withMaven65ceaa4e/maven-spy-20171106-211420-4734323727761713157281.log.tmp ... [DEBUG] Created new class realm maven.api [DEBUG] Importing foreign packages into class realm maven.api [DEBUG] Imported: javax.enterprise.inject.* < maven.ext [DEBUG] Imported: javax.enterprise.util.* < maven.ext *** lots clipped *** [DEBUG] Imported: org.slf4j.* < maven.ext [DEBUG] Imported: org.slf4j.helpers.* < maven.ext [DEBUG] Imported: org.slf4j.spi.* < maven.ext [DEBUG] Populating class realm maven.api [INFO] Error stacktraces are turned on. [DEBUG] Message scheme: color [DEBUG] Message styles: debug info warning error success failure strong mojo project [DEBUG] Reading global settings from /usr/share/maven/conf/settings.xml [DEBUG] Reading user settings from ?/.m2/settings.xml [DEBUG] Reading global toolchains from /usr/share/maven/conf/toolchains.xml [DEBUG] Reading user toolchains from ?/.m2/toolchains.xml [DEBUG] Using local repository at /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy/?/.m2/repository [DEBUG] Using manager EnhancedLocalRepositoryManager with priority 10.0 for /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy/?/.m2/repository [INFO] Scanning for projects... [DEBUG] Extension realms for project org.apache.maven:standalone-pom:pom:1: (none) [DEBUG] Looking up lifecycle mappings for packaging pom from ClassRealm[maven.ext, parent: ClassRealm[plexus.core, parent: null]] [DEBUG] Using transporter WagonTransporter with priority -1.0 for https://repo.maven.apache.org/maven2 [DEBUG] Using connector BasicRepositoryConnector with priority 0.0 for https://repo.maven.apache.org/maven2 Downloading: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/3.0.1/maven-dependency-plugin-3.0.1.pom [DEBUG] Writing tracking file /home/jenkins/workspace/mule-deployment-dashboard/taas-apigateway/ipe/deploy/?/.m2/repository/org/apache/maven/plugins/maven-dependency-plugin/3.0.1/maven-dependency-plugin-3.0.1.pom.lastUpdated [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 02:07 min This issue may be related https://stackoverflow.com/questions/12465186/why-does-maven-use-as-my-home-directory Thanks for any suggestions.      

          timdowney, dantran can you please test 3.0.3-beta-1 and prepend the '$MVN_CMD' variable to the '$PATH' variable in each 'sh' step that invokes 'mvn'?

          https://github.com/jenkinsci/pipeline-maven-plugin/releases/tag/pipeline-maven-3.0.3-beta-1

           

          node("linux-agent-with-docker") {
              docker.image('maven').inside {
                  withMaven(mavenSettingsConfig: 'my-maven-settings') {
          
                      git "https://github.com/cyrille-leclerc/my-jar.git"
          
                      // prepend the $MVN_CMD variable to the $PATH environment variable in each "sh" step that invokes mvn 
                      sh 'export PATH=$MVN_CMD:$PATH && mvn help:effective-settings'
                  }
              }
          }
          

          Cyrille Le Clerc added a comment - timdowney , dantran can you please test 3.0.3-beta-1 and prepend the '$MVN_CMD' variable to the '$PATH' variable in each 'sh' step that invokes 'mvn'? https://github.com/jenkinsci/pipeline-maven-plugin/releases/tag/pipeline-maven-3.0.3-beta-1   node( "linux-agent-with-docker" ) { docker.image( 'maven' ).inside { withMaven(mavenSettingsConfig: 'my-maven-settings' ) { git "https: //github.com/cyrille-leclerc/my-jar.git" // prepend the $MVN_CMD variable to the $PATH environment variable in each "sh" step that invokes mvn sh 'export PATH=$MVN_CMD:$PATH && mvn help:effective-settings' } } }

          Tim Downey added a comment -

          Ok, running with 3.0.3-beta-1 and Docker Pipeline 1.14, I'm still showing the same error.  It is ignoring my configured settings.xml.  You can see it getting hung up at the bottom.  I'm behind a corporate proxy and without the settings.xml pointing at my Artifactory, I can't pull anything in.

          I see the warning about sh and exporting the PATH.  I'm not going to have to do that in the future, am I?  That's would be a breaking change across a lot of builds and seems fairly ugly as well.

           

          [Pipeline] node
          Running on bf3998a459f6-ba9afd01 in /home/jenkins/workspace/JENKINS-40484
          [Pipeline] {
          [Pipeline] sh
          [JENKINS-40484] Running shell script
          + docker inspect -f . maven
          
          Error: No such object: maven
          [Pipeline] sh
          [JENKINS-40484] Running shell script
          + docker pull maven
          Using default tag: latest
          latest: Pulling from library/maven
          3e17c6eae66c: Already exists
          fdfb54153de7: Pulling fs layer
          
          ** snip **
          
          9b99cd964015: Pull complete
          9d5978195527: Pull complete
          Digest: sha256:e7c0e5eb42306cbf81aebf793f9bdf15b5e5e8702c683cba626dfb161db25c4d
          Status: Downloaded newer image for maven:latest
          [Pipeline] withDockerContainer
          bf3998a459f6-ba9afd01 seems to be running inside container bf3998a459f6307d84239443d665267c974714d868d90e35a560972f7db52e2b
          $ docker run -t -d -u 1000:482 -w /home/jenkins/workspace/JENKINS-40484 --volumes-from bf3998a459f6307d84239443d665267c974714d868d90e35a560972f7db52e2b -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat maven
          [Pipeline] {
          [withMaven] Options: []
          [withMaven] Available options: 
          [withMaven] WARNING: "withMaven(){...}" step running within "docker.image('image').inside {...}". Since the Docker Pipeline Plugin version 1.14, you MUST prepend the 'MVN_CMD' environment variable to the 'PATH' environment variable in every 'sh' step that invokes 'mvn'. See Pipeline Maven Plugin FAQ.
          [withMaven] Sample:
          [withMaven]    sh "export PATH=$MVN_CMD:$PATH && mvn ..."
          [withMaven] use JDK installation provided by the build agent
          [withMaven] use Maven settings provided by the Jenkins Managed Configuration File 'my-maven-settings' 
          [withMaven] use Maven settings.xml 'my-maven-settings' with NO Maven servers credentials provided by Jenkins
          $ docker exec 06b6784e5f8ec73ccca484bc3ba35e7fbc6514a89265331fff8c416b356e0941 env printenv MAVEN_HOME
          [withMaven] use Maven installation provided by the build agent with the environment variable MAVEN_HOME=/usr/share/maven
          [Pipeline] withMaven
          [Pipeline] {
          [Pipeline] git
          Cloning the remote Git repository
          Cloning repository https://github.com/cyrille-leclerc/my-jar.git
           > git init /home/jenkins/workspace/JENKINS-40484 # timeout=10
          Fetching upstream changes from https://github.com/cyrille-leclerc/my-jar.git
           > git --version # timeout=10
           > git fetch --tags --progress https://github.com/cyrille-leclerc/my-jar.git +refs/heads/*:refs/remotes/origin/*
           > git config remote.origin.url https://github.com/cyrille-leclerc/my-jar.git # timeout=10
           > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
           > git config remote.origin.url https://github.com/cyrille-leclerc/my-jar.git # timeout=10
          Fetching upstream changes from https://github.com/cyrille-leclerc/my-jar.git
           > git fetch --tags --progress https://github.com/cyrille-leclerc/my-jar.git +refs/heads/*:refs/remotes/origin/*
           > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
           > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10
          Checking out Revision 1a00e567ba6fbc47f7757c5cb160fa4483fbd33b (refs/remotes/origin/master)
           > git config core.sparsecheckout # timeout=10
           > git checkout -f 1a00e567ba6fbc47f7757c5cb160fa4483fbd33b
           > git branch -a -v --no-abbrev # timeout=10
           > git checkout -b master 1a00e567ba6fbc47f7757c5cb160fa4483fbd33b
          Commit message: "[maven-release-plugin] prepare for next development iteration"
          First time build. Skipping changelog.
          [Pipeline] sh
          [JENKINS-40484] Running shell script
          + export PATH=/home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6/mvn:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
          + mvn help:effective-settings
          Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6" 
          [jenkins-maven-event-spy] INFO generate /home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6/maven-spy-20171107-215042-2478671379137199221991.log.tmp ...
          [INFO] Scanning for projects...
          Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-release-plugin/2.5.3/maven-release-plugin-2.5.3.pom

           

          Tim Downey added a comment - Ok, running with 3.0.3-beta-1 and Docker Pipeline 1.14, I'm still showing the same error.  It is ignoring my configured settings.xml.  You can see it getting hung up at the bottom.  I'm behind a corporate proxy and without the settings.xml pointing at my Artifactory, I can't pull anything in. I see the warning about sh and exporting the PATH.  I'm not going to have to do that in the future, am I?  That's would be a breaking change across a lot of builds and seems fairly ugly as well.   [Pipeline] node Running on bf3998a459f6-ba9afd01 in /home/jenkins/workspace/JENKINS-40484 [Pipeline] { [Pipeline] sh [JENKINS-40484] Running shell script + docker inspect -f . maven Error: No such object: maven [Pipeline] sh [JENKINS-40484] Running shell script + docker pull maven Using default tag: latest latest: Pulling from library/maven 3e17c6eae66c: Already exists fdfb54153de7: Pulling fs layer ** snip ** 9b99cd964015: Pull complete 9d5978195527: Pull complete Digest: sha256:e7c0e5eb42306cbf81aebf793f9bdf15b5e5e8702c683cba626dfb161db25c4d Status: Downloaded newer image for maven:latest [Pipeline] withDockerContainer bf3998a459f6-ba9afd01 seems to be running inside container bf3998a459f6307d84239443d665267c974714d868d90e35a560972f7db52e2b $ docker run -t -d -u 1000:482 -w /home/jenkins/workspace/JENKINS-40484 --volumes-from bf3998a459f6307d84239443d665267c974714d868d90e35a560972f7db52e2b -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat maven [Pipeline] { [withMaven] Options: [] [withMaven] Available options: [withMaven] WARNING: "withMaven(){...}" step running within "docker.image('image').inside {...}". Since the Docker Pipeline Plugin version 1.14, you MUST prepend the 'MVN_CMD' environment variable to the 'PATH' environment variable in every 'sh' step that invokes 'mvn'. See Pipeline Maven Plugin FAQ. [withMaven] Sample: [withMaven] sh "export PATH=$MVN_CMD:$PATH && mvn ..." [withMaven] use JDK installation provided by the build agent [withMaven] use Maven settings provided by the Jenkins Managed Configuration File 'my-maven-settings' [withMaven] use Maven settings.xml 'my-maven-settings' with NO Maven servers credentials provided by Jenkins $ docker exec 06b6784e5f8ec73ccca484bc3ba35e7fbc6514a89265331fff8c416b356e0941 env printenv MAVEN_HOME [withMaven] use Maven installation provided by the build agent with the environment variable MAVEN_HOME=/usr/share/maven [Pipeline] withMaven [Pipeline] { [Pipeline] git Cloning the remote Git repository Cloning repository https://github.com/cyrille-leclerc/my-jar.git > git init /home/jenkins/workspace/JENKINS-40484 # timeout=10 Fetching upstream changes from https://github.com/cyrille-leclerc/my-jar.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/cyrille-leclerc/my-jar.git +refs/heads/*:refs/remotes/origin/* > git config remote.origin.url https://github.com/cyrille-leclerc/my-jar.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > git config remote.origin.url https://github.com/cyrille-leclerc/my-jar.git # timeout=10 Fetching upstream changes from https://github.com/cyrille-leclerc/my-jar.git > git fetch --tags --progress https://github.com/cyrille-leclerc/my-jar.git +refs/heads/*:refs/remotes/origin/* > git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 1a00e567ba6fbc47f7757c5cb160fa4483fbd33b (refs/remotes/origin/master) > git config core.sparsecheckout # timeout=10 > git checkout -f 1a00e567ba6fbc47f7757c5cb160fa4483fbd33b > git branch -a -v --no-abbrev # timeout=10 > git checkout -b master 1a00e567ba6fbc47f7757c5cb160fa4483fbd33b Commit message: "[maven-release-plugin] prepare for next development iteration" First time build. Skipping changelog. [Pipeline] sh [JENKINS-40484] Running shell script + export PATH=/home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6/mvn:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + mvn help:effective-settings Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6" [jenkins-maven-event-spy] INFO generate /home/jenkins/workspace/JENKINS-40484@tmp/withMaven4ae470d6/maven-spy-20171107-215042-2478671379137199221991.log.tmp ... [INFO] Scanning for projects... Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-release-plugin/2.5.3/maven-release-plugin-2.5.3.pom  

          dan tran added a comment -

          I cant test with beta ATM, however, the fix looks good on my sample basic declarative pipeline. I need do

          here is the result of your request

          [INFO]
          Effective user-specific configuration settings:

          <?xml version="1.0" encoding="UTF-8"?>
          <!-- ====================================================================== -->
          <!-- -->
          <!-- Generated by Maven Help Plugin on 2017-11-08T01:04:15 -->
          <!-- See: http://maven.apache.org/plugins/maven-help-plugin/ -->
          <!-- -->
          <!-- ====================================================================== -->

          <!-- ====================================================================== -->
          <!-- -->
          <!-- Effective Settings for '?' on '58a31892ecc5' -->
          <!-- -->
          <!-- ====================================================================== -->

          <settings xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd">
          <localRepository xmlns="http://maven.apache.org/SETTINGS/1.1.0">/space/jenkins/skysandbox/workspace/pipeline-test/?/.m2/repository</localRepository>
          <pluginGroups xmlns="http://maven.apache.org/SETTINGS/1.1.0">
          <pluginGroup>org.apache.maven.plugins</pluginGroup>
          <pluginGroup>org.codehaus.mojo</pluginGroup>
          </pluginGroups>
          </settings>

          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 28.352 s

          dan tran added a comment - I cant test with beta ATM, however, the fix looks good on my sample basic declarative pipeline. I need do here is the result of your request [INFO] Effective user-specific configuration settings: <?xml version="1.0" encoding="UTF-8"?> <!-- ====================================================================== --> <!-- --> <!-- Generated by Maven Help Plugin on 2017-11-08T01:04:15 --> <!-- See: http://maven.apache.org/plugins/maven-help-plugin/ --> <!-- --> <!-- ====================================================================== --> <!-- ====================================================================== --> <!-- --> <!-- Effective Settings for '?' on '58a31892ecc5' --> <!-- --> <!-- ====================================================================== --> <settings xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd "> <localRepository xmlns="http://maven.apache.org/SETTINGS/1.1.0">/space/jenkins/skysandbox/workspace/pipeline-test/?/.m2/repository</localRepository> <pluginGroups xmlns="http://maven.apache.org/SETTINGS/1.1.0"> <pluginGroup>org.apache.maven.plugins</pluginGroup> <pluginGroup>org.codehaus.mojo</pluginGroup> </pluginGroups> </settings> [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 28.352 s

          Tim Downey added a comment -

          dantran and cleclerc – I'm not sure I understand the original problem that led to this issue.  I've been using withDockerContainer & withMaven for months with no issue.  What is the problem that led to this change?

          Since this change any attempt to change the configured settings.xml is ignored and instead goes looking in <workspace>/?/.m2.

          cleclerc – In trying your example, you can see that  [withMaven] is attempting to configure things properly

          [withMaven] use Maven settings.xml 'my-maven-settings' with NO Maven servers credentials provided by Jenkins

          but later when Maven actually runs, this is ignored

          [DEBUG] Reading user settings from ?/.m2/settings.xml

           

          Tim Downey added a comment - dantran and cleclerc – I'm not sure I understand the original problem that led to this issue.  I've been using withDockerContainer & withMaven for months with no issue.  What is the problem that led to this change? Since this change any attempt to change the configured settings.xml is ignored and instead goes looking in <workspace>/?/.m2. cleclerc – In trying your example, you can see that  [withMaven] is attempting to configure things properly [withMaven] use Maven settings.xml 'my-maven-settings' with NO Maven servers credentials provided by Jenkins but later when Maven actually runs, this is ignored [DEBUG] Reading user settings from ?/.m2/settings.xml  

          dan tran added a comment -

          i can confirm that my build does not see config provider settings

          dan tran added a comment - i can confirm that my build does not see config provider settings

          Cyrille Le Clerc added a comment - - edited

          Cyrille Le Clerc added a comment - - edited Please follow-up on JENKINS-47823 . See my comment https://issues.jenkins-ci.org/browse/JENKINS-47823?focusedCommentId=319156&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-319156

          Steve Todorov added a comment -

          cleclerc I would like to start by saying that really appreciate your work and efforts in providing a fix/workaround for this issue. 

          However, I'm quite disappointed by this change. When we upgraded back in November 2017 (JENKINS-47805) all of our builds turned red because of the `$PATH` changes in the Docker Pipeline plugin. This makes me doubt what's the purpose of using `withMaven` if it can't do the things it's supposed to do and we're constantly having to use "workarounds"? To name just a few - we need to always have a `post -> always` block that ensures `jUnit` records the tests, now we have to use `configFileProvider` for the `settings.xml`, maybe use the mvn wrapper script... We could just as "easily" add 4 or 5 more pipeline declarations and do things manually instead of using `withMaven`. So what's exactly does `withMaven` do other than decorate the syntax?

          Steve Todorov added a comment - cleclerc I would like to start by saying that really appreciate your work and efforts in providing a fix/workaround for this issue.  However, I'm quite disappointed by this change. When we upgraded back in November 2017 ( JENKINS-47805 ) all of our builds turned red because of the `$PATH` changes in the Docker Pipeline plugin. This makes me doubt what's the purpose of using `withMaven` if it can't do the things it's supposed to do and we're constantly having to use "workarounds"? To name just a few - we need to always have a `post -> always` block that ensures `jUnit` records the tests, now we have to use `configFileProvider` for the `settings.xml`, maybe use the mvn wrapper script... We could just as "easily" add 4 or 5 more pipeline declarations and do things manually instead of using `withMaven`. So what's exactly does `withMaven` do other than decorate the syntax?

          Joshua Noble added a comment - - edited

          I hit this issue again today for the second time. Docker Pipeline 1.14 is unusable and causes `withMaven` to break when used from within a Docker container. Using containers is one of the best features of pipelines, so this is pretty critical functionality IMO.

          Joshua Noble added a comment - - edited I hit this issue again today for the second time. Docker Pipeline 1.14 is unusable and causes `withMaven` to break when used from within a Docker container. Using containers is one of the best features of pipelines, so this is pretty critical functionality IMO.

          Joshua Noble added a comment -

          Any updates on this?

          Joshua Noble added a comment - Any updates on this?

          Steve Todorov added a comment - - edited

          ++acejam try reverting to an older version of Docker Pipeline. I have been using the older version since this issue hit me in JENKINS-47805. It seems that you can fix this either by using a mvnw, downgrade to an older version of Docker Pipeline or wait for JENKINS-48050 which looks like it will fix it for good.

          Steve Todorov added a comment - - edited ++ acejam try reverting to an older version of Docker Pipeline. I have been using the older version since this issue hit me in JENKINS-47805 . It seems that you can fix this either by using a mvnw, downgrade to an older version of Docker Pipeline or wait for JENKINS-48050  which looks like it will fix it for good.

          Joshua Noble added a comment -

          I spent a good amount of time looking into this and came up with an acceptable workaround that works for my team. We have hundreds of branches across hundreds of repos, so updating all of the Java/Maven related Jenkinsfiles (on every branch) to use $MVN_CMD instead of "mvn" was not going to happen anytime soon. We also had several pending plugin updates, many of which required the latest version of the pipeline-maven plugin. In our case, we have a private internal Docker image that we use for all maven builds. It's forked off the official Docker Hub Maven image and includes a few additional tools and packages that we need.

          What I ended up doing was creating a bash script that simply wraps the "mvn" command and checks for existence of $MVN_CMD. If $MVN_CMD exists, it calls that and passes along the same parameters. If $MVN_CMD doesn't exist, it simply calls the real maven binary located at /usr/bin/mvn. This bash script lives at /usr/local/bin/mvn, which exists in our $PATH before the real maven binary, which is /usr/bin/mvn.

          Simply create a bash script like below, and ensure it has execute permissions: (chmod +x mvn.sh) 

          #!/bin/bash
          
          if [[ -v MVN_CMD ]]; then
            $MVN_CMD "$@"
          else
            /usr/bin/mvn "$@"
          fi
          

          Then inside of your Dockerfile, add the following instruction:

          COPY mvn.sh /usr/local/bin/mvn
          

          This has been tested with the maven:3.5.3-jdk-8 Docker image.

          Joshua Noble added a comment - I spent a good amount of time looking into this and came up with an acceptable workaround that works for my team. We have hundreds of branches across hundreds of repos, so updating all of the Java/Maven related Jenkinsfiles (on every branch) to use $MVN_CMD instead of "mvn" was not going to happen anytime soon. We also had several pending plugin updates, many of which required the latest version of the pipeline-maven plugin. In our case, we have a private internal Docker image that we use for all maven builds. It's forked off the official Docker Hub Maven image and includes a few additional tools and packages that we need. What I ended up doing was creating a bash script that simply wraps the "mvn" command and checks for existence of $MVN_CMD. If $MVN_CMD exists, it calls that and passes along the same parameters. If $MVN_CMD doesn't exist, it simply calls the real maven binary located at /usr/bin/mvn. This bash script lives at /usr/local/bin/mvn, which exists in our $PATH before the real maven binary, which is /usr/bin/mvn. Simply create a bash script like below, and ensure it has execute permissions: (chmod +x mvn.sh)   #!/bin/bash if [[ -v MVN_CMD ]]; then $MVN_CMD "$@" else /usr/bin/mvn "$@" fi Then inside of your Dockerfile, add the following instruction: COPY mvn.sh /usr/local/bin/mvn This has been tested with the maven:3.5.3-jdk-8 Docker image.

          Tim Downey added a comment -

          Thanks for the writeup acejam – That's pretty much exactly what joshtrow and I have done to deal with the problem.

          Tim Downey added a comment - Thanks for the writeup acejam – That's pretty much exactly what joshtrow and I have done to deal with the problem.

            Unassigned Unassigned
            testuser7 Test User
            Votes:
            13 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated: