Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48284

pipeline-maven 3.0.3 update issues with certain maven plugins.

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • CentOS7 Jenkins 2.92 with plugins fully updated since 20171128
      CentOS7 Build Server
      Ubuntu 12.04 based custom docker image, running on Build Server.

      Opening this issue as mentioned in JENKINS-47823.

       We are using a several maven plugins that uses cmake, boost libraries, etc, and after pipeline-maven update to 3.0.3, cmake and uname (used by cmake) are not found when running maven with settings file provided by Config File Management

      Jenkinsfile example to trigger the issue.

       

      node ('mydocker'){
          def dockerRegistryURI = 'registry_URI'
          def dockerImage = 'docker_image'
          docker.withRegistry("https://${dockerRegistryURI}/", "your_registry_id") {
              buildenv = docker.image("${dockerRegistryURI}/${dockerImage}")
              buildenv.pull()
              buildenv.inside("-v /var/lib/jenkins/.m2/repository/${EXECUTOR_NUMBER}:/home/jenkins/.m2/repository"){
                  stage("Maven") {
                      def mvnsettingsid = 'your_mvn_settings_id'
                      echo '##################################'
                      echo 'OUTSIDE WITHMAVEN'
                      echo '##################################'
                      sh "echo 'PATH outside withMaven:' $PATH"
                      echo '##################################'
                      sh "which cmake"
                      echo '##################################'
                      sh "env | grep ^PATH"
                      echo '##################################'
                      withMaven(mavenSettingsConfig: "${mvnsettingsid}") {
                          echo '##################################'
                          echo 'INSIDE WITHMAVEN'
                          echo '##################################'
                          sh "echo 'PATH inside withMaven:' $PATH"
                          echo '##################################'
                          sh "which cmake"
                          echo '##################################'
                          sh "env | grep ^PATH"
                          echo '##################################'
                          sh "export PATH=$MVN_CMD_DIR:$PATH && which mvn && which cmake"
                          echo '##################################'
                          sh "echo $PATH"
                          echo '##################################'
                          sh "which cmake"
                          echo '##################################'
                          sh "env | grep ^PATH"
                          echo '##################################'
                          deleteDir()
                      }
                  }
              }            
          }
      }
      

      Consule output:

       

      [Pipeline] node
      Running on mydocker in /var/lib/jenkins/workspace/Tests/withMaven_tests/withMaven-JENKINS-47823
      [Pipeline] {
      [Pipeline] withEnv
      [Pipeline] {
      [Pipeline] withDockerRegistry
      Wrote authentication to /var/lib/jenkins/.dockercfg
      [Pipeline] {
      [Pipeline] sh
      [withMaven-JENKINS-47823] Running shell script
      + docker pull registry_URI/docker_image
      Using default tag: latest
      latest: Pulling from docker_image
      Digest: sha256:e66ca6defd6cbc4088d522d7ed38df9ac4324807e25acd97abf7a486be0778e0
      Status: Image is up to date for registry_URI/docker_image:latest
      [Pipeline] sh
      [withMaven-JENKINS-47823] Running shell script
      + docker inspect -f . registry_URI/docker_image
      .
      [Pipeline] withDockerContainer
      centos-buildserver-03 does not seem to be running inside a container
      $ docker run -t -d -u 995:991 -v /var/lib/jenkins/.m2/repository/2:/home/jenkins/.m2/repository -w /var/lib/jenkins/workspace/Tests/withMaven_tests/withMaven-JENKINS-47823 -v /var/lib/jenkins/workspace/Tests/withMaven_tests/withMaven-JENKINS-47823:/var/lib/jenkins/workspace/Tests/withMaven_tests/withMaven-JENKINS-47823:rw,z -v /var/lib/jenkins/workspace/Tests/withMaven_tests/withMaven-JENKINS-47823@tmp:/var/lib/jenkins/workspace/Tests/withMaven_tests/withMaven-JENKINS-47823@tmp:rw,z -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat registry_URI/docker_image
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Maven)
      [Pipeline] sh
      [Pipeline] echo
      ##################################
      [Pipeline] echo
      OUTSIDE WITHMAVEN
      [Pipeline] echo
      ##################################
      [Pipeline] sh
      [withMaven-JENKINS-47823] Running shell script
      + echo PATH outside withMaven: /usr/local/bin:/usr/bin
      PATH outside withMaven: /usr/local/bin:/usr/bin
      [Pipeline] echo
      ##################################
      [Pipeline] sh
      [withMaven-JENKINS-47823] Running shell script
      + which cmake
      /opt/bin/cmake
      [Pipeline] echo
      ##################################
      [Pipeline] sh
      [withMaven-JENKINS-47823] Running shell script
      + env
      + grep ^PATH
      PATH=/opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      [Pipeline] echo
      ##################################
      [Pipeline] withMaven
      [withMaven] Options: []
      [withMaven] Available options:
      [withMaven] WARNING: "withMaven(){...}" step running within "docker.image('image').inside {...}". Since the Docker Pipeline Plugin version 1.14, you MUST:
      [withMaven] * Either prepend the 'MVN_CMD_DIR' environment variable to the 'PATH' environment variable in every 'sh' step that invokes 'mvn' (e.g. "sh 'export PATH=$MVN_CMD_DIR:$PATH && mvn clean deploy' ").
      [withMaven] * Or use Takari's Maven Wrapper (e.g. "sh './mvnw clean deploy'")
      [withMaven] See Pipeline Maven Plugin FAQ.
      [withMaven] use JDK installation provided by the build agent
      [withMaven] use Maven settings provided by the Jenkins Managed Configuration File 'cb938b15-01f0-4c9d-adf2-8005204cb786'
      [withMaven] use Maven settings.xml 'cb938b15-01f0-4c9d-adf2-8005204cb786' with NO Maven servers credentials provided by Jenkins
      $ docker exec c2598b72293418339a94e1ae2d9f99dc623db90bf540146e0c34feea10dd43bc env printenv MAVEN_HOME
      $ docker exec c2598b72293418339a94e1ae2d9f99dc623db90bf540146e0c34feea10dd43bc env printenv M2_HOME
      $ docker exec c2598b72293418339a94e1ae2d9f99dc623db90bf540146e0c34feea10dd43bc env /bin/sh -c "which mvn"
      [withMaven] use Maven installation provided by the build agent with executable /opt/apache-maven-3.1.1/bin/mvn
      [Pipeline] {
      [Pipeline] echo
      ##################################
      [Pipeline] echo
      INSIDE WITHMAVEN
      [Pipeline] echo
      ##################################
      [Pipeline] sh
      [withMaven-JENKINS-47823] Running shell script
      + echo PATH inside withMaven: /var/lib/jenkins/workspace/Tests/withMaven_tests/withMaven-JENKINS-47823@tmp/withMavene9754684:/usr/local/bin:/usr/bin
      PATH inside withMaven: /var/lib/jenkins/workspace/Tests/withMaven_tests/withMaven-JENKINS-47823@tmp/withMavene9754684:/usr/local/bin:/usr/bin
      [Pipeline] echo
      ##################################
      [Pipeline] sh
      [withMaven-JENKINS-47823] Running shell script
      + which cmake
      /opt/bin/cmake
      [Pipeline] echo
      ##################################
      [Pipeline] sh
      [withMaven-JENKINS-47823] Running shell script
      + env
      + grep ^PATH
      PATH=/opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      [Pipeline] echo
      ##################################
      [Pipeline] sh
      [withMaven-JENKINS-47823] Running shell script
      + export PATH=/var/lib/jenkins/workspace/Tests/withMaven_tests/withMaven-JENKINS-47823@tmp/withMavene9754684:/var/lib/jenkins/workspace/Tests/withMaven_tests/withMaven-JENKINS-47823@tmp/withMavene9754684:/usr/local/bin:/usr/bin
      + which mvn
      /var/lib/jenkins/workspace/Tests/withMaven_tests/withMaven-JENKINS-47823@tmp/withMavene9754684/mvn
      + which cmake
      [Pipeline] }
      [Pipeline] // withMaven
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      $ docker stop --time=1 c2598b72293418339a94e1ae2d9f99dc623db90bf540146e0c34feea10dd43bc
      $ docker rm -f c2598b72293418339a94e1ae2d9f99dc623db90bf540146e0c34feea10dd43bc
      [Pipeline] // withDockerContainer
      [Pipeline] }
      [Pipeline] // withDockerRegistry
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      [withMaven] pipelineGraphPublisher - triggerDownstreamPipelines
      [withMaven] Skip downstream job triggering for upstream build with ignored result status Tests/withMaven_tests/withMaven-JENKINS-47823 #35: FAILURE
      ERROR: script returned exit code 1
      Finished: FAILURE
      

       

      which cmake command does not sucessfully run because is not found.

      PATH variable is being used with different values by jenkins.
      Doing sh "echo $PATH" and sh "env | grep ^PATH" displays 2 different values, and from my point of view, they should provide the same values.

      Example from inside the container:

       

      jenkins@1e5054ecd825:~$ echo $PATH
      /opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      jenkins@1e5054ecd825:~$ env | grep ^PATH
      PATH=/opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
      jenkins@1e5054ecd825:~$ which uname
      /bin/uname
      

       

      When adding /opt/bin like sh "export PATH=$MVN_CMD_DIR:$PATH:/bin:/opt/bin && which mvn && which cmake", cmake and uname are found.

      This seems to cause any maven plugin that uses applications that were installed from source (in our case, cmake, etc), into a different folder, additional manual adjustment to PATH needs to be done, when exporting $MVN_CMD_DIR.

       

          [JENKINS-48284] pipeline-maven 3.0.3 update issues with certain maven plugins.

          ncosta which version of docker-pipeline are you using?

          • docker-pipeline 1.13- loads the host PATH which will cause issues as you have a CentOS host and an Ubuntu image
          • docker-pipeline 1.14+ prevents plugins to contribute to the PATH and filters the PATH of the host so pipeline-maven-plugin should not cause any harm

          FYI there is an ongoing effort with JENKINS-48050 to fix dockerized build agents problems, this effort is restricted to declarative pipeline for the moment.

          Cyrille Le Clerc added a comment - ncosta which version of docker-pipeline are you using? docker-pipeline 1.13- loads the host PATH which will cause issues as you have a CentOS host and an Ubuntu image docker-pipeline 1.14+ prevents plugins to contribute to the PATH and filters the PATH of the host so pipeline-maven-plugin should not cause any harm FYI there is an ongoing effort with JENKINS-48050 to fix dockerized build agents problems, this effort is restricted to declarative pipeline for the moment.

          Nuno Costa added a comment -

          cleclerc At the moment, I'm using the docker-workflow 1.14.

          Still not sure from where this issue is coming from. Not using the withMaven step at the moment.

          Noticed that this two comands should provide different values in our case and not seeing that at the moment:

          ...
          buildenv.inside("-v /var/lib/jenkins/.m2/repository/${EXECUTOR_NUMBER}:/home/jenkins/.m2/repository"){
              stage("Maven") {
                  def mvnsettingsid = 'your_mvn_settings_id'
                  echo '##################################'
                  echo 'OUTSIDE WITHMAVEN'
                  echo '##################################'
                  sh "echo $PATH"
                  echo '##################################'
                  echo PATH
                  echo '##################################'
                  sh "env | grep ^PATH"
                  echo '##################################'
          ...
          

          I would expect sh "echo $PATH" to provide the path from inside the container:

          /opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
          

          and echo PATH to provide PATH from Jenkins or Buildserver Host:

          /usr/local/bin:/usr/bin
          

          but I'm seeing the sh command return the same PATH value of Jenkins/Buildserver host:

          ##################################
          [Pipeline] echo
          OUTSIDE WITHMAVEN
          [Pipeline] echo
          ##################################
          [Pipeline] sh
          [withMaven-JENKINS-47823] Running shell script
          + echo /usr/local/bin:/usr/bin
          /usr/local/bin:/usr/bin
          [Pipeline] echo
          ##################################
          [Pipeline] echo
          /usr/local/bin:/usr/bin
          [Pipeline] echo
          ##################################
          [Pipeline] sh
          [withMaven-JENKINS-47823] Running shell script
          + env
          + grep ^PATH
          PATH=/opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
          [Pipeline] echo
          ##################################
          

          Am I reading this wrong?

          Nuno Costa added a comment - cleclerc At the moment, I'm using the docker-workflow 1.14. Still not sure from where this issue is coming from. Not using the withMaven step at the moment. Noticed that this two comands should provide different values in our case and not seeing that at the moment: ... buildenv.inside( "-v / var /lib/jenkins/.m2/repository/${EXECUTOR_NUMBER}:/home/jenkins/.m2/repository" ){ stage( "Maven" ) { def mvnsettingsid = 'your_mvn_settings_id' echo '##################################' echo 'OUTSIDE WITHMAVEN' echo '##################################' sh "echo $PATH" echo '##################################' echo PATH echo '##################################' sh "env | grep ^PATH" echo '##################################' ... I would expect sh "echo $PATH" to provide the path from inside the container : /opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin and echo PATH to provide PATH from Jenkins or Buildserver Host : /usr/local/bin:/usr/bin but I'm seeing the sh command return the same PATH value of Jenkins/Buildserver host: ################################## [Pipeline] echo OUTSIDE WITHMAVEN [Pipeline] echo ################################## [Pipeline] sh [withMaven-JENKINS-47823] Running shell script + echo /usr/local/bin:/usr/bin /usr/local/bin:/usr/bin [Pipeline] echo ################################## [Pipeline] echo /usr/local/bin:/usr/bin [Pipeline] echo ################################## [Pipeline] sh [withMaven-JENKINS-47823] Running shell script + env + grep ^PATH PATH=/opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin [Pipeline] echo ################################## Am I reading this wrong?

            Unassigned Unassigned
            ncosta Nuno Costa
            Votes:
            3 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated: