Code changed in jenkins
User: Cyrille Le Clerc
Path:
jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java
http://jenkins-ci.org/commit/pipeline-maven-plugin/bfe173baf37face565e7edfc9bf4faf78392739a
Log:
JENKINS-47823 Add warning for docker-pipeline incompatibility + export MAVEN_CONFIG + export MVN_CMD

Code changed in jenkins
User: Cyrille Le Clerc
Path:
jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java
http://jenkins-ci.org/commit/pipeline-maven-plugin/5fc03e78b2e9f7427422575e0c901ad1095f8489
Log:
JENKINS-47823 Add warning for docker-pipeline incompatibility + export MAVEN_CONFIG + export MVN_CMD

Code changed in jenkins
User: Cyrille Le Clerc
Path:
jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java
jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java
http://jenkins-ci.org/commit/pipeline-maven-plugin/c16068a952810405452389f9374c5256d7169628
Log:
JENKINS-47823 workaround a bug in environment variable handling

Code changed in jenkins
User: Cyrille Le Clerc
Path:
jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java
jenkins-plugin/src/test/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepOnMasterTest.java
http://jenkins-ci.org/commit/pipeline-maven-plugin/c7d85c5291603ecff2c721f2255e18d9b9175a4e
Log:
JENKINS-47823 Make withMaven compatible with docker-pipeline:1.14 (#108)

JENKINS-47823 Add warning for docker-pipeline incompatibility + export MAVEN_CONFIG + export MVN_CMD

Beta release available https://github.com/jenkinsci/pipeline-maven-plugin/releases/tag/pipeline-maven-3.0.3-beta-1

Am I missing something? It appears to correctly prepend the folder, but when I look into the folder itself there is nothing there - I updated to the 3.0.3-beta-1 version of the plugin beforehand to be sure I hadn't missed anything.

It's part of a plugin:

withDockerContainer(image: image, args: config.dockerArgs) {
  withMaven(args) {
    body()
  }
}

... snip ...

[withMaven] Options: [org.jenkinsci.plugins.pipeline.maven.publishers.GeneratedArtifactsPublisher[disabled=true]]
 [withMaven] Available options: 
 [withMaven] WARNING: "withMaven(){...}" step running within "docker.image('image').inside {...}". Since the Docker Pipeline Plugin version 1.14, you MUST prepend the 'MVN_CMD' environment variable to the 'PATH' environment variable in every 'sh' step that invokes 'mvn'.
 [withMaven] Sample:
 [withMaven] sh "export PATH=$MVN_CMD:$PATH && mvn ..."
 [withMaven] use JDK installation provided by the build agent
 [withMaven] use Maven settings provided by the Jenkins Managed Configuration File 'maven-settings' 
 [withMaven] use Maven settings.xml 'maven-settings' with NO Maven servers credentials provided by Jenkins
 $ docker exec 9e8aaf7407d81f49507a6ce429e0f826950e7d4af67450178406424d2b136675 env printenv MAVEN_HOME
 [withMaven] use Maven installation provided by the build agent with the environment variable MAVEN_HOME=/usr/share/maven
 [Pipeline] {
 [Pipeline] sh
 [roject_Pipelines_rm_develop-XBJPHN7TB7XCQRDEZR56CIZD26DWWTPH7BDYFOQ6SMIYZML37NZQ] Running shell script
 + export PATH=/home/jenkins/workspace/roject_Pipelines_rm_develop-XBJPHN7TB7XCQRDEZR56CIZD26DWWTPH7BDYFOQ6SMIYZML37NZQ@tmp/withMaven7dbea4cd/mvn:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-1.8-openjdk/jre/bin:/usr/lib/jvm/java-1.8-openjdk/bin
 + mvn -X -U -DskipTests -Dmaven.test.skip=true -Pno-tests clean install
 Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/home/jenkins/workspace/roject_Pipelines_rm_develop-XBJPHN7TB7XCQRDEZR56CIZD26DWWTPH7BDYFOQ6SMIYZML37NZQ@tmp/withMaven7dbea4cd/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/home/jenkins/workspace/roject_Pipelines_rm_develop-XBJPHN7TB7XCQRDEZR56CIZD26DWWTPH7BDYFOQ6SMIYZML37NZQ@tmp/withMaven7dbea4cd" 
 Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426; 2017-04-03T19:39:06Z)
 Maven home: /usr/share/maven
 Java version: 1.8.0_131, vendor: Oracle Corporation
 Java home: /usr/lib/jvm/java-1.8-openjdk/jre
 Default locale: en_US, platform encoding: UTF-8
 OS name: "linux", version: "3.10.0-514.21.2.el7.x86_64", arch: "amd64", family: "unix"
 [DEBUG] Included /home/jenkins/workspace/roject_Pipelines_rm_develop-XBJPHN7TB7XCQRDEZR56CIZD26DWWTPH7BDYFOQ6SMIYZML37NZQ@tmp/withMaven7dbea4cd/pipeline-maven-spy.jar
 [DEBUG] Populating class realm maven.ext

... snip ...

Then, with more chopped out...

[DEBUG] Reading global settings from /usr/share/maven/conf/settings.xml
 [DEBUG] Reading user settings from ?/.m2/settings.xml
 [DEBUG] Reading global toolchains from /usr/share/maven/conf/toolchains.xml
 [DEBUG] Reading user toolchains from ?/.m2/toolchains.xml
 [DEBUG] Using local repository at /home/jenkins/workspace/roject_Pipelines_rm_develop-XBJPHN7TB7XCQRDEZR56CIZD26DWWTPH7BDYFOQ6SMIYZML37NZQ/?/.m2/repository
 [DEBUG] Using manager EnhancedLocalRepositoryManager with priority 10.0 for /home/jenkins/workspace/roject_Pipelines_rm_develop-XBJPHN7TB7XCQRDEZR56CIZD26DWWTPH7BDYFOQ6SMIYZML37NZQ/?/.m2/repository
 [INFO] Scanning for projects...

@cleclerc The 3.0.3-beta-1 version works for me using the MVN_CMD-workaround. Thanks!

cleclerc, can you clarify what you mean with "specify the maven settings files using the Config File Provider Plugin" on the workaround mentioned in issue description?

I'm asking this because we are using the Config File Provider Plugin and have the same issue. We see the settings file locate on a workspace @tmp folder but withMaven is not using it.

My workaround for this was running a find command on workspace and create a variable with that path.
After that, I run the mvn command with --settings option, with the variable.

Example:

withMaven(mavenSettingsConfig: "${mvnsettingsid}") {
    def settingspath = sh (returnStdout: true, script: "find ${env.WORKSPACE}@tmp -iname settings.xml").trim()
    ...
    sh "mvn clean package --settings ${settingspath}"
}

Comment modified on 2017/11/09 to use 3.0.3-beta-2 and '$MVN_CMD_DIR' instead of '$MVN_CMD'

ncosta can you use pipeline-maven-plugin:3.0.3-beta-2 and prepend '$MVN_CMD_DIR' to the '$PATH' environment variable.

https://github.com/jenkinsci/pipeline-maven-plugin/releases/tag/pipeline-maven-3.0.3-beta-2

node("linux-agent-with-docker") {
    docker.image('maven').inside {
        withMaven(mavenSettingsConfig: 'my-maven-settings') {

            git "https://github.com/cyrille-leclerc/my-jar.git"

            // prepend the $MVN_CMD_DIR variable to the $PATH environment variable 
            // in each "sh" step that invokes mvn 
            sh 'export PATH=$MVN_CMD_DIR:$PATH && mvn help:effective-settings'
        }
    }
}

comment modified on 2017/11/09 to use 3.0.3-beta-2 and 'MVN_CMD_DIR' instead of 'MVN_CMD'

node("vagrant") {
    docker.image('maven:3.5.2-jdk-8').inside {
        withMaven() {
            git "https://github.com/takari/maven-wrapper.git"
            // prepend the $MVN_CMD_DIR variable to the $PATH environment variable 
            // in each "sh" step that invokes mvn
            // for troubleshooting, we display `which mvn` 
            sh 'export PATH=$MVN_CMD_DIR:$PATH && which mvn && env && mvn help:effective-settings'
        }
    }
}

Started by user anonymous
[Pipeline] node
Running on vagrant in /home/ubuntu/jenkins-oss/workspace/JENKINS-47823-docker-pipeline
[Pipeline] {
[Pipeline] sh
[JENKINS-47823-docker-pipeline] Running shell script
+ docker inspect -f . maven:3.5.2-jdk-8
.
[Pipeline] withDockerContainer
vagrant does not seem to be running inside a container
$ docker run -t -d -u ...
[Pipeline] {
[Pipeline] withMaven
[withMaven] Options: []
[withMaven] Available options: 
[withMaven] WARNING: "withMaven(){...}" step running within "docker.image('image').inside {...}". Since the Docker Pipeline Plugin version 1.14, you MUST prepend the 'MVN_CMD_DIR' environment variable to the 'PATH' environment variable in every 'sh' step that invokes 'mvn'. See Pipeline Maven Plugin FAQ.
[withMaven] Sample:
[withMaven]    sh "export PATH=$MVN_CMD_DIR:$PATH && mvn ..."
[withMaven] use JDK installation provided by the build agent
[withMaven] use Maven global settings provided by the Jenkins global configuration 'my-global-maven-config' 
[withMaven] use Maven global settings.xml 'my-global-maven-config' with Maven servers credentials provided by Jenkins (replaceAll: false): [mavenServerId: 'ze-global-creds-injected-by-jenkins-creds', jenkinsCredentials: 'creds-for-maven-global-settings', username: 'zze-username', type: 'UsernamePasswordCredentialsImpl'], [mavenServerId: 'nexus.beescloud.com', jenkinsCredentials: 'nexus', username: 'deployment', type: 'UsernamePasswordCredentialsImpl']
$ docker exec 7cc0d12e3f3dd81766ee30b571dbfe5cebd83115b92170c3040c15807eb692d7 ...
[withMaven] use Maven installation provided by the build agent with the environment variable MAVEN_HOME=/usr/share/maven
[Pipeline] {
[Pipeline] git
...
 > git rev-list 2ed51d1bb9660cc33ffaeebc40d870500c56ab56 # timeout=10
[Pipeline] sh
[JENKINS-47823-docker-pipeline] Running shell script
...
+ which mvn
/home/ubuntu/jenkins-oss/workspace/JENKINS-47823-docker-pipeline@tmp/withMavenef79825d/mvn
+ env
...
PATH=/home/ubuntu/jenkins-oss/workspace/JENKINS-47823-docker-pipeline@tmp/withMavenef79825d:/usr/local/sbin:...
...
MAVEN_CONFIG=--batch-mode --show-version --global-settings /home/ubuntu/jenkins-oss/workspace/JENKINS-47823-docker-pipeline@tmp/withMavenef79825d/globalSettings.xml 
+ mvn help:effective-settings
----- withMaven Wrapper script -----
Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/home/ubuntu/jenkins-oss/workspace/JENKINS-47823-docker-pipeline@tmp/withMavenef79825d/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/home/ubuntu/jenkins-oss/workspace/JENKINS-47823-docker-pipeline@tmp/withMavenef79825d" 
Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z)
Maven home: /usr/share/maven
Java version: 1.8.0_151, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "4.4.0-98-generic", arch: "amd64", family: "unix"
[jenkins-maven-event-spy] INFO generate /home/ubuntu/jenkins-oss/workspace/JENKINS-47823-docker-pipeline@tmp/withMavenef79825d/maven-spy-20171107-081802-6568024909890373450083.log.tmp ...
...
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

Code changed in jenkins
User: Cyrille Le Clerc
Path:
jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java
http://jenkins-ci.org/commit/pipeline-maven-plugin/556c80030ed42bb30caef85fd199797c385692e0
Log:
JENKINS-47823 expose the MVN_CMD_DIR environment variable to prepend it to the PATH

ncosta, jan_z,joshtrow, dantran, timdowney, stour, testuser7 I made a mistake in the shell command to augment the PATH with 3.0.3-beta-1 (prepending a file path instead of prepending the parent dir path of the file). Can you please test 3.0.3-beta-1 and use "MVN_CMD_DIR" instead of "MVN_CMD"

https://github.com/jenkinsci/pipeline-maven-plugin/releases/tag/pipeline-maven-3.0.3-beta-2
https://repo.jenkins-ci.org/releases/org/jenkins-ci/plugins/pipeline-maven/3.0.3-beta-2/pipeline-maven-3.0.3-beta-2.hpi

node("vagrant") {
    docker.image('maven:3.5.2-jdk-8').inside {
        withMaven() {
            git "https://github.com/takari/maven-wrapper.git"
            // prepend the $MVN_CMD_DIR variable to the $PATH environment variable 
            // in each "sh" step that invokes mvn
            // for troubleshooting, we display `which mvn` 
            sh 'export PATH=$MVN_CMD_DIR:$PATH && which mvn && env && mvn help:effective-settings'
        }
    }
}

confirm, i can see my settings with beta2 and using $MVN_CMD_DIR/mvn

As I understand this then, every script that wants to use this has to be updated? We have an entire library of scripts from many teams that all were using the old method (withDockerContainer(){ withMaven() { sh 'mvn clean install' } }), and we REALLY don't want to have to update each one - but it looks like that's not an option here, since each 'sh' would need the PATH updated individually.

I appreciate the work you are doing to give a function workaround certainly, but I fully expect we will stay on 1.13 of the docker-pipeline plugin for now because of the amount of changes required for this to work as it is in beta-2

Hi cleclerc

I'll take a look at the details of the some of the questions later when i'm free for longer, but the short answer set I can give is this:

1. mvnw looks to be an easier way to make the change, if we had to across the board certainly
2. Unless someone has changed it underneath me, we are using Alpine both for master and slave machines generally
3. I'll test beta-2 later on today
4. Direct calls to withMaven: 171 files found when I search our entire repository, and that's just the public ones I know about, there could be more

Thanks for explaining the root issue better though - it has never really been clear to me why the issue that 1.14 'fixed' was really an issue, and that does help to clarify the situation (Debian vs RHEL paths) somewhat. I'll admit to not being an expert in the differences between them so I don't still fully comprehend the full details of the issue, but I might be able to pick up the conversations in other issues better now.

cleclerc – I've tested with beta 2 and can confirm that it works.  Following on joshtrow comments:

• Updating our pipelines would be a real headache.  I have in the neighborhood of 100 to deal with when you look across repos and branches.
• This headache is worse since I'd need to change pipelines on Production and Hotfix branches which is enough to really scare management typically.  In other words, we can't start a change like this on develop and roll it through.  There's no real way around this since Jenkins is building across branches.
• I'm using the "official" Maven image just about exclusively.  Typically `maven:3-jdk8-alpine`.
• Takari Wrapper may be something in the future, but we'd still have to update all of our pipelines to use `mvnw` instead of `mvn`, right?  If so, that is certainly more aesthetically pleasing, but doesn't really change the work.
• Could MAVEN_CONFIG be set inside the pipeline?  It looks like that may help.  This is from the mvn shell script (3.5.2.1).  I'm not sure how far back it goes.

MAVEN_CMD_LINE_ARGS="$MAVEN_CONFIG $@"
export MAVEN_CMD_LINE_ARGS

But if you could set MAVEN_CONFIG and pass in the --settings or other options there may be a solution.  A quick Google doesn't turn up much on MAVEN_CONFIG so it may not be reliable across versions though...

• Perhaps failing all of this, Docker Pipeline could keep working as in 1.13 but provide an arg in 1.14 to prevent the PATH from being overwritten?  At least in this case compatibility would be preserved and those with problems would have a flag to throw to not overwrite the PATH.

Tim

timdowney we start exposing MAVEN_CONFIG with pipeline-maven 3.0.3-beta but it does not get picked by the "mvn" 3.5.2 executable as it gets picked by Takari's "mvnw". My understanding is that "MAVEN_CMD_LINE_ARGS" is not used anywhere in Maven.

https://github.com/apache/maven/blob/maven-3.5.2/apache-maven/src/bin/mvn#L188

Hmm...I should have looked further down.  You're right.  It does not appear to be used.

cleclerc, using the example you mentioned above works ok but I noticed some other issues. Not sure if this issues are only happening because this is just a beta:

1. When invoking mvn command, uname is not found

   ----- withMaven Wrapper script -----
   /opt/apache-maven-3.1.1/bin/mvn: 1: /opt/apache-maven-3.1.1/bin/mvn: uname: not found
   ...

1. We use maven plugins that triggers cmake, libboost, etc, and this commands/libraries are only found if I add its paths to the export command.

   ...
   sh "export PATH=$MVN_CMD_DIR:$PATH:opt/bin && mvn clean test"
   ...

   All those commands are already set in PATH but for some reason are not picked by the export command.
   Below more information about paths set inside a ubuntu 12.04 container. Build server on CentOS7.

   ...
   sh "echo $PATH"
   sh "which mvn"
   sh "which cmake"
   sh "echo $MVN_CMD_DIR"
   sh "env | grep PATH"
   sh "uname -r"
   sh "export PATH=$MVN_CMD_DIR:$PATH && which mvn && mvn -v"
   sh "echo $PATH"
   sh "which mvn"
   sh "which cmake"
   sh "env | grep PATH"
   ...
   

   Console output:

   Running shell script
   + echo /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin
   /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin
   sh
   Running shell script
   + which mvn
   /opt/apache-maven-3.1.1/bin/mvn
   sh
   Running shell script
   + which cmake
   /opt/bin/cmake
   sh
   Running shell script
   + echo /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4
   /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4
   sh
   Running shell script
   + env
   + grep PATH
   LIBRARY_PATH=/opt/qt/lib:/opt/lib:
   LD_LIBRARY_PATH=/opt/lib:
   PATH=/opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
   CLASSPATH=
   sh
   Running shell script
   + uname -r
   4.4.83-1.el7.elrepo.x86_64
   sh
   Running shell script
   + export PATH=/var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin
   + which mvn
   /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4/mvn
   + mvn -v
   ----- withMaven Wrapper script -----
   /opt/apache-maven-3.1.1/bin/mvn: 1: /opt/apache-maven-3.1.1/bin/mvn: uname: not found
   Warning: JAVA_HOME environment variable is not set.
   Picked up JAVA_TOOL_OPTIONS: -Dmaven.ext.class.path="/var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4/pipeline-maven-spy.jar" -Dorg.jenkinsci.plugins.pipeline.maven.reportsFolder="/var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4"
   Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17 16:22:22+0100)
   Maven home: /opt/apache-maven-3.1.1
   Java version: 1.8.0_111, vendor: Oracle Corporation
   Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
   Default locale: en_US, platform encoding: ANSI_X3.4-1968
   OS name: "linux", version: "4.4.83-1.el7.elrepo.x86_64", arch: "amd64", family: "unix"
   sh
   Running shell script
   + echo /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin
   /var/lib/jenkins/workspace/project/repo/branch/debug@tmp/withMaven476b8ef4:/usr/local/bin:/usr/bin
   sh
   Running shell script
   + which mvn
   /opt/apache-maven-3.1.1/bin/mvn
   sh
   Running shell script
   + which cmake
   /opt/bin/cmake
   sh
   Running shell script
   + env
   + grep PATH
   LIBRARY_PATH=/opt/qt/lib:/opt/lib:
   LD_LIBRARY_PATH=/opt/lib:
   PATH=/opt/bin:/opt/sonar-runner-2.4/bin:/opt/apache-maven-3.1.1/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
   CLASSPATH=
   

Code changed in jenkins
User: Cyrille Le Clerc
Path:
jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java
http://jenkins-ci.org/commit/pipeline-maven-plugin/52141ec07401eb73b94963e414f4075a2903e1b1
Log:
JENKINS-47823 better help in build logs

ncosta can you please open a different bug if the problem is not fixed. It may be a problem with the way docker-pipeline handles the $PATH or a problem with the way the $PATH is defined in your Docker image (please keep in mind that docker-pipeline no longer honors the entrypoint).

Closing this defect as pipeline-maven-plugin 3.0.3 offers a solution to work with docker-pipeline 1.14: https://wiki.jenkins.io/display/JENKINS/Pipeline+Maven+Plugin#PipelineMavenPlugin-withMavenAndDockerPipeline

cleclerc, just created JENKINS-48284 with some details.

joshtrow I know it's been a year, but I came up with a workaround for not having to update every mvn call here: https://issues.jenkins-ci.org/browse/JENKINS-40484?focusedCommentId=352635&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-352635