Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44089

[cygwin] Request that withMaven pipeline step produce both mvn and mvn.cmd wrappers on Windows

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • pipeline-maven-plugin

      Request that withMaven pipeline step produce both mvn and mvn.cmd wrappers on Windows.

      On Windows build agents we would like the withMaven build step to produce wrappers that work for both bash.exe and cmd.exe.  Currently the step adds a directory that adds a script "mvn" on Linux and adds a script "mvn.cmd" on Windows.  On Windows it is possible to configure the build server to make available the Git for Windows tool suite which includes bash.exe.  We write all of our build scripts in bash syntax so that we can be cross platform on build agents when doing builds that are platform agnostic, like running Java based Maven builds.

      We have a work around (see code below) that allows us to do the the build by calling "$COMSPEC" to execute the mvn.cmd.  It would be simpler and cleaner if the both a bash.exe compatible "mvn" wrapper and cmd.exe compatible "mvn.cmd" wrapper were produced on the Windows Platform.  Note the Git for Windows bash.exe accepts both Unix style paths (/c/Program\ Files, or '/c/Program Files') and Windows style paths (C:\\Program\ Files, or 'C:\Program Files') as long as they are properly escaped or quoted.

      When running a withMaven build from a sh() build step on Windows this is our current work around:

      withMaven(
         jdk: config['jdk'], 
         maven: config['maven'], 
         mavenLocalRepo: config['mavenLocalRepo'], 
         mavenSettingsConfig: config['mavenSettingsConfig']) {
      
      sh('''
      
      if [[ -e "pom.xml" ]]; then
           # Run Jenkins maven wrapper script using mvn in Unix bash 
           # or mvn.cmd in cmd.exe from Git for Windows bash
      
           if [[ -z "$COMSPEC" ]]; then
               cmd=(mvn)
           else
               cmd=("$COMSPEC" //c mvn.cmd)
           fi
           args=(--batch-mode -U -V --settings "$MVN_SETTINGS" clean install -P "$BUILD_PROFILE" 
               "-DxlDeployServerAddress=$XLDEPLOY_SERVER_ADDRESS" "-DxlDeployPort=$XLDEPLOY_PORT" 
               "-DxlDeploySecured=$XLDEPLOY_SECURED" "-DbuildUrl=$BUILD_URL")
           eval $(printf " %q" "$\{cmd[@]}" "$\{args[@]}")
       fi
      ''')
      
      }
      

          [JENKINS-44089] [cygwin] Request that withMaven pipeline step produce both mvn and mvn.cmd wrappers on Windows

          Can you please provide of the mvn "sh" script that would be needed with cygwin?

          Cyrille Le Clerc added a comment - Can you please provide of the mvn "sh" script that would be needed with cygwin?

          In the standard Maven distribution there is a "mvn" shell script and a "mvn.cmd" windows batch file.

          $ ls /c/Bretools/bretools-1.0.6/Maven/apache-maven-3.3.9/bin
          m2.conf mvn mvn.cmd mvnDebug mvnDebug.cmd mvnyjp

          When running Git bash on windows EITHER of these scripts will work on Windows to run Maven.  The issue i'm facing is with the wrappers that are created by Jenkins when using the withMaven() {} pipeline step.

          On when running inside a withMaven() { } configuration block the Jenkins updates the path to a Jenkins provided wrapper.  On non-windows it creates only a wrapper shell script called "mvn" that forwards to the shell script "mvn" inside the Maven tool directory.  On Windows it only creates is a "mvn.cmd" batch script wrapper that points to the Maven tool directory "mvn.cmd" batch script.  However in Git Bash that wrapper "mvn.cmd" is not a valid shell script so doesn't show up on the path as an executable script.  So as a work around I detect if we are on windows (COMSPEC environment variable is defined) and run mvn.cmd from a child batch script.

          The preferred solution would be that on Windows withMaven() {} create wrapper scripts for BOTH "mvn.cmd" which runs inside batch (CMD.EXE), and the "mvn" which runs in shells (e.g. BASH), the logic to do this exists but is currently only run on the non-windows hosts.

           

          Frederick Staats added a comment - In the standard Maven distribution there is a "mvn" shell script and a "mvn.cmd" windows batch file. $ ls /c/Bretools/bretools-1.0.6/Maven/apache-maven-3.3.9/bin m2.conf mvn mvn.cmd mvnDebug mvnDebug.cmd mvnyjp When running Git bash on windows EITHER of these scripts will work on Windows to run Maven.  The issue i'm facing is with the wrappers that are created by Jenkins when using the withMaven() {} pipeline step. On when running inside a withMaven() { } configuration block the Jenkins updates the path to a Jenkins provided wrapper.  On non-windows it creates only a wrapper shell script called "mvn" that forwards to the shell script "mvn" inside the Maven tool directory.  On Windows it only creates is a "mvn.cmd" batch script wrapper that points to the Maven tool directory "mvn.cmd" batch script.  However in Git Bash that wrapper "mvn.cmd" is not a valid shell script so doesn't show up on the path as an executable script.  So as a work around I detect if we are on windows (COMSPEC environment variable is defined) and run mvn.cmd from a child batch script. The preferred solution would be that on Windows withMaven() {} create wrapper scripts for BOTH "mvn.cmd" which runs inside batch (CMD.EXE), and the "mvn" which runs in shells (e.g. BASH), the logic to do this exists but is currently only run on the non-windows hosts.  

          Another clarification.  When on Windows inside bash you need to run "mvn" to run the shell script and "mvn.cmd" to run the batch shell script when running it from bash.  I would like to be able to always "mvn" inside a bash script regardless if I'm running on Windows or Linux.  What I'm trying to do is remove the need for end user platform specific code for a Maven build.  My snippet above used the boiler plate "$COMSPEC" //c mvn.cmd" but a simple "mvn.cmd" would have been sufficient.

          Please let me know if I'm not being clear.

          Frederick Staats added a comment - Another clarification.  When on Windows inside bash you need to run "mvn" to run the shell script and "mvn.cmd" to run the batch shell script when running it from bash.  I would like to be able to always "mvn" inside a bash script regardless if I'm running on Windows or Linux.  What I'm trying to do is remove the need for end user platform specific code for a Maven build.  My snippet above used the boiler plate "$COMSPEC" //c mvn.cmd" but a simple "mvn.cmd" would have been sufficient. Please let me know if I'm not being clear.

          flstaats I am not a Windows user, could you help with this?

          Cyrille Le Clerc added a comment - flstaats I am not a Windows user, could you help with this?

          @cleclerc sorry for the late response.  I've had some holiday (2017 American Eclipse!) and  I filed this from my personal rather than work account so it got lost it in the email.  Finally got caught up today.

          Yes I can help with this.  What kind of assistance would you like?

          Frederick Staats added a comment - @cleclerc sorry for the late response.  I've had some holiday (2017 American Eclipse!) and  I filed this from my personal rather than work account so it got lost it in the email.  Finally got caught up today. Yes I can help with this.  What kind of assistance would you like?

          I'm also in favour of generating all the mvn wrapper scripts and not only the one supposed to be applicable to the slave platform. This will at least ease the work of those running their scripts both on Unix and Cygwin platforms.

          As a reminder, Cygwin platforms appear to be Windows under Jenkins and not Unix. Currently only the mvn.cmd wrapper is generated whereas the mvn (unix form) wrapper can be used under Cygwin... And this is exactly the one most want to use when running with Cygwin... This makes pipelines portable across platform - reasons why they use Cygwin.

          Bertrand Renuart added a comment - I'm also in favour of generating all the mvn wrapper scripts and not only the one supposed to be applicable to the slave platform. This will at least ease the work of those running their scripts both on Unix and Cygwin platforms. As a reminder, Cygwin platforms appear to be Windows under Jenkins and not Unix. Currently only the mvn.cmd wrapper is generated whereas the mvn (unix form) wrapper can be used under Cygwin... And this is exactly the one most want to use when running with Cygwin... This makes pipelines portable across platform - reasons why they use Cygwin.

          Generating the mvn shell script for cygwin on Windows is likely to be tricky.

          I suspect that the path to the underlying "mvn" script provided by the maven distribution will be challenging...

          https://github.com/jenkinsci/pipeline-maven-plugin/blob/pipeline-maven-3.0.2/jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java#L350

          Cyrille Le Clerc added a comment - Generating the mvn shell script for cygwin on Windows is likely to be tricky. I suspect that the path to the underlying "mvn" script provided by the maven distribution will be challenging... https://github.com/jenkinsci/pipeline-maven-plugin/blob/pipeline-maven-3.0.2/jenkins-plugin/src/main/java/org/jenkinsci/plugins/pipeline/maven/WithMavenStepExecution.java#L350

          Bertrand Renuart added a comment - - edited

          Note that much... Here is how we patched WithMavenStepExecution:

          1/ We changed the signature of mavenWrapperContent to accept an additional isUnix parameter to force its value instead of detecting it from the running node. It also normalize the path to the maven executable accordingly:

          private String mavenWrapperContent(FilePath mvnExec, boolean isUnix, String settingsFile, String globalSettingsFile, String mavenLocalRepo) throws AbortException {
          
          // -- CHANGES START
          //        ArgumentListBuilder argList = new ArgumentListBuilder(mvnExec.getRemote());
          //        boolean isUnix = Boolean.TRUE.equals(getComputer().isUnix());
                  
            String mvn = isUnix ? FilenameUtils.separatorsToUnix(mvnExec.getRemote()):FilenameUtils.separatorsToWindows(mvnExec.getRemote());
            ArgumentListBuilder argList = new ArgumentListBuilder(mvn);
          // -- CHANGES END --

          2/ We patched the code of setupMaven() as follows:

          // -- CHANGES START --        
          // See https://issues.jenkins-ci.org/browse/JENKINS-44089
          
          // String content = mavenWrapperContent(mvnExec, setupSettingFile(credentials), setupGlobalSettingFile(credentials), setupMavenLocalRepo());
          // createWrapperScript(tempBinDir, mvnExec.getName(), content);
                  
          String settingFileLocalPath = setupSettingFile(credentials);
          String globalSettingFileLocalPath = setupGlobalSettingFile(credentials);
          String localRepoLocalPath = setupMavenLocalRepo();
          {
              FilePath mvn = mvnExec.sibling("mvn");
              String content = mavenWrapperContent(mvn, true, settingFileLocalPath, globalSettingFileLocalPath, localRepoLocalPath);
              createWrapperScript(tempBinDir, mvn.getName(), content);
          }
          {
              FilePath mvn = mvnExec.sibling("mvn.cmd");
              String content = mavenWrapperContent(mvn, false, settingFileLocalPath, globalSettingFileLocalPath, localRepoLocalPath);
              createWrapperScript(tempBinDir, mvn.getName(), content);
          }
          {
              FilePath mvn = mvnExec.sibling("mvn.bat");
              String content = mavenWrapperContent(mvn, true, settingFileLocalPath, globalSettingFileLocalPath, localRepoLocalPath);
              createWrapperScript(tempBinDir, mvn.getName(), content);
          }
          // -- CHANGES END --

          And it all works nicely... at least in our case.

          One thing to note tho is the only thing that matters is the location where maven is installed. This could be deduced from the installation (if done by the the withMaven step, environment variable or any other detection mechanism). If the maven installation doesn't provide a mvn.bat, a "file not found" will be thrown by the corresponding wrapper. If the wrapper did not exist, the caller would have a "file not found" too...

          Bertrand Renuart added a comment - - edited Note that much... Here is how we patched WithMavenStepExecution : 1/ We changed the signature of mavenWrapperContent to accept an additional isUnix parameter to force its value instead of detecting it from the running node. It also normalize the path to the maven executable accordingly: private String mavenWrapperContent(FilePath mvnExec, boolean isUnix, String settingsFile, String globalSettingsFile, String mavenLocalRepo) throws AbortException { // -- CHANGES START //        ArgumentListBuilder argList = new ArgumentListBuilder(mvnExec.getRemote()); //        boolean isUnix = Boolean .TRUE.equals(getComputer().isUnix());            String mvn = isUnix ? FilenameUtils.separatorsToUnix(mvnExec.getRemote()):FilenameUtils.separatorsToWindows(mvnExec.getRemote());   ArgumentListBuilder argList = new ArgumentListBuilder(mvn); // -- CHANGES END -- 2/ We patched the code of setupMaven()  as follows: // -- CHANGES START --         // See https://issues.jenkins-ci.org/browse/JENKINS-44089 //  String content = mavenWrapperContent(mvnExec, setupSettingFile(credentials), setupGlobalSettingFile(credentials), setupMavenLocalRepo()); // createWrapperScript(tempBinDir, mvnExec.getName(), content);          String settingFileLocalPath = setupSettingFile(credentials); String globalSettingFileLocalPath = setupGlobalSettingFile(credentials); String localRepoLocalPath = setupMavenLocalRepo(); {     FilePath mvn = mvnExec.sibling( "mvn" );     String content = mavenWrapperContent(mvn, true , settingFileLocalPath, globalSettingFileLocalPath, localRepoLocalPath);     createWrapperScript(tempBinDir, mvn.getName(), content); } {     FilePath mvn = mvnExec.sibling( "mvn.cmd" );     String content = mavenWrapperContent(mvn, false , settingFileLocalPath, globalSettingFileLocalPath, localRepoLocalPath);     createWrapperScript(tempBinDir, mvn.getName(), content); } {     FilePath mvn = mvnExec.sibling( "mvn.bat" );     String content = mavenWrapperContent(mvn, true , settingFileLocalPath, globalSettingFileLocalPath, localRepoLocalPath);     createWrapperScript(tempBinDir, mvn.getName(), content); } // -- CHANGES END -- And it all works nicely... at least in our case. One thing to note tho is the only thing that matters is the location where maven is installed. This could be deduced from the installation (if done by the the withMaven step, environment variable or any other detection mechanism). If the maven installation doesn't provide a mvn.bat , a "file not found" will be thrown by the corresponding wrapper. If the wrapper did not exist, the caller would have a "file not found" too...

          brenuart could you please propose a Pull Request?

          Cyrille Le Clerc added a comment - brenuart could you please propose a Pull Request?

          Frederick Staats added a comment - - edited

          We gave up using withMaven {} for setting up the build time environment for Maven some time ago because if the issues with using Bash for Windows and getting the environment setup correctly in a multi executor / multi node / multi architecture build environment.

          To summarize what we are doing instead so people know what is possible:

          We preinstall Git for Windows on all the Windows build agents in a w ell known area and set the PATH environment variable on the Windows nodes in the Node control panel to (with path separators ; converted to newlines to make more readable):

          /usr/bin
          /bin
          /mingw64/bin
          C:\WINDOWS\system32
          C:\WINDOWS
          C:\WINDOWS\System32\Wbem
          C:\WINDOWS\System32\WindowsPowerShell\v1.0
          C:\Program Files\Git\git-2.14\usr\bin
          C:\Program Files\Git\git-2.14\bin
          C:\Program Files\Git\git-2.14\mingw64\bin

          This gives us a base path so that when run in Bash for Windows the UNIX utilities at are at the start of the PATH, and when run in CMD.exe or PowerShell.exe has the Bash for Windows at the end of the PATH.  Note: Bash for Windows will use the Unix forward slash /usr/bin, /bin, /mingw64/bin in the users PATH even though it is a Windows machine (and strip out the later duplicate references in the PATH).  But CMD.exe and PowerShell.exe will just ignore the forward slash path entries.  This allows us to have one preset PATH but get Unix like or Windows like behavior depending on which shell we are using.  This is important since some commands (for example FIND) are very different and are used by a surprising number of third part scripts.

          In our build scripts we add to the base PATH using the env.PATH syntax directly at the start of the build script right after node is called, we add tools using the "tools" build script step to give us the path to the JDK, Maven, Gradle, etc.. WE DO NOT USE withEnv to set the PATH.  The reason for this is that withEnv and PATH does not play nice when using the node step to cross nodes of different architectures.   We have many build jobs that start on Linux but eventually do a node call over to Windows to run environment specific functional tests.  If you use withEnv and the PATH+TOOL syntax there is no easy way to reset your path to be correct for the architecture as Jenkins keeps prepending the wrong architectures tools paths to the start of PATH.

          Our more sophisticated scripts use a library step we wrote that not only sets up the path but divides the workspace into a "home" and a "work" area and sets up a home directory specific for the context of the jobs.  In the case of multibranch pipelines we lock the parent workspace we create and share the workspace between all the branches allowing Maven/Gradle cache reuse for all the branches for a specific Git repository but not between them.  This code node aware so that we can support a single job using multiple executors on the same or different nodes and it only sets up the shared home directory once on each unique node.  Conflicts between concurrent work in the "work" area is the responsibility of the job author.

           

           

           

          Frederick Staats added a comment - - edited We gave up using withMaven {} for setting up the build time environment for Maven some time ago because if the issues with using Bash for Windows and getting the environment setup correctly in a multi executor / multi node / multi architecture build environment. To summarize what we are doing instead so people know what is possible: We preinstall Git for Windows on all the Windows build agents in a w ell known area and set the PATH environment variable on the Windows nodes in the Node control panel to (with path separators ; converted to newlines to make more readable): /usr/bin /bin /mingw64/bin C:\WINDOWS\system32 C:\WINDOWS C:\WINDOWS\System32\Wbem C:\WINDOWS\System32\WindowsPowerShell\v1.0 C:\Program Files\Git\git-2.14\usr\bin C:\Program Files\Git\git-2.14\bin C:\Program Files\Git\git-2.14\mingw64\bin This gives us a base path so that when run in Bash for Windows the UNIX utilities at are at the start of the PATH, and when run in CMD.exe or PowerShell.exe has the Bash for Windows at the end of the PATH.  Note: Bash for Windows will use the Unix forward slash /usr/bin, /bin, /mingw64/bin in the users PATH even though it is a Windows machine (and strip out the later duplicate references in the PATH).  But CMD.exe and PowerShell.exe will just ignore the forward slash path entries.  This allows us to have one preset PATH but get Unix like or Windows like behavior depending on which shell we are using.  This is important since some commands (for example FIND) are very different and are used by a surprising number of third part scripts. In our build scripts we add to the base PATH using the env.PATH syntax directly at the start of the build script right after node is called, we add tools using the "tools" build script step to give us the path to the JDK, Maven, Gradle, etc.. WE DO NOT USE withEnv to set the PATH.  The reason for this is that withEnv and PATH does not play nice when using the node step to cross nodes of different architectures.   We have many build jobs that start on Linux but eventually do a node call over to Windows to run environment specific functional tests.  If you use withEnv and the PATH+TOOL syntax there is no easy way to reset your path to be correct for the architecture as Jenkins keeps prepending the wrong architectures tools paths to the start of PATH. Our more sophisticated scripts use a library step we wrote that not only sets up the path but divides the workspace into a "home" and a "work" area and sets up a home directory specific for the context of the jobs.  In the case of multibranch pipelines we lock the parent workspace we create and share the workspace between all the branches allowing Maven/Gradle cache reuse for all the branches for a specific Git repository but not between them.  This code node aware so that we can support a single job using multiple executors on the same or different nodes and it only sets up the shared home directory once on each unique node.  Conflicts between concurrent work in the "work" area is the responsibility of the job author.      

            Unassigned Unassigned
            flstaats Frederick Staats
            Votes:
            3 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: