Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-44334

Pipeline: Job v2.11 breaks build with bat() command

      Hi there!

      We just upgraded our Pipeline:Job plugin from 2.10 to version 2.11. We're using the following pipeline script:

      node {
      
      stage('Prepare/Checkout') {
          dir('repo-master') {
              git credentialsId: 'secret_access', poll: false , url: 'ssh://git@company.url.com:7999/xxx/repo.git'
          }
          bat('del -f d:\\xxx\\yy.zip >nul 2>&1')
      }
      }
      

      Unfortunately since 2.11, we receive the following issue during build at the bat() call:

      ...
       > git.exe checkout -b master 3579a579a89ccee5892fbb94951f868020a9d4f7
       > git.exe rev-list 3579a579a89ccee5892fbb94951f868020a9d4f7 # timeout=10
      [Pipeline] }
      [Pipeline] // dir
      [Pipeline] bat
      [workspace] Running batch script
      'cmd' is not recognized as an internal or external command,
      operable program or batch file.
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      ERROR: script returned exit code 9009
      Finished: FAILURE
      

      It looks like the cmd.exe is not found any more in the PATH. It is strange as the cmd is not moved anywhere, and also downgrading to version 2.10 "resolves" the problem. We did not yet try 2.12, as we still use Jenkins 2.46.2

      We did not test any other "external" tool but it seems git is found in the first step.

      Plugin list is attached.

      Thanks in advance!

          [JENKINS-44334] Pipeline: Job v2.11 breaks build with bat() command

          Andrew Bayer added a comment -

          Is this still happening?

          Andrew Bayer added a comment - Is this still happening?

          Jesse Glick added a comment -

          Given that this appears specifically after a workflow-job update, I doubt it is related directly to JENKINS-41339. More likely it is a regression from JENKINS-43396. Both involve global node properties, though; check whether you have some of these defined.

          Jesse Glick added a comment - Given that this appears specifically after a workflow-job update, I doubt it is related directly to  JENKINS-41339 . More likely it is a regression from  JENKINS-43396 . Both involve global node properties, though; check whether you have some of these defined.

          I confirm jglick, abayer that you can easily reproduce the issue by defining a global environment variable PATH at the master level and then launch a pipeline on a windows agent. Same thing with a linux agent if you define a windows PATH at the master level 

          Arnaud Héritier added a comment - I confirm jglick , abayer that you can easily reproduce the issue by defining a global environment variable PATH at the master level and then launch a pipeline on a windows agent. Same thing with a linux agent if you define a windows PATH at the master level 

          Andrew Bayer added a comment -

          Global env vars and node vars are such a pain with Pipeline. Grr.

          Andrew Bayer added a comment - Global env vars and node vars are such a pain with Pipeline. Grr.

          Andrew Bayer added a comment -

          So here's my question: what should be happening here? I feel like the global env var should be overriding, unless you've set a per-node env var.

          Andrew Bayer added a comment - So here's my question: what should be happening here? I feel like the global env var should be overriding, unless you've set a per-node env var.

          Andrew Bayer added a comment -

          This also gets more interesting in that I can't quite reproduce it in a unit test in workflow-durable-task-step (after bumping its workflow-job dependency). If I set PATH in globalNodeProperties, everything still works. If I set PATH in an agent's nodeProperties, I get

             7.222 [demo #1] [Pipeline] sh
             7.486 [demo #1] [demo] Running shell script
             7.486 [demo #1] sh: touch: command not found
             7.486 [demo #1] sh: sleep: command not found
             7.486 [demo #1] sh: touch: command not found
             7.486 [demo #1] sh: sleep: command not found
             7.486 [demo #1] sh: touch: command not found
             7.486 [demo #1] sh: sleep: command not found
             7.487 [demo #1] + echo ONSLAVE=true
             7.487 [demo #1] ONSLAVE=true
          

          So sh still runs.

          I'm guessing this is something that's a giant pain to test for in a unit test.

          Andrew Bayer added a comment - This also gets more interesting in that I can't quite reproduce it in a unit test in workflow-durable-task-step (after bumping its workflow-job dependency). If I set PATH in globalNodeProperties , everything still works. If I set PATH in an agent's nodeProperties , I get 7.222 [demo #1] [Pipeline] sh 7.486 [demo #1] [demo] Running shell script 7.486 [demo #1] sh: touch: command not found 7.486 [demo #1] sh: sleep: command not found 7.486 [demo #1] sh: touch: command not found 7.486 [demo #1] sh: sleep: command not found 7.486 [demo #1] sh: touch: command not found 7.486 [demo #1] sh: sleep: command not found 7.487 [demo #1] + echo ONSLAVE= true 7.487 [demo #1] ONSLAVE= true So sh still runs. I'm guessing this is something that's a giant pain to test for in a unit test.

          Jesse Glick added a comment -

          Unlikely to be reproducible using JenkinsRule unless you are very careful. See the SpecialEnvSlave hack in some tests, or you might even need to use docker-fixtures.

          My recommendation: do not change behavior. Anything you touch here will regress somebody’s use case. Rather, make workflow-durable-task-step print a warning to the build log if you are using global node properties, basically saying all bets are off. If you want to define environment variables, use withEnv or the like.

          Jesse Glick added a comment - Unlikely to be reproducible using JenkinsRule unless you are very careful. See the SpecialEnvSlave hack in some tests, or you might even need to use docker-fixtures . My recommendation: do not change behavior. Anything you touch here will regress somebody’s use case. Rather, make workflow-durable-task-step print a warning to the build log if you are using global node properties, basically saying all bets are off. If you want to define environment variables, use withEnv or the like.

          My 2 cents abayer:

          • I agree with jglick, we should change the behavior if possible, but for this we should need to have a test/documentation explaining the existing (or expected) behavior.
          • Environment variables can be set at many levels and I think that the most general one should be overridable by the most local one (but maybe not ).
            • Thus the var defined at the master level could be redefined at the folder level and at the job level.
            • But it could also be defined at the agent config level (in that case it overrides the master value but the folder/job can override it).
            • There is also the operating system / shell level from which we launch the JVM or a command/shell.
          • I'm not talking about the envinject plugin which can reset or redefine them at various levels (let's focus on core behaviors)
          • I'm not talking about all others plugins which are just contributing by adding some entries in the PATH or by adding some others environment variables.
          • We have to take care that PATH (and perhaps few others env var like HOME, USER...) is probably a specific case where the expected behavior could be different than for some others vars because it is has a huge impact at the system level, is often OS dependent and might need a different approach (for PATH maybe you would prefer append than override - or not again )
          • With a real environment (I'm not able to judge about the unit test) I was easily breaking a sh() step with a master/agent on linux by adding a crappy PATH entry (d:) as global env on the master 2.89.x with pipeline job 2.17

          HTH 

          Arnaud Héritier added a comment - My 2 cents abayer : I agree with jglick , we should change the behavior if possible, but for this we should need to have a test/documentation explaining the existing (or expected) behavior. Environment variables can be set at many levels and I think that the most general one should be overridable by the most local one (but maybe not ). Thus the var defined at the master level could be redefined at the folder level and at the job level. But it could also be defined at the agent config level (in that case it overrides the master value but the folder/job can override it). There is also the operating system / shell level from which we launch the JVM or a command/shell. I'm not talking about the envinject plugin which can reset or redefine them at various levels (let's focus on core behaviors) I'm not talking about all others plugins which are just contributing by adding some entries in the PATH or by adding some others environment variables. We have to take care that PATH (and perhaps few others env var like HOME, USER...) is probably a specific case where the expected behavior could be different than for some others vars because it is has a huge impact at the system level, is often OS dependent and might need a different approach (for PATH maybe you would prefer append than override - or not again ) With a real environment (I'm not able to judge about the unit test) I was easily breaking a sh() step with a master/agent on linux by adding a crappy PATH entry (d:) as global env on the master 2.89.x with pipeline job 2.17 HTH 

          Mariano added a comment - - edited

          Hi,
          I am using the Jenkins version 2.121.1 LTS, creating a basic pipeline with stages that consult versions of Java, Ant or Maven, when trying to execute a command from CMD (bat), it returns the error:

          "cmd" is not recognized as an internal or external command, operable program or batch file

          Only the master node exists and the bat commands work in a free style project

          The pipeline is:

          pipeline {
          agent any
          stages {
          stage('CheckOutFromSVN') {
          steps

          { cleanWs() checkout([$class: 'SubversionSCM', additionalCredentials: [], excludedCommitMessages: '', excludedRegions: '', excludedRevprop: '', excludedUsers: '', filterChangelog: false, ignoreDirPropChanges: false, includedRegions: '', locations: [[cancelProcessOnExternalsFail: true, credentialsId: '*XXXXX*', depthOption: 'infinity', ignoreExternalsOption: true, local: '.', remote: '*XXXXX*']], quietOperation: true, workspaceUpdater: [$class: 'UpdateUpdater']]) }

          }
          stage('Invoke Ant Build') {
          steps { * bat 'mvn -version' * }
          }

          stage('Stage 3') {
          steps

          { echo 'Hello world Maven!' }

          }
          }
          }

          Mariano added a comment - - edited Hi, I am using the Jenkins version 2.121.1 LTS, creating a basic pipeline with stages that consult versions of Java, Ant or Maven, when trying to execute a command from CMD (bat), it returns the error: "cmd" is not recognized as an internal or external command, operable program or batch file Only the master node exists and the bat commands work in a free style project The pipeline is: pipeline { agent any stages { stage('CheckOutFromSVN') { steps { cleanWs() checkout([$class: 'SubversionSCM', additionalCredentials: [], excludedCommitMessages: '', excludedRegions: '', excludedRevprop: '', excludedUsers: '', filterChangelog: false, ignoreDirPropChanges: false, includedRegions: '', locations: [[cancelProcessOnExternalsFail: true, credentialsId: '*XXXXX*', depthOption: 'infinity', ignoreExternalsOption: true, local: '.', remote: '*XXXXX*']], quietOperation: true, workspaceUpdater: [$class: 'UpdateUpdater']]) } } stage('Invoke Ant Build') { steps { * bat 'mvn -version' * } } stage('Stage 3') { steps { echo 'Hello world Maven!' } } } }

          Ryan Ashcraft added a comment -

          For those looking to work around this in the meantime - we had exact same situation as above.  Rolling back to 2.10 did work, but then caused a number of other dependency issues across the suite of Pipeline plugins.  The root cause for us was that the global PATH had been set to linux-specific values.  Upon eliminating the global setting, everything works with the latest version (2.25).

          Ryan Ashcraft added a comment - For those looking to work around this in the meantime - we had exact same situation as above.  Rolling back to 2.10 did work, but then caused a number of other dependency issues across the suite of Pipeline plugins.  The root cause for us was that the global PATH had been set to linux-specific values.  Upon eliminating the global setting, everything works with the latest version (2.25).

            Unassigned Unassigned
            bpedro István Bede
            Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: