I have Jenkins slave nodes configured on Windows Server 2012 with a very simple pipeline script:

      node('js-win') {
      	stage 'Checkout'
      	checkout scm
      	stage 'Build'
      	bat 'dir'
      }
      

      At the end of the script, from the Console output, it would just spin and hang after listing all the files.

          [JENKINS-33164] Pipeline bat stuck on Windows Server 2012

          mocsharp added a comment - - edited

          I tested it again with local Windows 2012 R2 and it worked fine. However, when it's configured on ec2 using WinRM, the process would never exit.

          If slave is conneted using JNLP, the process exits fine.

          mocsharp added a comment - - edited I tested it again with local Windows 2012 R2 and it worked fine. However, when it's configured on ec2 using WinRM, the process would never exit. If slave is conneted using JNLP, the process exits fine.

          Carlos Lucas added a comment - - edited

          I am having this problem on a Windows 2012 too, but I have it in the main node of jenkins.

          Downgrading the Durable Task Plugin to 1.7 solves de problem.

          Carlos Lucas added a comment - - edited I am having this problem on a Windows 2012 too, but I have it in the main node of jenkins. Downgrading the Durable Task Plugin to 1.7 solves de problem.

          Samuel Garcia added a comment -

          I'm having the same problem on a Windows 7 Professional SP1. Jenkins 1.652, Durable Task Plugin 1.8.

          My pipeline script that uses the 'bat' command hangs after the command has been successfully executed. I can't even kill the job, not even with the 'forcibly terminate' link that appears on the job's console. I have to stop Jenkins from the 'Services' window of Windows, then remove the job's folder.

          Is there any workaround for this? Since Jenkins won't allow me to downgrade the Durable Task Plugin

          Samuel Garcia added a comment - I'm having the same problem on a Windows 7 Professional SP1. Jenkins 1.652, Durable Task Plugin 1.8. My pipeline script that uses the 'bat' command hangs after the command has been successfully executed. I can't even kill the job, not even with the 'forcibly terminate' link that appears on the job's console. I have to stop Jenkins from the 'Services' window of Windows, then remove the job's folder. Is there any workaround for this? Since Jenkins won't allow me to downgrade the Durable Task Plugin

          Andreas Schroeder added a comment - - edited

          I have the same problem now on a Windows Server 2008 R2 Standard 64 Bit machine with Jenkins 1.654 and Pipeline plugin 1.15.

          I investigated a bit by using this Groovy script:
          node
          {
          for(int i=0; i < 100; ++i)

          { bat('echo "Hello from batch file."' + i.toString()) }

          }
          Jenkins creates two temporary batch files to run each "bat" call: jenkins-main.bat with the "echo ..." command and jenkins-wrap.bat to redirect output and capture the exit code in two text files jenkins-log.txt and jenkins-result.txt.

          At first the Groovy script is executing the bat call a few times correctly and then stops waiting. The console output does nothing but spin this little icon. When this happens, you can find the following state:

          • a "durable-XXXXXXXX" folder (x are hex numbers) in a temporary workspace containing the two batch files jenkins-main.bat and jenkins-wrap.bat.
          • The task manager does not show any of the batch files executing.
          • The two text files jenkins-log.txt and jenkins-result.txt are not there

          Now I have found two ways to make Jenkins continue execting the Groovy script:

          • run the jenkins-wrap.bat manually
          • create the jenkins-log.txt file (can be empty) and the jenkins-result.txt file with a integer number as content, e.g. 0.

          I guess there are two ways this could happen (currently I do not know how to verify this): Either the jenkins-wrap.bat terminates without creating the text files. Or there is some race condition in the Jenkins code deleting the text files before they can be read.

          Andreas Schroeder added a comment - - edited I have the same problem now on a Windows Server 2008 R2 Standard 64 Bit machine with Jenkins 1.654 and Pipeline plugin 1.15. I investigated a bit by using this Groovy script: node { for(int i=0; i < 100; ++i) { bat('echo "Hello from batch file."' + i.toString()) } } Jenkins creates two temporary batch files to run each "bat" call: jenkins-main.bat with the "echo ..." command and jenkins-wrap.bat to redirect output and capture the exit code in two text files jenkins-log.txt and jenkins-result.txt. At first the Groovy script is executing the bat call a few times correctly and then stops waiting. The console output does nothing but spin this little icon. When this happens, you can find the following state: a "durable-XXXXXXXX" folder (x are hex numbers) in a temporary workspace containing the two batch files jenkins-main.bat and jenkins-wrap.bat. The task manager does not show any of the batch files executing. The two text files jenkins-log.txt and jenkins-result.txt are not there Now I have found two ways to make Jenkins continue execting the Groovy script: run the jenkins-wrap.bat manually create the jenkins-log.txt file (can be empty) and the jenkins-result.txt file with a integer number as content, e.g. 0. I guess there are two ways this could happen (currently I do not know how to verify this): Either the jenkins-wrap.bat terminates without creating the text files. Or there is some race condition in the Jenkins code deleting the text files before they can be read.

          KK G added a comment -

          Very simple repro. Pipeline code is:

          node('master') {
          for(int i=0; i < 100; ++i)
          { bat('echo "Hello from batch file."' + i.toString()) }}

          Click "build now" 5 times.
          v 1.9 stuck all 5 jobs.
          After downgrading to v1.7, 1 of 5 can finish while other 4 still stuck.

          Please help. thanks.

          KK G added a comment - Very simple repro. Pipeline code is: node('master') { for(int i=0; i < 100; ++i) { bat('echo "Hello from batch file."' + i.toString()) }} Click "build now" 5 times. v 1.9 stuck all 5 jobs. After downgrading to v1.7, 1 of 5 can finish while other 4 still stuck. Please help. thanks.

          KK G added a comment -

          I have downgraded everything to v1.13. bat still got stuck.

          KK G added a comment - I have downgraded everything to v1.13. bat still got stuck.

          KK G added a comment -

          Today, I tried on windows 10/ windows 7/ windows 2008 too with Jenkins LTS and Jenkins 2.0. None of them works. This basic issue makes Jenkins pipeline unusable on Windows OS. May you please give it a high priority.

          KK G added a comment - Today, I tried on windows 10/ windows 7/ windows 2008 too with Jenkins LTS and Jenkins 2.0. None of them works. This basic issue makes Jenkins pipeline unusable on Windows OS. May you please give it a high priority.

          Geoff Rowell added a comment -

          This bug prevents use of pipeline plugins in an environment with Windows agents.

          Geoff Rowell added a comment - This bug prevents use of pipeline plugins in an environment with Windows agents.

          KK G added a comment -

          Thanks. Giving it a high priority is the right decision.

          BTW, not only Windows agents but also Windows master hit this bug.

          KK G added a comment - Thanks. Giving it a high priority is the right decision. BTW, not only Windows agents but also Windows master hit this bug.

          Martin S. added a comment -

          Since there is no known workaround, this is a critical issue if not blocker on Windows at least. I am eagerly waiting to test the pipelines/workflows because initial tests have shown that we can reduce our build times a lot with the added flexibility of workflows.

          Martin S. added a comment - Since there is no known workaround, this is a critical issue if not blocker on Windows at least. I am eagerly waiting to test the pipelines/workflows because initial tests have shown that we can reduce our build times a lot with the added flexibility of workflows.

          Martin S. added a comment - - edited

          JENKINS-28759 seems to be a similar issue.

          Martin S. added a comment - - edited JENKINS-28759 seems to be a similar issue.

          KK G added a comment -

          This one https://issues.jenkins-ci.org/browse/JENKINS-33749 is also related.

          This is definitely a blocker bug for me. If Jenkins cannot even run batch files, it would be useless on Windows OS. I spent quite time on Jenkins setup. If this bug cannot be solved, i have to switch to other CI. Please help.

          KK G added a comment - This one https://issues.jenkins-ci.org/browse/JENKINS-33749 is also related. This is definitely a blocker bug for me. If Jenkins cannot even run batch files, it would be useless on Windows OS. I spent quite time on Jenkins setup. If this bug cannot be solved, i have to switch to other CI. Please help.

          Martin Karing added a comment -

          JENKINS-34150 is also related to this.

          Martin Karing added a comment - JENKINS-34150 is also related to this.

          Christophe Carpentier added a comment - - edited

          Yup, definitely a blocker.

          Christophe Carpentier added a comment - - edited Yup, definitely a blocker.

          Geoff Rowell added a comment -

          I believe that this issue is caused by having the same agent/node executing the pipeline script as is targeted by the script.

          My initial setup had a Win7x64 master node that ran a pipeline script that included "node" commands that targeted the master node - as well as a secondary Win2008r2 node. This was able to reproduce the hang.

          Once I moved the master node to a CentOS 7 VM, and adjusted the script accordingly, I was able to reliably run the job. With this setup, all of the "heavyweight" executors are delegated to the remote agents.

          BTW, I'm using Jenkins 2.0.

          Geoff Rowell added a comment - I believe that this issue is caused by having the same agent/node executing the pipeline script as is targeted by the script. My initial setup had a Win7x64 master node that ran a pipeline script that included "node" commands that targeted the master node - as well as a secondary Win2008r2 node. This was able to reproduce the hang. Once I moved the master node to a CentOS 7 VM, and adjusted the script accordingly, I was able to reliably run the job. With this setup, all of the "heavyweight" executors are delegated to the remote agents. BTW, I'm using Jenkins 2.0.

          Nick Sonneveld added a comment - - edited

          Sorry that might be a red herring. I have been testing exclusively with a Linux master and a combination of linux and windows nodes. I still see this behaviour.

          edit1: However, I usually see it after running many batch scripts (around 400) on 30 executors across 4 nodes.
          edit2: It might also involve terminating jobs in the middle of running scripts

          Nick Sonneveld added a comment - - edited Sorry that might be a red herring. I have been testing exclusively with a Linux master and a combination of linux and windows nodes. I still see this behaviour. edit1: However, I usually see it after running many batch scripts (around 400) on 30 executors across 4 nodes. edit2: It might also involve terminating jobs in the middle of running scripts

          Andrew Coats added a comment - - edited

          I am having the same issue. Using Windows 10 Pro (Version 1511). Jenkins 2.1 with all the plugins at the latest version (5/5/2016).

          My workaround is to continue using Freestyle projects - but I was hoping to use the pipeline projects in order get a history of changes in GIT.

          Results (nunit2) saved as CalAgPermits.Maps.Web.UnitTests.xml
          Aborted by Andrew Coats
          Sending interrupt signal to process
          Click here to forcibly terminate running steps

          The "click here to forcibly terminate running steps" does not seem to work. Once the pipeline job is stuck - I have to reboot the machine and delete the pipeline job. If I do not delete the pipeline job... the job gets stuck in the build queue again even after the reboot (proving that pipeline jobs are durable!).

          Andrew Coats added a comment - - edited I am having the same issue. Using Windows 10 Pro (Version 1511). Jenkins 2.1 with all the plugins at the latest version (5/5/2016). My workaround is to continue using Freestyle projects - but I was hoping to use the pipeline projects in order get a history of changes in GIT. Results (nunit2) saved as CalAgPermits.Maps.Web.UnitTests.xml Aborted by Andrew Coats Sending interrupt signal to process Click here to forcibly terminate running steps The "click here to forcibly terminate running steps" does not seem to work. Once the pipeline job is stuck - I have to reboot the machine and delete the pipeline job. If I do not delete the pipeline job... the job gets stuck in the build queue again even after the reboot (proving that pipeline jobs are durable!).

          Julien Faixo added a comment -

          Same issue here.

          OS: Windows Server 2012 R2
          Jenkins: 2.5, all plugins up to date (20/05/2016)

          Unable to run any bat command (the first one hangs and never returns). Same workarounds as said :

          • in order to stop the task : click on the red cross on the build in order to send a killing signal, then go to the console output and click the "Terminate" link
          • in order to be able to build something : stick with the Freestyle projects for now

          It makes Pipeline unusable on Windows.

          Julien Faixo added a comment - Same issue here. OS: Windows Server 2012 R2 Jenkins: 2.5, all plugins up to date (20/05/2016) Unable to run any bat command (the first one hangs and never returns). Same workarounds as said : in order to stop the task : click on the red cross on the build in order to send a killing signal, then go to the console output and click the "Terminate" link in order to be able to build something : stick with the Freestyle projects for now It makes Pipeline unusable on Windows.

          Jesse Glick added a comment -

          all plugins up to date (20/05/2016)

          I need more details than that. durable-task 1.10, with an important fix for Windows users, was released around that time. And it is driven by workflow-durable-task-step.

          Jesse Glick added a comment - all plugins up to date (20/05/2016) I need more details than that. durable-task 1.10, with an important fix for Windows users, was released around that time. And it is driven by workflow-durable-task-step .

          Jesse Glick added a comment -

          Tried again to reproduce on 2012 without success. Also checked stopping and restarting Jenkins as java -jar jenkins.war while

          node {
              bat 'ping -n 100 localhost'
          }
          

          was running, which worked fine. I did see that if you install Jenkins as a Windows service and use /safeRestart (not currently a GUI option), and the build is using a master executor, the restart itself kills the ping process (according to Task Manager), so upon resuming, the build will either hang (until you abort it and then “forcibly kill running steps”) or print

          Reply from ::1: time<1ms 
          Reply from ::1: time<1ms 
          Resuming build
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          ERROR: script returned exit code -1
          Finished: FAILURE
          

          since pending JENKINS-25053 Jenkins does not have a way of knowing that the process exited abnormally. This also seems to happen when using a JNLP slave installed as a service, which is more relevant (since running builds on the master is discouraged anyway). JENKINS-27617 would probably help.

          At any rate, issues with the actual process being inappropriately killed during restarts are not related to the far more fundamental problem reported here, which I am unable to reproduce.

          Jesse Glick added a comment - Tried again to reproduce on 2012 without success. Also checked stopping and restarting Jenkins as java -jar jenkins.war while node { bat 'ping -n 100 localhost' } was running, which worked fine. I did see that if you install Jenkins as a Windows service and use /safeRestart (not currently a GUI option), and the build is using a master executor, the restart itself kills the ping process (according to Task Manager), so upon resuming, the build will either hang (until you abort it and then “forcibly kill running steps”) or print Reply from ::1: time<1ms Reply from ::1: time<1ms Resuming build [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline ERROR: script returned exit code -1 Finished: FAILURE since pending JENKINS-25053 Jenkins does not have a way of knowing that the process exited abnormally. This also seems to happen when using a JNLP slave installed as a service, which is more relevant (since running builds on the master is discouraged anyway). JENKINS-27617 would probably help. At any rate, issues with the actual process being inappropriately killed during restarts are not related to the far more fundamental problem reported here, which I am unable to reproduce.

          I have had this happen when my job name contained å,ä or ö.

          Erik Jonsson Thorén added a comment - I have had this happen when my job name contained å,ä or ö.

          Is somebody looking at this issue? I am also experiencing the same problem on Windows 10 and Windows 7 whenever i run a program with Batch.

          Enrique Ramirez added a comment - Is somebody looking at this issue? I am also experiencing the same problem on Windows 10 and Windows 7 whenever i run a program with Batch.

          We also hit this constantly on Win10. In our case it seems like if pipeline is trying to delete a running process then it will just be stuck there indefinitely. I can understand the headache of deleting files in use in general on windows, but I would hope that the Pipeline side of things could give up in accordance with the timeout settings we provide. Instead it will hang forever until we do the 3 step abort of terminating in the console.

          Justin Rainwater added a comment - We also hit this constantly on Win10. In our case it seems like if pipeline is trying to delete a running process then it will just be stuck there indefinitely. I can understand the headache of deleting files in use in general on windows, but I would hope that the Pipeline side of things could give up in accordance with the timeout settings we provide. Instead it will hang forever until we do the 3 step abort of terminating in the console.

            jglick Jesse Glick
            mocsharp mocsharp
            Votes:
            24 Vote for this issue
            Watchers:
            35 Start watching this issue

              Created:
              Updated: