Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33164

Pipeline bat stuck on Windows Server 2012

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I have Jenkins slave nodes configured on Windows Server 2012 with a very simple pipeline script:

      node('js-win') {
      	stage 'Checkout'
      	checkout scm
      	stage 'Build'
      	bat 'dir'
      }
      

      At the end of the script, from the Console output, it would just spin and hang after listing all the files.

        Attachments

          Issue Links

            Activity

            mocsharp mocsharp created issue -
            mocsharp mocsharp made changes -
            Field Original Value New Value
            Priority Major [ 3 ] Minor [ 4 ]
            Hide
            mocsharp mocsharp added a comment - - edited

            I tested it again with local Windows 2012 R2 and it worked fine. However, when it's configured on ec2 using WinRM, the process would never exit.

            If slave is conneted using JNLP, the process exits fine.

            Show
            mocsharp mocsharp added a comment - - edited I tested it again with local Windows 2012 R2 and it worked fine. However, when it's configured on ec2 using WinRM, the process would never exit. If slave is conneted using JNLP, the process exits fine.
            mocsharp mocsharp made changes -
            Component/s ec2-plugin [ 15625 ]
            Hide
            carlos Carlos Lucas added a comment - - edited

            I am having this problem on a Windows 2012 too, but I have it in the main node of jenkins.

            Downgrading the Durable Task Plugin to 1.7 solves de problem.

            Show
            carlos Carlos Lucas added a comment - - edited I am having this problem on a Windows 2012 too, but I have it in the main node of jenkins. Downgrading the Durable Task Plugin to 1.7 solves de problem.
            carlos Carlos Lucas made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            carlos Carlos Lucas made changes -
            Priority Major [ 3 ] Minor [ 4 ]
            Hide
            samuelkinesys Samuel Garcia added a comment -

            I'm having the same problem on a Windows 7 Professional SP1. Jenkins 1.652, Durable Task Plugin 1.8.

            My pipeline script that uses the 'bat' command hangs after the command has been successfully executed. I can't even kill the job, not even with the 'forcibly terminate' link that appears on the job's console. I have to stop Jenkins from the 'Services' window of Windows, then remove the job's folder.

            Is there any workaround for this? Since Jenkins won't allow me to downgrade the Durable Task Plugin

            Show
            samuelkinesys Samuel Garcia added a comment - I'm having the same problem on a Windows 7 Professional SP1. Jenkins 1.652, Durable Task Plugin 1.8. My pipeline script that uses the 'bat' command hangs after the command has been successfully executed. I can't even kill the job, not even with the 'forcibly terminate' link that appears on the job's console. I have to stop Jenkins from the 'Services' window of Windows, then remove the job's folder. Is there any workaround for this? Since Jenkins won't allow me to downgrade the Durable Task Plugin
            Hide
            motlib Andreas Schroeder added a comment - - edited

            I have the same problem now on a Windows Server 2008 R2 Standard 64 Bit machine with Jenkins 1.654 and Pipeline plugin 1.15.

            I investigated a bit by using this Groovy script:
            node
            {
            for(int i=0; i < 100; ++i)

            { bat('echo "Hello from batch file."' + i.toString()) }

            }
            Jenkins creates two temporary batch files to run each "bat" call: jenkins-main.bat with the "echo ..." command and jenkins-wrap.bat to redirect output and capture the exit code in two text files jenkins-log.txt and jenkins-result.txt.

            At first the Groovy script is executing the bat call a few times correctly and then stops waiting. The console output does nothing but spin this little icon. When this happens, you can find the following state:

            • a "durable-XXXXXXXX" folder (x are hex numbers) in a temporary workspace containing the two batch files jenkins-main.bat and jenkins-wrap.bat.
            • The task manager does not show any of the batch files executing.
            • The two text files jenkins-log.txt and jenkins-result.txt are not there

            Now I have found two ways to make Jenkins continue execting the Groovy script:

            • run the jenkins-wrap.bat manually
            • create the jenkins-log.txt file (can be empty) and the jenkins-result.txt file with a integer number as content, e.g. 0.

            I guess there are two ways this could happen (currently I do not know how to verify this): Either the jenkins-wrap.bat terminates without creating the text files. Or there is some race condition in the Jenkins code deleting the text files before they can be read.

            Show
            motlib Andreas Schroeder added a comment - - edited I have the same problem now on a Windows Server 2008 R2 Standard 64 Bit machine with Jenkins 1.654 and Pipeline plugin 1.15. I investigated a bit by using this Groovy script: node { for(int i=0; i < 100; ++i) { bat('echo "Hello from batch file."' + i.toString()) } } Jenkins creates two temporary batch files to run each "bat" call: jenkins-main.bat with the "echo ..." command and jenkins-wrap.bat to redirect output and capture the exit code in two text files jenkins-log.txt and jenkins-result.txt. At first the Groovy script is executing the bat call a few times correctly and then stops waiting. The console output does nothing but spin this little icon. When this happens, you can find the following state: a "durable-XXXXXXXX" folder (x are hex numbers) in a temporary workspace containing the two batch files jenkins-main.bat and jenkins-wrap.bat. The task manager does not show any of the batch files executing. The two text files jenkins-log.txt and jenkins-result.txt are not there Now I have found two ways to make Jenkins continue execting the Groovy script: run the jenkins-wrap.bat manually create the jenkins-log.txt file (can be empty) and the jenkins-result.txt file with a integer number as content, e.g. 0. I guess there are two ways this could happen (currently I do not know how to verify this): Either the jenkins-wrap.bat terminates without creating the text files. Or there is some race condition in the Jenkins code deleting the text files before they can be read.
            jglick Jesse Glick made changes -
            Component/s durable-task-plugin [ 18622 ]
            Component/s ec2-plugin [ 15625 ]
            Component/s workflow-plugin [ 18820 ]
            Hide
            flyingosprey KK G added a comment -

            Very simple repro. Pipeline code is:

            node('master') {
            for(int i=0; i < 100; ++i)
            { bat('echo "Hello from batch file."' + i.toString()) }}

            Click "build now" 5 times.
            v 1.9 stuck all 5 jobs.
            After downgrading to v1.7, 1 of 5 can finish while other 4 still stuck.

            Please help. thanks.

            Show
            flyingosprey KK G added a comment - Very simple repro. Pipeline code is: node('master') { for(int i=0; i < 100; ++i) { bat('echo "Hello from batch file."' + i.toString()) }} Click "build now" 5 times. v 1.9 stuck all 5 jobs. After downgrading to v1.7, 1 of 5 can finish while other 4 still stuck. Please help. thanks.
            Hide
            flyingosprey KK G added a comment -

            I have downgraded everything to v1.13. bat still got stuck.

            Show
            flyingosprey KK G added a comment - I have downgraded everything to v1.13. bat still got stuck.
            Hide
            flyingosprey KK G added a comment -

            Today, I tried on windows 10/ windows 7/ windows 2008 too with Jenkins LTS and Jenkins 2.0. None of them works. This basic issue makes Jenkins pipeline unusable on Windows OS. May you please give it a high priority.

            Show
            flyingosprey KK G added a comment - Today, I tried on windows 10/ windows 7/ windows 2008 too with Jenkins LTS and Jenkins 2.0. None of them works. This basic issue makes Jenkins pipeline unusable on Windows OS. May you please give it a high priority.
            geoff_nuance Geoff Rowell made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            Hide
            geoff_nuance Geoff Rowell added a comment -

            This bug prevents use of pipeline plugins in an environment with Windows agents.

            Show
            geoff_nuance Geoff Rowell added a comment - This bug prevents use of pipeline plugins in an environment with Windows agents.
            Hide
            flyingosprey KK G added a comment -

            Thanks. Giving it a high priority is the right decision.

            BTW, not only Windows agents but also Windows master hit this bug.

            Show
            flyingosprey KK G added a comment - Thanks. Giving it a high priority is the right decision. BTW, not only Windows agents but also Windows master hit this bug.
            Hide
            maskmaster Martin S. added a comment -

            Since there is no known workaround, this is a critical issue if not blocker on Windows at least. I am eagerly waiting to test the pipelines/workflows because initial tests have shown that we can reduce our build times a lot with the added flexibility of workflows.

            Show
            maskmaster Martin S. added a comment - Since there is no known workaround, this is a critical issue if not blocker on Windows at least. I am eagerly waiting to test the pipelines/workflows because initial tests have shown that we can reduce our build times a lot with the added flexibility of workflows.
            maskmaster Martin S. made changes -
            Link This issue duplicates JENKINS-28759 [ JENKINS-28759 ]
            Hide
            maskmaster Martin S. added a comment - - edited

            JENKINS-28759 seems to be a similar issue.

            Show
            maskmaster Martin S. added a comment - - edited JENKINS-28759 seems to be a similar issue.
            Hide
            flyingosprey KK G added a comment -

            This one https://issues.jenkins-ci.org/browse/JENKINS-33749 is also related.

            This is definitely a blocker bug for me. If Jenkins cannot even run batch files, it would be useless on Windows OS. I spent quite time on Jenkins setup. If this bug cannot be solved, i have to switch to other CI. Please help.

            Show
            flyingosprey KK G added a comment - This one https://issues.jenkins-ci.org/browse/JENKINS-33749 is also related. This is definitely a blocker bug for me. If Jenkins cannot even run batch files, it would be useless on Windows OS. I spent quite time on Jenkins setup. If this bug cannot be solved, i have to switch to other CI. Please help.
            Hide
            nitram Martin Karing added a comment -

            JENKINS-34150 is also related to this.

            Show
            nitram Martin Karing added a comment - JENKINS-34150 is also related to this.
            amuniz Antonio Muñiz made changes -
            Link This issue is related to JENKINS-34150 [ JENKINS-34150 ]
            Hide
            saucistophe Christophe Carpentier added a comment - - edited

            Yup, definitely a blocker.

            Show
            saucistophe Christophe Carpentier added a comment - - edited Yup, definitely a blocker.
            jglick Jesse Glick made changes -
            Labels windows
            Hide
            geoff_nuance Geoff Rowell added a comment -

            I believe that this issue is caused by having the same agent/node executing the pipeline script as is targeted by the script.

            My initial setup had a Win7x64 master node that ran a pipeline script that included "node" commands that targeted the master node - as well as a secondary Win2008r2 node. This was able to reproduce the hang.

            Once I moved the master node to a CentOS 7 VM, and adjusted the script accordingly, I was able to reliably run the job. With this setup, all of the "heavyweight" executors are delegated to the remote agents.

            BTW, I'm using Jenkins 2.0.

            Show
            geoff_nuance Geoff Rowell added a comment - I believe that this issue is caused by having the same agent/node executing the pipeline script as is targeted by the script. My initial setup had a Win7x64 master node that ran a pipeline script that included "node" commands that targeted the master node - as well as a secondary Win2008r2 node. This was able to reproduce the hang. Once I moved the master node to a CentOS 7 VM, and adjusted the script accordingly, I was able to reliably run the job. With this setup, all of the "heavyweight" executors are delegated to the remote agents. BTW, I'm using Jenkins 2.0.
            Hide
            sonneveldsmartward Nick Sonneveld added a comment - - edited

            Sorry that might be a red herring. I have been testing exclusively with a Linux master and a combination of linux and windows nodes. I still see this behaviour.

            edit1: However, I usually see it after running many batch scripts (around 400) on 30 executors across 4 nodes.
            edit2: It might also involve terminating jobs in the middle of running scripts

            Show
            sonneveldsmartward Nick Sonneveld added a comment - - edited Sorry that might be a red herring. I have been testing exclusively with a Linux master and a combination of linux and windows nodes. I still see this behaviour. edit1: However, I usually see it after running many batch scripts (around 400) on 30 executors across 4 nodes. edit2: It might also involve terminating jobs in the middle of running scripts
            Hide
            awcoats Andrew Coats added a comment - - edited

            I am having the same issue. Using Windows 10 Pro (Version 1511). Jenkins 2.1 with all the plugins at the latest version (5/5/2016).

            My workaround is to continue using Freestyle projects - but I was hoping to use the pipeline projects in order get a history of changes in GIT.

            Results (nunit2) saved as CalAgPermits.Maps.Web.UnitTests.xml
            Aborted by Andrew Coats
            Sending interrupt signal to process
            Click here to forcibly terminate running steps

            The "click here to forcibly terminate running steps" does not seem to work. Once the pipeline job is stuck - I have to reboot the machine and delete the pipeline job. If I do not delete the pipeline job... the job gets stuck in the build queue again even after the reboot (proving that pipeline jobs are durable!).

            Show
            awcoats Andrew Coats added a comment - - edited I am having the same issue. Using Windows 10 Pro (Version 1511). Jenkins 2.1 with all the plugins at the latest version (5/5/2016). My workaround is to continue using Freestyle projects - but I was hoping to use the pipeline projects in order get a history of changes in GIT. Results (nunit2) saved as CalAgPermits.Maps.Web.UnitTests.xml Aborted by Andrew Coats Sending interrupt signal to process Click here to forcibly terminate running steps The "click here to forcibly terminate running steps" does not seem to work. Once the pipeline job is stuck - I have to reboot the machine and delete the pipeline job. If I do not delete the pipeline job... the job gets stuck in the build queue again even after the reboot (proving that pipeline jobs are durable!).
            Hide
            jfaixo Julien Faixo added a comment -

            Same issue here.

            OS: Windows Server 2012 R2
            Jenkins: 2.5, all plugins up to date (20/05/2016)

            Unable to run any bat command (the first one hangs and never returns). Same workarounds as said :

            • in order to stop the task : click on the red cross on the build in order to send a killing signal, then go to the console output and click the "Terminate" link
            • in order to be able to build something : stick with the Freestyle projects for now

            It makes Pipeline unusable on Windows.

            Show
            jfaixo Julien Faixo added a comment - Same issue here. OS: Windows Server 2012 R2 Jenkins: 2.5, all plugins up to date (20/05/2016) Unable to run any bat command (the first one hangs and never returns). Same workarounds as said : in order to stop the task : click on the red cross on the build in order to send a killing signal, then go to the console output and click the "Terminate" link in order to be able to build something : stick with the Freestyle projects for now It makes Pipeline unusable on Windows.
            Hide
            jglick Jesse Glick added a comment -

            all plugins up to date (20/05/2016)

            I need more details than that. durable-task 1.10, with an important fix for Windows users, was released around that time. And it is driven by workflow-durable-task-step.

            Show
            jglick Jesse Glick added a comment - all plugins up to date (20/05/2016) I need more details than that. durable-task 1.10, with an important fix for Windows users, was released around that time. And it is driven by workflow-durable-task-step .
            Hide
            jglick Jesse Glick added a comment -

            Tried again to reproduce on 2012 without success. Also checked stopping and restarting Jenkins as java -jar jenkins.war while

            node {
                bat 'ping -n 100 localhost'
            }
            

            was running, which worked fine. I did see that if you install Jenkins as a Windows service and use /safeRestart (not currently a GUI option), and the build is using a master executor, the restart itself kills the ping process (according to Task Manager), so upon resuming, the build will either hang (until you abort it and then “forcibly kill running steps”) or print

            Reply from ::1: time<1ms 
            Reply from ::1: time<1ms 
            Resuming build
            [Pipeline] }
            [Pipeline] // node
            [Pipeline] End of Pipeline
            ERROR: script returned exit code -1
            Finished: FAILURE
            

            since pending JENKINS-25053 Jenkins does not have a way of knowing that the process exited abnormally. This also seems to happen when using a JNLP slave installed as a service, which is more relevant (since running builds on the master is discouraged anyway). JENKINS-27617 would probably help.

            At any rate, issues with the actual process being inappropriately killed during restarts are not related to the far more fundamental problem reported here, which I am unable to reproduce.

            Show
            jglick Jesse Glick added a comment - Tried again to reproduce on 2012 without success. Also checked stopping and restarting Jenkins as java -jar jenkins.war while node { bat 'ping -n 100 localhost' } was running, which worked fine. I did see that if you install Jenkins as a Windows service and use /safeRestart (not currently a GUI option), and the build is using a master executor, the restart itself kills the ping process (according to Task Manager), so upon resuming, the build will either hang (until you abort it and then “forcibly kill running steps”) or print Reply from ::1: time<1ms Reply from ::1: time<1ms Resuming build [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline ERROR: script returned exit code -1 Finished: FAILURE since pending JENKINS-25053 Jenkins does not have a way of knowing that the process exited abnormally. This also seems to happen when using a JNLP slave installed as a service, which is more relevant (since running builds on the master is discouraged anyway). JENKINS-27617 would probably help. At any rate, issues with the actual process being inappropriately killed during restarts are not related to the far more fundamental problem reported here, which I am unable to reproduce.
            jglick Jesse Glick made changes -
            Epic Link JENKINS-35399 [ 171192 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 169037 ] JNJira + In-Review [ 183335 ]
            Hide
            thecopy Erik Jonsson Thorén added a comment -

            I have had this happen when my job name contained å,ä or ö.

            Show
            thecopy Erik Jonsson Thorén added a comment - I have had this happen when my job name contained å,ä or ö.
            Hide
            esramirez Enrique Ramirez added a comment -

            Is somebody looking at this issue? I am also experiencing the same problem on Windows 10 and Windows 7 whenever i run a program with Batch.

            Show
            esramirez Enrique Ramirez added a comment - Is somebody looking at this issue? I am also experiencing the same problem on Windows 10 and Windows 7 whenever i run a program with Batch.
            Hide
            rainwaj Justin Rainwater added a comment -

            We also hit this constantly on Win10. In our case it seems like if pipeline is trying to delete a running process then it will just be stuck there indefinitely. I can understand the headache of deleting files in use in general on windows, but I would hope that the Pipeline side of things could give up in accordance with the timeout settings we provide. Instead it will hang forever until we do the 3 step abort of terminating in the console.

            Show
            rainwaj Justin Rainwater added a comment - We also hit this constantly on Win10. In our case it seems like if pipeline is trying to delete a running process then it will just be stuck there indefinitely. I can understand the headache of deleting files in use in general on windows, but I would hope that the Pipeline side of things could give up in accordance with the timeout settings we provide. Instead it will hang forever until we do the 3 step abort of terminating in the console.
            vivek Vivek Pandey made changes -
            Labels windows triaged-2018-11 windows
            slonopotamusorama Marat Radchenko made changes -
            Link This issue relates to JENKINS-59152 [ JENKINS-59152 ]

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              mocsharp mocsharp
              Votes:
              24 Vote for this issue
              Watchers:
              35 Start watching this issue

                Dates

                Created:
                Updated: