Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31536

Jobs hanging in Build Executor even if it is finished

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • core

      Since a couple of days (since updating to 1.609.3 LTS) we have the problem that is the builds stick in the bild executor.If you open the build details the build is finished allready. But the executor is still blocked leading to a Build-queue that is constantly growing. Sometime the executor will display "DEAD " (German: "Tot "). There are no specific entries in the log file.

      Master is running on a virtual Windows Server 2012. Slaves are Windoes Server 2012 and Windows 7.

        1. BuildExecutors.png
          BuildExecutors.png
          80 kB
        2. jenkins_dump.txt
          451 kB
        3. jenkins-plugins.txt
          3 kB
        4. Thread Dump [Jenkins].pdf
          288 kB

          [JENKINS-31536] Jobs hanging in Build Executor even if it is finished

          ASHOK MOHANTY added a comment -

          Same for me - Jenkins ver. 2.121.3 and we don't have node stalker plugin

          ASHOK MOHANTY added a comment - Same for me - Jenkins ver. 2.121.3 and we don't have node stalker plugin

          Naresh BS added a comment -

          We are running into similar issues with our Jenkins setup on version 2.121.1.

          Has anyone been able to figure out the root cause or how to fix this? Has this been fixed in any version of Jenkins yet.

          Thanks!

          Naresh BS added a comment - We are running into similar issues with our Jenkins setup on version 2.121.1. Has anyone been able to figure out the root cause or how to fix this? Has this been fixed in any version of Jenkins yet. Thanks!

          laurent gil added a comment -

          I  have a similar issue with the version 2.167. The first occurrence was with the version 2.166. Any help will be appreciated.

           

          Thanks !

          laurent gil added a comment - I  have a similar issue with the version 2.167. The first occurrence was with the version 2.166. Any help will be appreciated.   Thanks !

          Daniel Beck added a comment -

          Anyone experiencing this should try to identify the culprit via https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump at the time the problem occurs. What is the node busy with? File a new issue with this information, ideally against the identified component.

          Daniel Beck added a comment - Anyone experiencing this should try to identify the culprit via https://wiki.jenkins.io/display/JENKINS/Obtaining+a+thread+dump at the time the problem occurs. What is the node busy with? File a new issue with this information, ideally against the identified component.

          laurent gil added a comment - - edited

          Since roughlty 3 weeks our Jenkins instance started to have finished jobs which don't release the worker. They appear to be running but when we go to the job console there is a jenkins message saying 'SUCCESS' or 'FAILURE'. Since 2 days this issue occurs several time a day. This morning i tried to rollback to the latest LTS version but without success. You will find the thread dump attached.

          Any help will be appreciated as our users are strated to complain and we really don't know how to sort it out.

          Jenkins Version: 2.150.3
          Plugins: jenkins-plugins.txt

          Java version:  openjdk version "1.8.0_201" build "1.8.0_201_b09" 64-Bit Server VM (build 25.201-b09, mixed mode)

          Regards

           

          laurent gil added a comment - - edited Since roughlty 3 weeks our Jenkins instance started to have finished jobs which don't release the worker. They appear to be running but when we go to the job console there is a jenkins message saying 'SUCCESS' or 'FAILURE'. Since 2 days this issue occurs several time a day. This morning i tried to rollback to the latest LTS version but without success. You will find the thread dump attached. Any help will be appreciated as our users are strated to complain and we really don't know how to sort it out. Jenkins Version: 2.150.3 Plugins:   jenkins-plugins.txt Java version:   openjdk version "1.8.0_201" build "1.8.0_201_b09" 64-Bit Server VM (build 25.201-b09, mixed mode) Regards  

          Jenkins Ark added a comment -

          Same problem.

          Seems like this problem happens after upgrading to 2.164.1

          After downgrade to 2.150.3 everything is fine.

           

          Jenkins Ark added a comment - Same problem. Seems like this problem happens after upgrading to 2.164.1 After downgrade to 2.150.3 everything is fine.  

          Daniel Beck added a comment -

          Since you all are contradicting each other, you're making further investigation impossible. Why bother commenting at all? Please file new issues and provide information beyond version number. Some ideas: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue

          Daniel Beck added a comment - Since you all are contradicting each other, you're making further investigation impossible. Why bother commenting at all? Please file new issues and provide information beyond version number. Some ideas: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue

          Jenkins Ark added a comment -

          Sorry for the unclear comment. My bad!

          Actually after some test I found that downgrade to 2.150.3 CANNOT solve the problem. 

           

          Jenkins Ark added a comment - Sorry for the unclear comment. My bad! Actually after some test I found that downgrade to 2.150.3 CANNOT solve the problem.   

          Jenkins Ark added a comment - - edited

          FYI the problem occurs at a Windows machine, after some investigation I found that the master(Linux machine) failed to ping the Windows slave (time out). Although the master can successfully ping the slave.

          So I turn on the ping access on the Windows Firewall, and the problem is gone!

           

          Below is a piece of the SystemInfo of the Windows slave that inspired me:

          Ping thread for channel hudson.remoting.Channel@432d3ea4:JNLP4-connect connection to 10.5.57.xxx/10.5.57.xxx:59095

          "Ping thread for channel hudson.remoting.Channel@432d3ea4:JNLP4-connect connection to 10.5.57.xxx/10.5.57.xxx:59095" Id=28 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:95)

           

          Jenkins Ark added a comment - - edited FYI the problem occurs at a Windows machine, after some investigation I found that the master(Linux machine) failed to ping the Windows slave (time out). Although the master can successfully ping the slave. So I turn on the ping access on the Windows Firewall, and the problem is gone!   Below is a piece of the SystemInfo of the Windows slave that inspired me: Ping thread for channel hudson.remoting.Channel@432d3ea4:JNLP4-connect connection to 10.5.57.xxx/10.5.57.xxx:59095 "Ping thread for channel hudson.remoting.Channel@432d3ea4:JNLP4-connect connection to 10.5.57.xxx/10.5.57.xxx:59095" Id=28 Group=main TIMED_WAITING at java.lang.Thread.sleep(Native Method) at hudson.remoting.PingThread.run(PingThread.java:95)  

          Jenkins Ark added a comment -

          FYI. Turn on the ping access on the Windows slave DONOT solve the problem. It was an illusion.....

          Jenkins Ark added a comment - FYI. Turn on the ping access on the Windows slave DONOT solve the problem. It was an illusion.....

            Unassigned Unassigned
            herrmann81 Stefan Herrmann
            Votes:
            6 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: