Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-39179

All builds hang, JNA load deadlock on Windows slave

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I hate to create a general "core" bug, as I wish I could redirect this to the correct component. Unfortunately, I can not identify which component is hanging and why, so I do not know how to direct this problem.

      This problem started about 2 weeks ago, as we have been adding new Pipeline builds to our build server. So it could be related to one of the pipeline plugins.

      The behavior is the following:

      • 1 to 2 times a day, all builds on all build slaves will hang. The console log of the build just stops moving forward, and stays stuck at the last line executed / last line returned.
      • Once this occurs, attempting to stop a build fails. Clicking stop results in no change in the build status or console log output
      • New builds will not start. They sit in the queue, but the slaves will not be started.
      • The UI continues to function, so it is possible to view config, get threaddumps, etc.

      The only resolution is to restart the Jenkins server.

      We are using the vCenter plugin to dynamically start all build slaves. Though, we have been using this configuration for months, and the problem just started.

      We have recreated this on both latest Jenkins level (2.26) and Jenkins LTS version 2.19.1

      I am attaching a threaddump of the server at the time of one of these hangs.

      I can provide any other information that might help in diagnosing this problem

        Attachments

          Issue Links

            Activity

            gregcovertsmith Greg Smith created issue -
            gregcovertsmith Greg Smith made changes -
            Field Original Value New Value
            Component/s durable-task-plugin [ 18622 ]
            Component/s core [ 15593 ]
            gregcovertsmith Greg Smith made changes -
            Environment Jenkins 2.19.1 LTS
            and
            Jenkins 2.26
            Jenkins 2.19.1 LTS
            and
            Jenkins 2.26
            Durable Task Plugin 1.12
            gregcovertsmith Greg Smith made changes -
            Summary All builds hang, Builds cannot be stopped, only restart solves All builds hang, Builds cannot be stopped, hung FileMonitoringCleanup
            gregcovertsmith Greg Smith made changes -
            Summary All builds hang, Builds cannot be stopped, hung FileMonitoringCleanup All builds hang, hung FileMonitoringTask.cleanup / get attributes on Windows 10
            gregcovertsmith Greg Smith made changes -
            Component/s core [ 15593 ]
            gregcovertsmith Greg Smith made changes -
            Component/s core [ 15593 ]
            gregcovertsmith Greg Smith made changes -
            Attachment JenkinsHungThreadStack2.txt [ 34469 ]
            gregcovertsmith Greg Smith made changes -
            Component/s core [ 15593 ]
            gregcovertsmith Greg Smith made changes -
            Link This issue is related to JENKINS-19445 [ JENKINS-19445 ]
            gregcovertsmith Greg Smith made changes -
            Summary All builds hang, hung FileMonitoringTask.cleanup / get attributes on Windows 10 All builds hang, JNA load deadlock on Windows slave
            gregcovertsmith Greg Smith made changes -
            Attachment JenkinsHungThreadStack3.txt [ 34671 ]
            gregcovertsmith Greg Smith made changes -
            Environment Jenkins 2.19.1 LTS
            and
            Jenkins 2.26
            Durable Task Plugin 1.12
            Jenkins 2.19.1 LTS
            Jenkins 2.19.2 LTS
            Jenkins 2.26
            Durable Task Plugin 1.12
            gregcovertsmith Greg Smith made changes -
            Attachment JenkinsHungThreadStack4.txt [ 34672 ]
            gregcovertsmith Greg Smith made changes -
            Component/s slave-status-plugin [ 15981 ]
            gregcovertsmith Greg Smith made changes -
            Link This issue is related to JENKINS-16070 [ JENKINS-16070 ]
            gregcovertsmith Greg Smith made changes -
            Link This issue is related to JENKINS-38834 [ JENKINS-38834 ]
            oleg_nenashev Oleg Nenashev made changes -
            Component/s remoting [ 15489 ]
            Component/s slave-status-plugin [ 15981 ]
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is related to JENKINS-36088 [ JENKINS-36088 ]
            oleg_nenashev Oleg Nenashev made changes -
            Labels java-ibm
            oleg_nenashev Oleg Nenashev made changes -
            Link This issue is duplicated by JENKINS-22824 [ JENKINS-22824 ]
            jamesdumay James Dumay made changes -
            Remote Link This issue links to "CloudBees Internal OSS-2493 (Web Link)" [ 18273 ]
            jglick Jesse Glick made changes -
            Attachment avoid-agent-jna-deadlock.hpi [ 40700 ]
            heldermagalhaes Helder Magalhães made changes -
            Attachment MyLinuxSlave-SystemInformation.txt [ 42250 ]
            vivek Vivek Pandey made changes -
            Labels java-ibm java-ibm triaged-2018-11

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              gregcovertsmith Greg Smith
              Votes:
              4 Vote for this issue
              Watchers:
              18 Start watching this issue

                Dates

                Created:
                Updated: