Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56441

Threads in TIMED_WAITING causing build agents to hold on to completed tasks

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Since roughlty 3 weeks our Jenkins instance started to have finished jobs which don't release the worker. They appear to be running but when we go to the job console there is a jenkins message saying 'SUCCESS' or 'FAILURE'. Since 2 days this issue occurs several time a day. This morning i tried to rollback to the latest LTS version but without success. You will find the thread dump attached.

      Any help will be appreciated as our users are strated to complain and we really don't know how to sort it out.

      Jenkins Version: 2.150.3
      Plugins: jenkins-plugins.txt

      Java version:  openjdk version "1.8.0_201" build "1.8.0_201_b09" 64-Bit Server VM (build 25.201-b09, mixed mode)

      May be related with JENKINS-56480

      Regards

       

        Attachments

          Issue Links

            Activity

            Hide
            angegar laurent gil added a comment -

            We finally managed to find the root cause.

            We are using Github Enterprise which as a quota on the number of API request per hour per user. We increased the limit to 10K however the issue persists. What we find is that in decreasing the number of request made by Jenkins we no longer have the issue. After some investigations i think that the policy used to decrease the Jenkins activity depending on the quota is too agressive. Indeed we noticed some sleep thread with warning messages related with the quota whereas we had more than half of the quota available. In an other hand the slept thread never wake up so i think about a kind of deadlock into the code but as newbee i could not figure it out.

            Hope those information will help.

            Show
            angegar laurent gil added a comment - We finally managed to find the root cause. We are using Github Enterprise which as a quota on the number of API request per hour per user. We increased the limit to 10K however the issue persists. What we find is that in decreasing the number of request made by Jenkins we no longer have the issue. After some investigations i think that the policy used to decrease the Jenkins activity depending on the quota is too agressive. Indeed we noticed some sleep thread with warning messages related with the quota whereas we had more than half of the quota available. In an other hand the slept thread never wake up so i think about a kind of deadlock into the code but as newbee i could not figure it out. Hope those information will help.
            Hide
            angegar laurent gil added a comment -

            We find a way to fix it, however we think there is a bug which prevent thread to wake up.

            Show
            angegar laurent gil added a comment - We find a way to fix it, however we think there is a bug which prevent thread to wake up.
            Hide
            oli_at_jsi Olivier Dagenais added a comment -

            Do you mind sharing your discoveries and your solution(s)?

            Thanks!

            Show
            oli_at_jsi Olivier Dagenais added a comment - Do you mind sharing your discoveries and your solution(s)? Thanks!

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              angegar laurent gil
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: