Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56441

Threads in TIMED_WAITING causing build agents to hold on to completed tasks

XMLWordPrintable

      Since roughlty 3 weeks our Jenkins instance started to have finished jobs which don't release the worker. They appear to be running but when we go to the job console there is a jenkins message saying 'SUCCESS' or 'FAILURE'. Since 2 days this issue occurs several time a day. This morning i tried to rollback to the latest LTS version but without success. You will find the thread dump attached.

      Any help will be appreciated as our users are strated to complain and we really don't know how to sort it out.

      Jenkins Version: 2.150.3
      Plugins: jenkins-plugins.txt

      Java version:  openjdk version "1.8.0_201" build "1.8.0_201_b09" 64-Bit Server VM (build 25.201-b09, mixed mode)

      May be related with JENKINS-56480

      Regards

       

        1. jenkins-plugins.txt
          3 kB

          [JENKINS-56441] Threads in TIMED_WAITING causing build agents to hold on to completed tasks

          laurent gil added a comment -

          We finally managed to find the root cause.

          We are using Github Enterprise which as a quota on the number of API request per hour per user. We increased the limit to 10K however the issue persists. What we find is that in decreasing the number of request made by Jenkins we no longer have the issue. After some investigations i think that the policy used to decrease the Jenkins activity depending on the quota is too agressive. Indeed we noticed some sleep thread with warning messages related with the quota whereas we had more than half of the quota available. In an other hand the slept thread never wake up so i think about a kind of deadlock into the code but as newbee i could not figure it out.

          Hope those information will help.

          laurent gil added a comment - We finally managed to find the root cause. We are using Github Enterprise which as a quota on the number of API request per hour per user. We increased the limit to 10K however the issue persists. What we find is that in decreasing the number of request made by Jenkins we no longer have the issue. After some investigations i think that the policy used to decrease the Jenkins activity depending on the quota is too agressive. Indeed we noticed some sleep thread with warning messages related with the quota whereas we had more than half of the quota available. In an other hand the slept thread never wake up so i think about a kind of deadlock into the code but as newbee i could not figure it out. Hope those information will help.

          laurent gil added a comment -

          We find a way to fix it, however we think there is a bug which prevent thread to wake up.

          laurent gil added a comment - We find a way to fix it, however we think there is a bug which prevent thread to wake up.

          Olivier Dagenais added a comment -

          Do you mind sharing your discoveries and your solution(s)?

          Thanks!

          Olivier Dagenais added a comment - Do you mind sharing your discoveries and your solution(s)? Thanks!

            Unassigned Unassigned
            angegar laurent gil
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: