Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67579

Jenkins become unresponsive - thread deadlocks and high CPU

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • blueocean-plugin, (1)
      core
    • None

      Recently we performed an upgrade from jenkins/jenkins:2.263.3-lts to jenkins/jenkins:2.303.2-lts-jdk11, as well as updated most of our plugins. 

      Since then, every multiple days Jenkins becomes unresponsive, with a very high CPU, while threads are being deadlocked (always the same ones).

      Found one Java-level deadlock:
      =============================
      "Connector-Scheduler-69e1dd28-1":
      waiting to lock monitor 0x00007f10f8262b00 (object 0x000000077182da90, a org.eclipse.jetty.server.HttpChannelState),
      which is held by "Jetty (winstone)-2368"
      "Jetty (winstone)-2368":
      waiting to lock monitor 0x00007f1064546a80 (object 0x0000000716af29d8, a org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint),
      which is held by "GuavaPubsubBus.ThreadPoolExecutor"
      "GuavaPubsubBus.ThreadPoolExecutor":
      waiting to lock monitor 0x00007f10f8262b00 (object 0x000000077182da90, a org.eclipse.jetty.server.HttpChannelState),
      which is held by "Jetty (winstone)-2368"

      Our only solution is restarting the container each and every time.  We also upgraded again the Jenkins version to jenkins/jenkins:2.303.3-lts-jdk11, in hope it will maybe solve the issue, but it did not help.

      We've noticed that 'jenkins-blueocean-core-js' threads are always getting BLOCKED, as shown in the attachments.

      We'll happily share any additional info if needed.

          [JENKINS-67579] Jenkins become unresponsive - thread deadlocks and high CPU

          Mark Waite added a comment -

          I don't know if the advice will help in any way, but you might consider removing the deprecated plugins like

          DEPRECATED Blue Ocean Executor Info (blueocean-executor-info): 1.25.2
          

          Any other deprecated plugins should be removed..

          I've never found the Blue Ocean autofavorite plugin useful. You might consider removing it as well.

          Outdated plugins should be upgraded to their latest releases.

          Plugins with known security issues should be removed.

          Unused plugins should be removed.

          If that is not enough, then you might consider removing the monitoring plugin that uses Java Melody. It is mentioned in one of the deadlock stack traces.

          Mark Waite added a comment - I don't know if the advice will help in any way, but you might consider removing the deprecated plugins like DEPRECATED Blue Ocean Executor Info (blueocean-executor-info): 1.25.2 Any other deprecated plugins should be removed.. I've never found the Blue Ocean autofavorite plugin useful. You might consider removing it as well. Outdated plugins should be upgraded to their latest releases. Plugins with known security issues should be removed. Unused plugins should be removed. If that is not enough, then you might consider removing the monitoring plugin that uses Java Melody. It is mentioned in one of the deadlock stack traces.

            Unassigned Unassigned
            romjacoby Rom Jacoby
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: