Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67579

Jenkins become unresponsive - thread deadlocks and high CPU

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • blueocean-plugin, core
    • None

      Recently we performed an upgrade from jenkins/jenkins:2.263.3-lts to jenkins/jenkins:2.303.2-lts-jdk11, as well as updated most of our plugins. 

      Since then, every multiple days Jenkins becomes unresponsive, with a very high CPU, while threads are being deadlocked (always the same ones).

      Found one Java-level deadlock:
      =============================
      "Connector-Scheduler-69e1dd28-1":
      waiting to lock monitor 0x00007f10f8262b00 (object 0x000000077182da90, a org.eclipse.jetty.server.HttpChannelState),
      which is held by "Jetty (winstone)-2368"
      "Jetty (winstone)-2368":
      waiting to lock monitor 0x00007f1064546a80 (object 0x0000000716af29d8, a org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint),
      which is held by "GuavaPubsubBus.ThreadPoolExecutor"
      "GuavaPubsubBus.ThreadPoolExecutor":
      waiting to lock monitor 0x00007f10f8262b00 (object 0x000000077182da90, a org.eclipse.jetty.server.HttpChannelState),
      which is held by "Jetty (winstone)-2368"

      Our only solution is restarting the container each and every time.  We also upgraded again the Jenkins version to jenkins/jenkins:2.303.3-lts-jdk11, in hope it will maybe solve the issue, but it did not help.

      We've noticed that 'jenkins-blueocean-core-js' threads are always getting BLOCKED, as shown in the attachments.

      We'll happily share any additional info if needed.

            Unassigned Unassigned
            romjacoby Rom Jacoby
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: