We are experiencing dead locks in Jenkins. I am not sure if its related but we recently upgraded to Jenkins 2.289.1. Since then we had two dead locks about 3 days apart. Jenkins does not shut down gracefully and does only stop when using kill -9.

      Found one Java-level deadlock:
      =============================
      "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/ecommerceapps%252Fecommerce/branches/ from 127.0.0.1 : Jetty (winstone)-263503":
        waiting to lock monitor 0x00007f0bc0186788 (object 0x000000046cb7ddc8, a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution),
        which is held by "Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]"
      "Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]":
        waiting to lock monitor 0x00007f0b9c01ed18 (object 0x000000046cb7de38, a org.jenkinsci.plugins.workflow.graph.StandardGraphLookupView),
        which is held by "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/pipelines/ecommerceapps%2Fecommerce/branches/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/runs/17/ from 10.120.1.90 : Jetty (winstone)-72660"
      "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/pipelines/ecommerceapps%2Fecommerce/branches/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/runs/17/ from 10.120.1.90 : Jetty (winstone)-72660":
        waiting to lock monitor 0x00007f0bc0186788 (object 0x000000046cb7ddc8, a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution),
        which is held by "Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]" 

       

       

      Java stack information for the threads listed above:
      ===================================================
      "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/ecommerceapps%252Fecommerce/branches/ from 127.0.0.1 : Jetty (winstone)-263503":
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.getCurrentHeads(CpsFlowExecution.java:981)
              - waiting to lock <0x000000046cb7ddc8> (a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution)
              at org.jenkinsci.plugins.workflow.flow.FlowExecution.isComplete(FlowExecution.java:208)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.isComplete(CpsFlowExecution.java:1238) 

          [JENKINS-65885] Deadlock in CpsFlowExecution

          Kyle added a comment -

          Was able to workaround the issue by downgrading a number of packages necessary to roll back the Groovy plugin v2.92 -> v2.90 which is where I suspect the issue is.

          Here's the working package diff still on Jenkins LTS v2.289.1

          --- plugins-2021-06-14.json	2021-06-15 17:01:57.537752150 -0700
          +++ plugins-2021-06-14-fix.json	2021-06-15 19:15:19.422134736 -0700
          @@ -4,11 +4,11 @@
             "Authorize Project": "1.4.0",
             "Autofavorite for Blue Ocean": "1.2.4",
             "Bitbucket Branch Source Plugin": "2.9.9",
          -  "Bitbucket Pipeline for Blue Ocean": "1.24.7",
          +  "Bitbucket Pipeline for Blue Ocean": "1.24.6",
             "Bitbucket Plugin": "1.1.29",
          -  "Blue Ocean": "1.24.7",
          +  "Blue Ocean": "1.24.6",
             "Blue Ocean Core JS": "1.24.7",
          -  "Blue Ocean Pipeline Editor": "1.24.7",
          +  "Blue Ocean Pipeline Editor": "1.24.6",
             "Bootstrap 4 API Plugin": "4.6.0-3",
             "Bootstrap 5 API Plugin": "5.0.1-2",
             "Branch API Plugin": "2.6.4",
          @@ -31,18 +31,18 @@
             "Durable Task Plugin": "1.37",
             "ECharts API Plugin": "5.1.2-2",
             "Email Extension Plugin": "2.83",
          -  "Events API for Blue Ocean": "1.24.7",
          +  "Events API for Blue Ocean": "1.24.6",
             "External Monitor Job Type Plugin": "1.7",
             "Favorite": "2.3.3",
             "Folders Plugin": "6.15",
             "Font Awesome API Plugin": "5.15.3-3",
             "GIT server Plugin": "1.9",
          -  "Git Pipeline for Blue Ocean": "1.24.7",
          +  "Git Pipeline for Blue Ocean": "1.24.6",
             "Git client plugin": "3.7.2",
             "Git plugin": "4.7.2",
             "GitHub API Plugin": "1.123",
             "GitHub Branch Source Plugin": "2.11.1",
          -  "GitHub Pipeline for Blue Ocean": "1.24.7",
          +  "GitHub Pipeline for Blue Ocean": "1.24.6",
             "GitHub plugin": "1.33.1",
             "GitLab Branch Source Plugin": "1.5.7",
             "GitLab Plugin": "1.5.20",
          @@ -50,7 +50,7 @@
             "Google Login Plugin": "1.6",
             "Gradle Plugin": "1.36",
             "HTML Publisher plugin": "1.25",
          -  "HTTP Request Plugin": "1.8.27",
          +  "HTTP Request Plugin": "1.9.0",
             "Handy Uri Templates 2.x API Plugin": "2.1.8-1.0",
             "JIRA Integration for Blue Ocean": "1.24.7",
             "JQuery3 API Plugin": "3.6.0-1",
          @@ -80,30 +80,30 @@
             "Oracle Java SE Development Kit Installer Plugin": "1.5",
             "PAM Authentication plugin": "1.6",
             "Personalization for Blue Ocean": "1.24.7",
          -  "Pipeline": "2.6",
          -  "Pipeline Graph Analysis Plugin": "1.11",
          +  "Pipeline": "2.5",
          +  "Pipeline Graph Analysis Plugin": "1.10",
             "Pipeline SCM API for Blue Ocean": "1.24.7",
             "Pipeline Utility Steps": "2.8.0",
          -  "Pipeline implementation for Blue Ocean": "1.24.7",
          +  "Pipeline implementation for Blue Ocean": "1.24.6",
             "Pipeline: API": "2.44",
             "Pipeline: Basic Steps": "2.23",
             "Pipeline: Build Step": "2.13",
             "Pipeline: Declarative": "1.8.5",
             "Pipeline: Declarative Extension Points API": "1.8.5",
             "Pipeline: GitHub Groovy Libraries": "1.0",
          -  "Pipeline: Groovy": "2.92",
          +  "Pipeline: Groovy": "2.90",
             "Pipeline: Input Step": "2.12",
             "Pipeline: Job": "2.41",
             "Pipeline: Milestone Step": "1.3.2",
             "Pipeline: Model API": "1.8.5",
             "Pipeline: Multibranch": "2.24",
             "Pipeline: Nodes and Processes": "2.39",
          -  "Pipeline: REST API Plugin": "2.19",
          +  "Pipeline: REST API Plugin": "2.17",
             "Pipeline: SCM Step": "2.12",
             "Pipeline: Shared Groovy Libraries": "2.19",
             "Pipeline: Stage Step": "2.5",
             "Pipeline: Stage Tags Metadata": "1.8.5",
          -  "Pipeline: Stage View Plugin": "2.19",
          +  "Pipeline: Stage View Plugin": "2.17",
             "Pipeline: Step API": "2.23",
             "Pipeline: Supporting APIs": "3.8",
             "Plain Credentials Plugin": "1.7",
          @@ -126,7 +126,7 @@
             "Slack Notification Plugin": "2.48",
             "Snakeyaml API Plugin": "1.29.1",
             "Structs Plugin": "1.23",
          -  "Subversion Plug-in": "2.14.3",
          +  "Subversion Plug-in": "2.14.4",
             "Timestamper": "1.13",
             "Token Macro Plugin": "2.15",
             "Trilead API Plugin": "1.0.13",
          

          The HTTP Request Plugin and Subversion Plug-in are likely extraneous changes from earlier debugging and unapplied updates.

          Not familiar enough with Jenkins and plugins to offer much assistance in patching it, but hopefully this helps narrow it down and resolve issues for others encountering the same issue.

          Kyle added a comment - Was able to workaround the issue by downgrading a number of packages necessary to roll back the Groovy plugin v2.92 -> v2.90 which is where I suspect the issue is. Here's the working package diff still on Jenkins LTS v2.289.1 --- plugins-2021-06-14.json 2021-06-15 17:01:57.537752150 -0700 +++ plugins-2021-06-14-fix.json 2021-06-15 19:15:19.422134736 -0700 @@ -4,11 +4,11 @@ "Authorize Project" : "1.4.0" , "Autofavorite for Blue Ocean" : "1.2.4" , "Bitbucket Branch Source Plugin" : "2.9.9" , - "Bitbucket Pipeline for Blue Ocean" : "1.24.7" , + "Bitbucket Pipeline for Blue Ocean" : "1.24.6" , "Bitbucket Plugin" : "1.1.29" , - "Blue Ocean" : "1.24.7" , + "Blue Ocean" : "1.24.6" , "Blue Ocean Core JS" : "1.24.7" , - "Blue Ocean Pipeline Editor" : "1.24.7" , + "Blue Ocean Pipeline Editor" : "1.24.6" , "Bootstrap 4 API Plugin" : "4.6.0-3" , "Bootstrap 5 API Plugin" : "5.0.1-2" , "Branch API Plugin" : "2.6.4" , @@ -31,18 +31,18 @@ "Durable Task Plugin" : "1.37" , "ECharts API Plugin" : "5.1.2-2" , "Email Extension Plugin" : "2.83" , - "Events API for Blue Ocean" : "1.24.7" , + "Events API for Blue Ocean" : "1.24.6" , "External Monitor Job Type Plugin" : "1.7" , "Favorite" : "2.3.3" , "Folders Plugin" : "6.15" , "Font Awesome API Plugin" : "5.15.3-3" , "GIT server Plugin" : "1.9" , - "Git Pipeline for Blue Ocean" : "1.24.7" , + "Git Pipeline for Blue Ocean" : "1.24.6" , "Git client plugin" : "3.7.2" , "Git plugin" : "4.7.2" , "GitHub API Plugin" : "1.123" , "GitHub Branch Source Plugin" : "2.11.1" , - "GitHub Pipeline for Blue Ocean" : "1.24.7" , + "GitHub Pipeline for Blue Ocean" : "1.24.6" , "GitHub plugin" : "1.33.1" , "GitLab Branch Source Plugin" : "1.5.7" , "GitLab Plugin" : "1.5.20" , @@ -50,7 +50,7 @@ "Google Login Plugin" : "1.6" , "Gradle Plugin" : "1.36" , "HTML Publisher plugin" : "1.25" , - "HTTP Request Plugin" : "1.8.27" , + "HTTP Request Plugin" : "1.9.0" , "Handy Uri Templates 2.x API Plugin" : "2.1.8-1.0" , "JIRA Integration for Blue Ocean" : "1.24.7" , "JQuery3 API Plugin" : "3.6.0-1" , @@ -80,30 +80,30 @@ "Oracle Java SE Development Kit Installer Plugin" : "1.5" , "PAM Authentication plugin" : "1.6" , "Personalization for Blue Ocean" : "1.24.7" , - "Pipeline" : "2.6" , - "Pipeline Graph Analysis Plugin" : "1.11" , + "Pipeline" : "2.5" , + "Pipeline Graph Analysis Plugin" : "1.10" , "Pipeline SCM API for Blue Ocean" : "1.24.7" , "Pipeline Utility Steps" : "2.8.0" , - "Pipeline implementation for Blue Ocean" : "1.24.7" , + "Pipeline implementation for Blue Ocean" : "1.24.6" , "Pipeline: API" : "2.44" , "Pipeline: Basic Steps" : "2.23" , "Pipeline: Build Step" : "2.13" , "Pipeline: Declarative" : "1.8.5" , "Pipeline: Declarative Extension Points API" : "1.8.5" , "Pipeline: GitHub Groovy Libraries" : "1.0" , - "Pipeline: Groovy" : "2.92" , + "Pipeline: Groovy" : "2.90" , "Pipeline: Input Step" : "2.12" , "Pipeline: Job" : "2.41" , "Pipeline: Milestone Step" : "1.3.2" , "Pipeline: Model API" : "1.8.5" , "Pipeline: Multibranch" : "2.24" , "Pipeline: Nodes and Processes" : "2.39" , - "Pipeline: REST API Plugin" : "2.19" , + "Pipeline: REST API Plugin" : "2.17" , "Pipeline: SCM Step" : "2.12" , "Pipeline: Shared Groovy Libraries" : "2.19" , "Pipeline: Stage Step" : "2.5" , "Pipeline: Stage Tags Metadata" : "1.8.5" , - "Pipeline: Stage View Plugin" : "2.19" , + "Pipeline: Stage View Plugin" : "2.17" , "Pipeline: Step API" : "2.23" , "Pipeline: Supporting APIs" : "3.8" , "Plain Credentials Plugin" : "1.7" , @@ -126,7 +126,7 @@ "Slack Notification Plugin" : "2.48" , "Snakeyaml API Plugin" : "1.29.1" , "Structs Plugin" : "1.23" , - "Subversion Plug-in" : "2.14.3" , + "Subversion Plug-in" : "2.14.4" , "Timestamper" : "1.13" , "Token Macro Plugin" : "2.15" , "Trilead API Plugin" : "1.0.13" , The HTTP Request Plugin and Subversion Plug-in are likely extraneous changes from earlier debugging and unapplied updates. Not familiar enough with Jenkins and plugins to offer much assistance in patching it, but hopefully this helps narrow it down and resolve issues for others encountering the same issue.

          Mirko added a comment -

          Thanks 2bluesc Maybe you can confirm in tomorrow if Jenkins is running stable for you. Based on your comment I have updated the components affected in this ticket and added 'groovy-plugin'.

          Mirko added a comment - Thanks 2bluesc  Maybe you can confirm in tomorrow if Jenkins is running stable for you. Based on your comment I have updated the components affected in this ticket and added 'groovy-plugin'.

          Tim Jacomb added a comment -

          I suspect it's https://github.com/jenkinsci/workflow-api-plugin/releases/tag/workflow-api-2.44, can anyone who is affected just roll back that plugin

          Tim Jacomb added a comment - I suspect it's https://github.com/jenkinsci/workflow-api-plugin/releases/tag/workflow-api-2.44 , can anyone who is affected just roll back that plugin

          ethorsa added a comment -

          I'll give it a try.

          ethorsa added a comment - I'll give it a try.

          kutzi added a comment -

          Also seeing this issue. Rolled back the workflow-api plugin to 2.42 and monitoring.
          Probably related to https://github.com/jenkinsci/workflow-api-plugin/pull/153 ?

          kutzi added a comment - Also seeing this issue. Rolled back the workflow-api plugin to 2.42 and monitoring. Probably related to https://github.com/jenkinsci/workflow-api-plugin/pull/153  ?

          kutzi added a comment - - edited

          Has been running now without deadlocks for a couple of hours with workflow-api plugin 2.42.
          Previously we had 2-3 deadlock within some minutes after a restart

          kutzi added a comment - - edited Has been running now without deadlocks for a couple of hours with workflow-api plugin 2.42. Previously we had 2-3 deadlock within some minutes after a restart

          Liam Newman added a comment -

          Reverted change that introduced this issue. Released in v2.45.

          Liam Newman added a comment - Reverted change that introduced this issue. Released in v2.45.

          ethorsa added a comment -

          No deadlocks since 2.45 .

          ethorsa added a comment - No deadlocks since 2.45 .

          Liam Newman added a comment -

          Released another attempt at fixing JENKINS-65821 in v2.46. Anyone watching this issue who is willing to upgrade sooner rather than later and provide feedback here or in JENKINS-65821, it would be much appreciated.

          Liam Newman added a comment - Released another attempt at fixing JENKINS-65821 in v2.46. Anyone watching this issue who is willing to upgrade sooner rather than later and provide feedback here or in JENKINS-65821 , it would be much appreciated.

          Phil McArdle added a comment -

          bitwiseman v2.46 seems to have fixed this for us, thank you

          We've seen no issues at all in around 24h.

          Phil McArdle added a comment - bitwiseman v2.46 seems to have fixed this for us, thank you We've seen no issues at all in around 24h.

            bitwiseman Liam Newman
            mpluhar Mirko
            Votes:
            5 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: