-
Bug
-
Resolution: Fixed
-
Blocker
-
openjdk version "1.8.0_292"
Jenkins 2.289.1
Blue Ocean 1.24.7
We are experiencing dead locks in Jenkins. I am not sure if its related but we recently upgraded to Jenkins 2.289.1. Since then we had two dead locks about 3 days apart. Jenkins does not shut down gracefully and does only stop when using kill -9.
Found one Java-level deadlock: ============================= "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/ecommerceapps%252Fecommerce/branches/ from 127.0.0.1 : Jetty (winstone)-263503": waiting to lock monitor 0x00007f0bc0186788 (object 0x000000046cb7ddc8, a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution), which is held by "Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]" "Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]": waiting to lock monitor 0x00007f0b9c01ed18 (object 0x000000046cb7de38, a org.jenkinsci.plugins.workflow.graph.StandardGraphLookupView), which is held by "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/pipelines/ecommerceapps%2Fecommerce/branches/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/runs/17/ from 10.120.1.90 : Jetty (winstone)-72660" "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/pipelines/ecommerceapps%2Fecommerce/branches/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/runs/17/ from 10.120.1.90 : Jetty (winstone)-72660": waiting to lock monitor 0x00007f0bc0186788 (object 0x000000046cb7ddc8, a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution), which is held by "Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]"
Java stack information for the threads listed above: =================================================== "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/ecommerceapps%252Fecommerce/branches/ from 127.0.0.1 : Jetty (winstone)-263503": at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.getCurrentHeads(CpsFlowExecution.java:981) - waiting to lock <0x000000046cb7ddc8> (a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution) at org.jenkinsci.plugins.workflow.flow.FlowExecution.isComplete(FlowExecution.java:208) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.isComplete(CpsFlowExecution.java:1238)
- is caused by
-
JENKINS-65821 Prevent some race condition in workflow-api
-
- Resolved
-
- relates to
-
JENKINS-65901 Thread deadlock since workflow-api-plugin-2.44
-
- Closed
-
Was able to workaround the issue by downgrading a number of packages necessary to roll back the Groovy plugin v2.92 -> v2.90 which is where I suspect the issue is.
Here's the working package diff still on Jenkins LTS v2.289.1
The HTTP Request Plugin and Subversion Plug-in are likely extraneous changes from earlier debugging and unapplied updates.
Not familiar enough with Jenkins and plugins to offer much assistance in patching it, but hopefully this helps narrow it down and resolve issues for others encountering the same issue.