Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65885

Deadlock in CpsFlowExecution

    XMLWordPrintable

Details

    Description

      We are experiencing dead locks in Jenkins. I am not sure if its related but we recently upgraded to Jenkins 2.289.1. Since then we had two dead locks about 3 days apart. Jenkins does not shut down gracefully and does only stop when using kill -9.

      Found one Java-level deadlock:
=============================
"Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/ecommerceapps%252Fecommerce/branches/ from 127.0.0.1 : Jetty (winstone)-263503":
  waiting to lock monitor 0x00007f0bc0186788 (object 0x000000046cb7ddc8, a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution),
  which is held by "Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]"
"Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]":
  waiting to lock monitor 0x00007f0b9c01ed18 (object 0x000000046cb7de38, a org.jenkinsci.plugins.workflow.graph.StandardGraphLookupView),
  which is held by "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/pipelines/ecommerceapps%2Fecommerce/branches/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/runs/17/ from 10.120.1.90 : Jetty (winstone)-72660"
"Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/pipelines/ecommerceapps%2Fecommerce/branches/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/runs/17/ from 10.120.1.90 : Jetty (winstone)-72660":
  waiting to lock monitor 0x00007f0bc0186788 (object 0x000000046cb7ddc8, a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution),
  which is held by "Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]"

       

       

      Java stack information for the threads listed above:
===================================================
"Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/ecommerceapps%252Fecommerce/branches/ from 127.0.0.1 : Jetty (winstone)-263503":
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.getCurrentHeads(CpsFlowExecution.java:981)
        - waiting to lock <0x000000046cb7ddc8> (a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution)
        at org.jenkinsci.plugins.workflow.flow.FlowExecution.isComplete(FlowExecution.java:208)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.isComplete(CpsFlowExecution.java:1238)

      Attachments

        Issue Links

          is caused by

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-65821 Prevent some race condition in workflow-api

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-65901 Thread deadlock since workflow-api-plugin-2.44

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              bitwiseman Liam Newman
              mpluhar Mirko
              Votes:
              5 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: