Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65885

Deadlock in CpsFlowExecution

    XMLWordPrintable

Details

    Description

      We are experiencing dead locks in Jenkins. I am not sure if its related but we recently upgraded to Jenkins 2.289.1. Since then we had two dead locks about 3 days apart. Jenkins does not shut down gracefully and does only stop when using kill -9.

      Found one Java-level deadlock:
=============================
"Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/ecommerceapps%252Fecommerce/branches/ from 127.0.0.1 : Jetty (winstone)-263503":
  waiting to lock monitor 0x00007f0bc0186788 (object 0x000000046cb7ddc8, a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution),
  which is held by "Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]"
"Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]":
  waiting to lock monitor 0x00007f0b9c01ed18 (object 0x000000046cb7de38, a org.jenkinsci.plugins.workflow.graph.StandardGraphLookupView),
  which is held by "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/pipelines/ecommerceapps%2Fecommerce/branches/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/runs/17/ from 10.120.1.90 : Jetty (winstone)-72660"
"Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/pipelines/ecommerceapps%2Fecommerce/branches/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/runs/17/ from 10.120.1.90 : Jetty (winstone)-72660":
  waiting to lock monitor 0x00007f0bc0186788 (object 0x000000046cb7ddc8, a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution),
  which is held by "Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]"

       

       

      Java stack information for the threads listed above:
===================================================
"Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/ecommerceapps%252Fecommerce/branches/ from 127.0.0.1 : Jetty (winstone)-263503":
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.getCurrentHeads(CpsFlowExecution.java:981)
        - waiting to lock <0x000000046cb7ddc8> (a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution)
        at org.jenkinsci.plugins.workflow.flow.FlowExecution.isComplete(FlowExecution.java:208)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.isComplete(CpsFlowExecution.java:1238)

      Attachments

        Issue Links

          is caused by

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-65821 Prevent some race condition in workflow-api

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-65901 Thread deadlock since workflow-api-plugin-2.44

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            View 10 older comments
            kutzi kutzi added a comment - - edited

            Has been running now without deadlocks for a couple of hours with workflow-api plugin 2.42.
            Previously we had 2-3 deadlock within some minutes after a restart

            kutzi kutzi added a comment - - edited Has been running now without deadlocks for a couple of hours with workflow-api plugin 2.42. Previously we had 2-3 deadlock within some minutes after a restart
            bitwiseman Liam Newman added a comment -

            Reverted change that introduced this issue. Released in v2.45.

            bitwiseman Liam Newman added a comment - Reverted change that introduced this issue. Released in v2.45.
            ethorsa ethorsa added a comment -

            No deadlocks since 2.45 .

            ethorsa ethorsa added a comment - No deadlocks since 2.45 .
            bitwiseman Liam Newman added a comment -

            Released another attempt at fixing JENKINS-65821 in v2.46. Anyone watching this issue who is willing to upgrade sooner rather than later and provide feedback here or in JENKINS-65821, it would be much appreciated.

            bitwiseman Liam Newman added a comment - Released another attempt at fixing JENKINS-65821 in v2.46. Anyone watching this issue who is willing to upgrade sooner rather than later and provide feedback here or in JENKINS-65821 , it would be much appreciated.
            philmcardlecg Phil McArdle added a comment -

            bitwiseman v2.46 seems to have fixed this for us, thank you

            We've seen no issues at all in around 24h.

            philmcardlecg Phil McArdle added a comment - bitwiseman v2.46 seems to have fixed this for us, thank you We've seen no issues at all in around 24h.

            People

              bitwiseman Liam Newman
              mpluhar Mirko
              Votes:
              5 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: