Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65885

Deadlock in CpsFlowExecution

    XMLWordPrintable

Details

    Description

      We are experiencing dead locks in Jenkins. I am not sure if its related but we recently upgraded to Jenkins 2.289.1. Since then we had two dead locks about 3 days apart. Jenkins does not shut down gracefully and does only stop when using kill -9.

      Found one Java-level deadlock:
      =============================
      "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/ecommerceapps%252Fecommerce/branches/ from 127.0.0.1 : Jetty (winstone)-263503":
        waiting to lock monitor 0x00007f0bc0186788 (object 0x000000046cb7ddc8, a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution),
        which is held by "Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]"
      "Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]":
        waiting to lock monitor 0x00007f0b9c01ed18 (object 0x000000046cb7de38, a org.jenkinsci.plugins.workflow.graph.StandardGraphLookupView),
        which is held by "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/pipelines/ecommerceapps%2Fecommerce/branches/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/runs/17/ from 10.120.1.90 : Jetty (winstone)-72660"
      "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/pipelines/ecommerceapps%2Fecommerce/branches/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/runs/17/ from 10.120.1.90 : Jetty (winstone)-72660":
        waiting to lock monitor 0x00007f0bc0186788 (object 0x000000046cb7ddc8, a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution),
        which is held by "Running CpsFlowExecution[Owner[acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test/17:acme-gitlab/ecommerceapps%2Fecommerce/PLAT-475_Set_up_S3_bucket_for_xyz_replacement_on_test #17]]" 

       

       

      Java stack information for the threads listed above:
      ===================================================
      "Handling GET /blue/rest/organizations/jenkins/pipelines/acme-gitlab/ecommerceapps%252Fecommerce/branches/ from 127.0.0.1 : Jetty (winstone)-263503":
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.getCurrentHeads(CpsFlowExecution.java:981)
              - waiting to lock <0x000000046cb7ddc8> (a org.jenkinsci.plugins.workflow.cps.CpsFlowExecution)
              at org.jenkinsci.plugins.workflow.flow.FlowExecution.isComplete(FlowExecution.java:208)
              at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.isComplete(CpsFlowExecution.java:1238) 

      Attachments

        Issue Links

          Activity

            kutzi kutzi added a comment - - edited

            Has been running now without deadlocks for a couple of hours with workflow-api plugin 2.42.
            Previously we had 2-3 deadlock within some minutes after a restart

            kutzi kutzi added a comment - - edited Has been running now without deadlocks for a couple of hours with workflow-api plugin 2.42. Previously we had 2-3 deadlock within some minutes after a restart
            bitwiseman Liam Newman added a comment -

            Reverted change that introduced this issue. Released in v2.45.

            bitwiseman Liam Newman added a comment - Reverted change that introduced this issue. Released in v2.45.
            ethorsa ethorsa added a comment -

            No deadlocks since 2.45 .

            ethorsa ethorsa added a comment - No deadlocks since 2.45 .
            bitwiseman Liam Newman added a comment -

            Released another attempt at fixing JENKINS-65821 in v2.46. Anyone watching this issue who is willing to upgrade sooner rather than later and provide feedback here or in JENKINS-65821, it would be much appreciated.

            bitwiseman Liam Newman added a comment - Released another attempt at fixing JENKINS-65821 in v2.46. Anyone watching this issue who is willing to upgrade sooner rather than later and provide feedback here or in JENKINS-65821 , it would be much appreciated.
            philmcardlecg Phil McArdle added a comment -

            bitwiseman v2.46 seems to have fixed this for us, thank you

            We've seen no issues at all in around 24h.

            philmcardlecg Phil McArdle added a comment - bitwiseman v2.46 seems to have fixed this for us, thank you We've seen no issues at all in around 24h.

            People

              bitwiseman Liam Newman
              mpluhar Mirko
              Votes:
              5 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: