Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59083

Builds stuck on "is already in progress" forever

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed
    • workflow-job-plugin
    • Centos 7.2.1511 (master and slaves)
      Jenkins 2.176.2
      Jenkins master - open-jdk 1.8.0_212
      Jenkins slaves - Oracle jdk 1.8.0_60

      Jenkins running as docker container (standalone)
      Attached plugin versions as file

    • workflow-job 2.35, workflow-cps 2.74

    Description

      After upgrading our jenkins master and plugins, some of our build jobs - specifically it seemed like most pipeline jobs - were stuck waiting for previous build to complete, even though it already completed.

       

      This seems to have been caused by JENKINS-46076.

      After donwngrading the plugin to version 2.32, all builds were working correctly.

       

      From some testing, it seems that calling

      Jenkins.instance.getItemByFullName(...).getBuildByNumber(..).isLogUpdated()

      On the previous build (in the sample case build #185) returns true, although the build is completed and the log seems to have been completed.

      This issue happened immediately after restarting the jenkins instance with any of our pipeline jobs

      Attachments

        1. Capture.PNG
          Capture.PNG
          70 kB
        2. Capture2.PNG
          Capture2.PNG
          66 kB
        3. failed.PNG
          failed.PNG
          70 kB
        4. image.png
          image.png
          19 kB
        5. plugins.txt
          5 kB
        6. stuck-build-2.tar.gz
          16 kB
        7. tdump
          75 kB

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-61604 Pipeline deadlock caused by fix for JENKINS-59083

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-56947 File handle leak in EnvActionImpl when accessing environment of completed builds

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-46076 Test flake in CpsThreadTest.stop

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-59328 job hangs although failed after ConcurrentModificationException of ExwsAllocateActionImpl

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open
          links to

          Web Link jenkinsci/workflow-cps-plugin #317

          Web Link workflow-job #137

          Web Link workflow-job #140

          Activity

            Ascending order - Click to sort in descending order
            View 14 older comments
            hellspam Roy Arnon added a comment -

            Yes, I do see a stack trace like that in the log:

            Aug 27, 2019 7:37:09 PM org.jenkinsci.plugins.workflow.job.WorkflowRun getListener
WARNING: null
java.lang.IllegalStateException: trying to open a build log on Lint.Hooks #1923 after it has completed
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:219)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$300(WorkflowRun.java:133)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.getListener(WorkflowRun.java:955)
        at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getListener(EnvActionImpl.java:77)
        at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:69)
        at org.jenkinsci.plugin.taboola.ElasticSearchManager.getConsoleEnvs(ElasticSearchManager.java:277)
        at org.jenkinsci.plugin.taboola.ElasticSearchManager.addCommonDataAndSend(ElasticSearchManager.java:110)
        at org.jenkinsci.plugin.taboola.ElasticSearchManager.sendStageData(ElasticSearchManager.java:179)
        at org.jenkinsci.plugin.taboola.events.TaboolaStagesListener.onNewHead(TaboolaStagesListener.java:145)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:37)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
            hellspam Roy Arnon added a comment - Yes, I do see a stack trace like that in the log: Aug 27, 2019 7:37:09 PM org.jenkinsci.plugins.workflow.job.WorkflowRun getListener WARNING: null java.lang.IllegalStateException: trying to open a build log on Lint.Hooks #1923 after it has completed at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:219) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$300(WorkflowRun.java:133) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.getListener(WorkflowRun.java:955) at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getListener(EnvActionImpl.java:77) at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:69) at org.jenkinsci.plugin.taboola.ElasticSearchManager.getConsoleEnvs(ElasticSearchManager.java:277) at org.jenkinsci.plugin.taboola.ElasticSearchManager.addCommonDataAndSend(ElasticSearchManager.java:110) at org.jenkinsci.plugin.taboola.ElasticSearchManager.sendStageData(ElasticSearchManager.java:179) at org.jenkinsci.plugin.taboola.events.TaboolaStagesListener.onNewHead(TaboolaStagesListener.java:145) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:37) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)
            hellspam Roy Arnon added a comment -

            Actually, taking a look at it, it seems it has to do with one of our own plugins - I will look into that.

            hellspam Roy Arnon added a comment - Actually, taking a look at it, it seems it has to do with one of our own plugins - I will look into that.
            jglick Jesse Glick added a comment -

            workflow-job #140 should suppress the stack trace, which at least in this case seems to be innocuous.

            jglick Jesse Glick added a comment - workflow-job #140 should suppress the stack trace, which at least in this case seems to be innocuous.
            dnusbaum Devin Nusbaum added a comment - - edited

            Ok, at least one way that this could be happening should be fixed as of Pipeline: Job Plugin 2.35. If you end up finding this ticket when running Pipeline: Job Plugin 2.35 or higher because you are seeing the following error in your logs, please file a new ticket, and include the full stack trace:

            java.lang.IllegalStateException: trying to open a build log on … after it has completed

            EDIT: Note that if you are not running Pipeline: Groovy Plugin 2.74 or later, you will see this error message for one case that has already been fixed, so I would recommend upgrading to Pipeline: Groovy Plugin 2.74 or later.

            dnusbaum Devin Nusbaum added a comment - - edited Ok, at least one way that this could be happening should be fixed as of Pipeline: Job Plugin 2.35. If you end up finding this ticket when running Pipeline: Job Plugin 2.35 or higher because you are seeing the following error in your logs, please file a new ticket, and include the full stack trace: java.lang.IllegalStateException: trying to open a build log on … after it has completed EDIT: Note that if you are not running Pipeline: Groovy Plugin 2.74 or later, you will see this error message for one case that has already been fixed, so I would recommend upgrading to Pipeline: Groovy Plugin 2.74 or later.
            jglick Jesse Glick added a comment -

            upgrading to Pipeline: Job Plugin 2.35 or newer which will automatically update Pipeline: Groovy Plugin as well

            No it will not—this is just a test dependency.

            Plugin-Dependencies: workflow-api:2.36,workflow-step-api:2.20,workflow-support:3.3
            jglick Jesse Glick added a comment - upgrading to Pipeline: Job Plugin 2.35 or newer which will automatically update Pipeline: Groovy Plugin as well No it will not—this is just a test dependency. Plugin-Dependencies: workflow-api:2.36,workflow-step-api:2.20,workflow-support:3.3

            People

              jglick Jesse Glick
              hellspam Roy Arnon
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: