[JENKINS-59083] Builds stuck on "is already in progress" forever

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: workflow-job-plugin
Labels:
- regression
Environment:
Centos 7.2.1511 (master and slaves)
Jenkins 2.176.2
Jenkins master - open-jdk 1.8.0_212
Jenkins slaves - Oracle jdk 1.8.0_60

Jenkins running as docker container (standalone)
Attached plugin versions as file

Similar Issues:
Powered by SuggestiMate

Show
Released As:
workflow-job 2.35, workflow-cps 2.74

After upgrading our jenkins master and plugins, some of our build jobs - specifically it seemed like most pipeline jobs - were stuck waiting for previous build to complete, even though it already completed.

This seems to have been caused by ~~JENKINS-46076~~.

After donwngrading the plugin to version 2.32, all builds were working correctly.

From some testing, it seems that calling

Jenkins.instance.getItemByFullName(...).getBuildByNumber(..).isLogUpdated()

On the previous build (in the sample case build #185) returns true, although the build is completed and the log seems to have been completed.

This issue happened immediately after restarting the jenkins instance with any of our pipeline jobs

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

Capture.PNG
70 kB
2019-08-26 20:13
Capture2.PNG
66 kB
2019-08-26 21:32
failed.PNG
70 kB
2019-08-26 21:44
image.png
19 kB
2019-08-26 14:47
plugins.txt
5 kB
2019-08-26 12:53
stuck-build-2.tar.gz
16 kB
2019-08-26 21:49
tdump
75 kB
2019-08-26 20:17

causes

JENKINS-61604 Pipeline deadlock caused by fix for JENKINS-59083

Resolved

is duplicated by

JENKINS-56947 File handle leak in EnvActionImpl when accessing environment of completed builds

Closed

is related to

JENKINS-46076 Test flake in CpsThreadTest.stop

Resolved

relates to

JENKINS-59328 job hangs although failed after ConcurrentModificationException of ExwsAllocateActionImpl

Open

links to

jenkinsci/workflow-cps-plugin #317

workflow-job #137

workflow-job #140

(2 links to)

Roy Arnon added a comment - 2019-08-26 21:49

I went ahead and reproduced it again.

This time I simulated a failure in the pipeline job - you can see build #1916 waiting for the failed build #1915.

I have attached the builds folder again - stuck-build-2.tar.gz

Roy Arnon added a comment - 2019-08-26 21:49 I went ahead and reproduced it again. This time I simulated a failure in the pipeline job - you can see build #1916 waiting for the failed build #1915. I have attached the builds folder again - stuck-build-2.tar.gz

Jesse Glick added a comment - 2019-08-26 22:50

I do not get what you mean nothing problematic in the screenshot.

Sorry, I meant nothing problematic other than the new build waiting, which is probably just due to the incorrect return value from isLogUpdated mentioned to begin with. Everything else looks normal enough.

each build was kicking off downstream jobs

A plausible lead, though apparently not the explanation for hellspam.

Jesse Glick added a comment - 2019-08-26 22:50 I do not get what you mean nothing problematic in the screenshot. Sorry, I meant nothing problematic other than the new build waiting, which is probably just due to the incorrect return value from isLogUpdated mentioned to begin with. Everything else looks normal enough. each build was kicking off downstream jobs A plausible lead, though apparently not the explanation for hellspam .

Jesse Glick added a comment - 2019-08-26 23:01 - edited

Anyone affected should please try upgrading to this build. Purely speculative, but in the absence of any known steps to reproduce that is the best we can do.

Jesse Glick added a comment - 2019-08-26 23:01 - edited Anyone affected should please try upgrading to this build . Purely speculative, but in the absence of any known steps to reproduce that is the best we can do.

Roy Arnon added a comment - 2019-08-27 18:24

Hi,

I can confirm that the build you've uploaded fixes the issue for us. Thanks!

Roy Arnon added a comment - 2019-08-27 18:24 Hi, I can confirm that the build you've uploaded fixes the issue for us. Thanks!

Jesse Glick added a comment - 2019-08-27 18:53

hellspam good; now do you see any stack traces in your system log starting with

java.lang.IllegalStateException: trying to open a build log on … after it has completed

? If so, we need to see the stack trace to try to understand the root cause.

Jesse Glick added a comment - 2019-08-27 18:53 hellspam good; now do you see any stack traces in your system log starting with java.lang.IllegalStateException: trying to open a build log on … after it has completed ? If so, we need to see the stack trace to try to understand the root cause.

Roy Arnon added a comment - 2019-08-27 19:50

Yes, I do see a stack trace like that in the log:

Aug 27, 2019 7:37:09 PM org.jenkinsci.plugins.workflow.job.WorkflowRun getListener
WARNING: null
java.lang.IllegalStateException: trying to open a build log on Lint.Hooks #1923 after it has completed
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:219)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$300(WorkflowRun.java:133)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.getListener(WorkflowRun.java:955)
        at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getListener(EnvActionImpl.java:77)
        at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:69)
        at org.jenkinsci.plugin.taboola.ElasticSearchManager.getConsoleEnvs(ElasticSearchManager.java:277)
        at org.jenkinsci.plugin.taboola.ElasticSearchManager.addCommonDataAndSend(ElasticSearchManager.java:110)
        at org.jenkinsci.plugin.taboola.ElasticSearchManager.sendStageData(ElasticSearchManager.java:179)
        at org.jenkinsci.plugin.taboola.events.TaboolaStagesListener.onNewHead(TaboolaStagesListener.java:145)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:37)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Roy Arnon added a comment - 2019-08-27 19:50 Yes, I do see a stack trace like that in the log: Aug 27, 2019 7:37:09 PM org.jenkinsci.plugins.workflow.job.WorkflowRun getListener WARNING: null java.lang.IllegalStateException: trying to open a build log on Lint.Hooks #1923 after it has completed at org.jenkinsci.plugins.workflow.job.WorkflowRun.getListener(WorkflowRun.java:219) at org.jenkinsci.plugins.workflow.job.WorkflowRun.access$300(WorkflowRun.java:133) at org.jenkinsci.plugins.workflow.job.WorkflowRun$Owner.getListener(WorkflowRun.java:955) at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getListener(EnvActionImpl.java:77) at org.jenkinsci.plugins.workflow.cps.EnvActionImpl.getEnvironment(EnvActionImpl.java:69) at org.jenkinsci.plugin.taboola.ElasticSearchManager.getConsoleEnvs(ElasticSearchManager.java:277) at org.jenkinsci.plugin.taboola.ElasticSearchManager.addCommonDataAndSend(ElasticSearchManager.java:110) at org.jenkinsci.plugin.taboola.ElasticSearchManager.sendStageData(ElasticSearchManager.java:179) at org.jenkinsci.plugin.taboola.events.TaboolaStagesListener.onNewHead(TaboolaStagesListener.java:145) at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.notifyListeners(CpsFlowExecution.java:1463) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$3.run(CpsThreadGroup.java:458) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:37) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)

Roy Arnon added a comment - 2019-08-27 19:51

Actually, taking a look at it, it seems it has to do with one of our own plugins - I will look into that.

Roy Arnon added a comment - 2019-08-27 19:51 Actually, taking a look at it, it seems it has to do with one of our own plugins - I will look into that.

Jesse Glick added a comment - 2019-08-27 20:25

workflow-job #140 should suppress the stack trace, which at least in this case seems to be innocuous.

Jesse Glick added a comment - 2019-08-27 20:25 workflow-job #140 should suppress the stack trace, which at least in this case seems to be innocuous.

Devin Nusbaum added a comment - 2019-09-10 20:17 - edited

Ok, at least one way that this could be happening should be fixed as of Pipeline: Job Plugin 2.35. If you end up finding this ticket when running Pipeline: Job Plugin 2.35 or higher because you are seeing the following error in your logs, please file a new ticket, and include the full stack trace:

java.lang.IllegalStateException: trying to open a build log on … after it has completed

EDIT: Note that if you are not running Pipeline: Groovy Plugin 2.74 or later, you will see this error message for one case that has already been fixed, so I would recommend upgrading to Pipeline: Groovy Plugin 2.74 or later.

Devin Nusbaum added a comment - 2019-09-10 20:17 - edited Ok, at least one way that this could be happening should be fixed as of Pipeline: Job Plugin 2.35. If you end up finding this ticket when running Pipeline: Job Plugin 2.35 or higher because you are seeing the following error in your logs, please file a new ticket, and include the full stack trace: java.lang.IllegalStateException: trying to open a build log on … after it has completed EDIT: Note that if you are not running Pipeline: Groovy Plugin 2.74 or later, you will see this error message for one case that has already been fixed, so I would recommend upgrading to Pipeline: Groovy Plugin 2.74 or later.

Jesse Glick added a comment - 2019-09-10 21:44

upgrading to Pipeline: Job Plugin 2.35 or newer which will automatically update Pipeline: Groovy Plugin as well

No it will not—this is just a test dependency.

Plugin-Dependencies: workflow-api:2.36,workflow-step-api:2.20,workflow-support:3.3

Jesse Glick added a comment - 2019-09-10 21:44 upgrading to Pipeline: Job Plugin 2.35 or newer which will automatically update Pipeline: Groovy Plugin as well No it will not—this is just a test dependency. Plugin-Dependencies: workflow-api:2.36,workflow-step-api:2.20,workflow-support:3.3

Assignee:: Jesse Glick

Reporter:: Roy Arnon

Votes:: 0 Vote for this issue

Watchers:: 5 Start watching this issue

Created:: 2019-08-26 12:58

Updated:: 2020-08-19 18:09

Resolved:: 2019-09-10 20:17

Jenkins

Details

Description

Attachments

Attachments

Issue Links

Activity

Collapse comment: Roy Arnon added a comment - 2019-08-26 21:49

Expand comment: Roy Arnon added a comment - 2019-08-26 21:49

Collapse comment: Jesse Glick added a comment - 2019-08-26 22:50

Expand comment: Jesse Glick added a comment - 2019-08-26 22:50

Collapse comment: Jesse Glick added a comment - 2019-08-26 23:01, Edited by Jesse Glick - 2019-08-27 14:18

Expand comment: Jesse Glick added a comment - 2019-08-26 23:01, Edited by Jesse Glick - 2019-08-27 14:18

Collapse comment: Roy Arnon added a comment - 2019-08-27 18:24

Expand comment: Roy Arnon added a comment - 2019-08-27 18:24

Collapse comment: Jesse Glick added a comment - 2019-08-27 18:53

Expand comment: Jesse Glick added a comment - 2019-08-27 18:53

Collapse comment: Roy Arnon added a comment - 2019-08-27 19:50

Expand comment: Roy Arnon added a comment - 2019-08-27 19:50

Collapse comment: Roy Arnon added a comment - 2019-08-27 19:51

Expand comment: Roy Arnon added a comment - 2019-08-27 19:51

Collapse comment: Jesse Glick added a comment - 2019-08-27 20:25

Expand comment: Jesse Glick added a comment - 2019-08-27 20:25

Collapse comment: Devin Nusbaum added a comment - 2019-09-10 20:17, Edited by Devin Nusbaum - 2019-09-13 15:38

Expand comment: Devin Nusbaum added a comment - 2019-09-10 20:17, Edited by Devin Nusbaum - 2019-09-13 15:38

Collapse comment: Jesse Glick added a comment - 2019-09-10 21:44

Expand comment: Jesse Glick added a comment - 2019-09-10 21:44

People

Dates