[JENKINS-53775] FileNotFoundException for program.dat when running a Pipeline Job concurrently with the Job DSL plugin - Jenkins Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Minor
Resolution: Fixed
Component/s: job-dsl-plugin, workflow-cps-plugin, workflow-job-plugin
Labels:
None
Environment:

Hide
Jenkins 2.138.1 LTS
workflow-aggregator:2.5
workflow-api:2.29
workflow-basic-steps:2.11
workflow-cps-global-lib:2.11
workflow-cps:2.54
workflow-durable-task-step:2.21
workflow-job:2.25
workflow-multibranch:2.20
workflow-scm-step:2.6
workflow-step-api:2.16
workflow-support:2.20
pipeline-build-step:2.7
pipeline-github-lib:1.0
pipeline-graph-analysis:1.7
pipeline-input-step:2.8
pipeline-milestone-step:1.3.1
pipeline-model-api:1.3.2
pipeline-model-declarative-agent:1.1.1
pipeline-model-definition:1.3.2
pipeline-model-extensions:1.3.2
pipeline-rest-api:2.10
pipeline-stage-step:2.3
pipeline-stage-tags-metadata:1.3.2
pipeline-stage-view:2.10
pipeline-utility-steps:2.1.0

Show
Jenkins 2.138.1 LTS workflow-aggregator:2.5 workflow-api:2.29 workflow-basic-steps:2.11 workflow-cps-global-lib:2.11 workflow-cps:2.54 workflow-durable-task-step:2.21 workflow-job:2.25 workflow-multibranch:2.20 workflow-scm-step:2.6 workflow-step-api:2.16 workflow-support:2.20 pipeline-build-step:2.7 pipeline-github-lib:1.0 pipeline-graph-analysis:1.7 pipeline-input-step:2.8 pipeline-milestone-step:1.3.1 pipeline-model-api:1.3.2 pipeline-model-declarative-agent:1.1.1 pipeline-model-definition:1.3.2 pipeline-model-extensions:1.3.2 pipeline-rest-api:2.10 pipeline-stage-step:2.3 pipeline-stage-tags-metadata:1.3.2 pipeline-stage-view:2.10 pipeline-utility-steps:2.1.0

Similar Issues:

Show
Released As:
job-dsl 1.76

Description

I unfortunately don't have an easy reproduer, but this bug happens to me fairly regularly (at least a few times a month). For this failure to occur:

A Freestyle job is running and performing a "Process Job DSLs" step. Part of this involves updating an existing Pipeline job.
That existing Pipeline job is also running (or starting) at around the same time as the "Process Job DLSs" step is trying to update it.

When the timing is just right, the Pipeline job fails with:

java.io.FileNotFoundException: /var/jenkins_home/jobs/devops-gate/jobs/projects/jobs/dx4linux/jobs/delphix-build-and-snapshots/jobs/ami-snapshots/builds/1173/program.dat (No such file or directory)
	at java.io.FileInputStream.open0(Native Method)
	at java.io.FileInputStream.open(FileInputStream.java:195)
	at java.io.FileInputStream.<init>(FileInputStream.java:138)
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader.openStreamAt(RiverReader.java:188)
	at org.jenkinsci.plugins.workflow.support.pickles.serialization.RiverReader.restorePickles(RiverReader.java:136)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.loadProgramAsync(CpsFlowExecution.java:773)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.onLoad(CpsFlowExecution.java:739)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.getExecution(WorkflowRun.java:875)
	at org.jenkinsci.plugins.workflow.job.WorkflowRun.onLoad(WorkflowRun.java:745)
	at hudson.model.RunMap.retrieve(RunMap.java:225)
	at hudson.model.RunMap.retrieve(RunMap.java:57)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:499)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.load(AbstractLazyLoadRunMap.java:481)
	at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:379)
	at jenkins.model.lazy.LazyBuildMixIn.getBuildByNumber(LazyBuildMixIn.java:231)
	at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:234)
	at org.jenkinsci.plugins.workflow.job.WorkflowJob.getBuildByNumber(WorkflowJob.java:105)
	at hudson.model.Run.fromExternalizableId(Run.java:2436)
	at org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper.getRawBuild(RunWrapper.java:71)
	at org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper.build(RunWrapper.java:75)
	at org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper.setResult(RunWrapper.java:87)
	at sun.reflect.GeneratedMethodAccessor820.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at groovy.lang.MetaClassImpl.setProperty(MetaClassImpl.java:2725)
	at groovy.lang.MetaClassImpl.setProperty(MetaClassImpl.java:3770)
	at org.codehaus.groovy.runtime.InvokerHelper.setProperty(InvokerHelper.java:201)
	at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.setProperty(ScriptBytecodeAdapter.java:484)
	at org.kohsuke.groovy.sandbox.impl.Checker$7.call(Checker.java:347)
	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onSetProperty(GroovyInterceptor.java:84)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onSetProperty(SandboxInterceptor.java:197)
	at org.kohsuke.groovy.sandbox.impl.Checker$7.call(Checker.java:344)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedSetProperty(Checker.java:351)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.setProperty(SandboxInvoker.java:33)
	at com.cloudbees.groovy.cps.impl.PropertyAccessBlock.rawSet(PropertyAccessBlock.java:24)
	at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.set(PropertyishBlock.java:88)
	at com.cloudbees.groovy.cps.impl.AssignmentBlock$ContinuationImpl.assignAndDone(AssignmentBlock.java:70)
	at sun.reflect.GeneratedMethodAccessor706.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
Caused: java.io.IOException: Failed to load build state
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:854)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$3.onSuccess(CpsFlowExecution.java:852)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$4$1.run(CpsFlowExecution.java:906)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$1.run(CpsVmExecutorService.java:35)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE
[Pipeline] stage
[Pipeline] { (Cloning repos)
[Pipeline] node
19:23:07 Running on jenkins-agent3 in /var/tmp/jenkins_slaves/jenkins-ops/workspace/devops-gate/projects/dx4linux/delphix-build-and-snapshots/ami-snapshots
[Pipeline] {
[Pipeline] checkout
19:23:07 Wiping out workspace first.
19:23:08 Cloning the remote Git repository
[...]
[Pipeline] // timestamps
[Pipeline] End of Pipeline
Finished: SUCCESS

Several things are unusual:

The job continues running, even after the Pipeline plugin printed "Finished: FAILURE."
If the job was running from a parent job, the parent job shows the job as failed (even though the child job is still running).
The job gets to the finally block of my pipeline and sends me a notification that it failed, however, the value of currentBuild.result is paradoxically SUCCESS, which leads me to believe that somehow the pipeline got to my finally block but not my catch block (which sets currentBuild.result to FAILURE on any exception). While the details of this behavior might be specific to my pipeline, the general behavior is very wonky.

Attachments

Issue Links

is related to

JENKINS-53359 job-dsl has deprecated concurrentBuild() on pipeline jobs

Closed

Activity

Ascending order - Click to sort in descending order

View 14 older comments

Nikita Bochenko added a comment - 2019-07-10 08:46

P.S.

Had a quick look and in many places we are using setConcurrentProperty which is not marked deprecated in docs. This is not related to DSL, however it seems to me that this is the same bug, just in a different context.

Nikita Bochenko added a comment - 2019-07-10 08:46 P.S. Had a quick look and in many places we are using setConcurrentProperty which is not marked deprecated in docs. This is not related to DSL, however it seems to me that this is the same bug, just in a different context.

Nikita Bochenko added a comment - 2019-07-10 09:10

P.P.S.

For freestyle jobs I have to set up {{concurrentBuild(true).}}I think by default freestyle jobs are not allowed concurrent execution. Not related to pipeline, of course, but feels inconsistent and may cause confusion?

Nikita Bochenko added a comment - 2019-07-10 09:10 P.P.S. For freestyle jobs I have to set up {{concurrentBuild(true).}}I think by default freestyle jobs are not allowed concurrent execution. Not related to pipeline, of course, but feels inconsistent and may cause confusion?

Basil Crow added a comment - 2019-07-10 15:59

I'm not surprised Throttle Concurrent Builds is on the scene of the crime here, but I don't want to jump to any conclusions yet without knowing more details about your configuration. It may or may not be related to the Pipeline race being described here. As an aside, if you're using Throttle Concurrent Builds with categories and Pipeline, you really should be using my patch from throttle-concurrent-builds-plugin#57, which improves CPU usage drastically and also has some correctness benefits. But that may or may not be related to this bug.

Basil Crow added a comment - 2019-07-10 15:59 I'm not surprised Throttle Concurrent Builds is on the scene of the crime here, but I don't want to jump to any conclusions yet without knowing more details about your configuration. It may or may not be related to the Pipeline race being described here. As an aside, if you're using Throttle Concurrent Builds with categories and Pipeline, you really should be using my patch from throttle-concurrent-builds-plugin#57 , which improves CPU usage drastically and also has some correctness benefits. But that may or may not be related to this bug.

Basil Crow added a comment - 2019-07-10 16:05

PS if you use that Throttle Concurrent Builds patch and find that it helps, please comment on the PR. I've been using it successfully for over 6 months and trying to get it merged/released (including asking for the privileges to merge/release it myself) but so far have received no response.

Basil Crow added a comment - 2019-07-10 16:05 PS if you use that Throttle Concurrent Builds patch and find that it helps, please comment on the PR. I've been using it successfully for over 6 months and trying to get it merged/released (including asking for the privileges to merge/release it myself) but so far have received no response.

Basil Crow added a comment - 2019-08-07 21:19

I've been running with the new syntax described in my previous comment for about a month, and this issue hasn't occurred again. Previously it occurred almost every time I deployed changes to a Job DSL pipeline.

Basil Crow added a comment - 2019-08-07 21:19 I've been running with the new syntax described in my previous comment for about a month, and this issue hasn't occurred again. Previously it occurred almost every time I deployed changes to a Job DSL pipeline.

Jenkins

FileNotFoundException for program.dat when running a Pipeline Job concurrently with the Job DSL plugin

Details

Description

Attachments

Issue Links

Activity

People

Dates