svanoort, does this seem similar to other issues you've investigated where jobs mysteriously resume after they've "completed"?

Hit this again today. This has become an annoyance for us, since when we push a change to a Job DSL job, we frequently see this behavior on any running instances of that job.

jtaboada Could I persuade you to please take a look at this? It looks like it should be a synchronization bug around pickle loading. I'm not actually as concerned with JobDSL + Pipeline because that's kind of an edge case situation, but I strongly suspect that a threading issue here will be causing other failures that people just don't know the cause of. Thanks!

jtaboada Could you please provide some details? There should not be a race condition possible (if so, that's a bugfix we need to do). It sounds like you have what you need to fix that, from what you're saying. Thanks!

Also seeing this error pop up fairly consistently when a number of PR builds all fire at the same time on a multibranch job.  After the error, the jobs all stick around until the master is rebooted.

Hit this again today.

I'm not actually as concerned with JobDSL + Pipeline because that's kind of an edge case situation

svanoort, can you explain why you consider this an edge case situation? Is it because you'd rather that users use a Jenkinsfile with the Pipeline Multibranch Plugin rather than Job DSL? I'd love to do that, but I can't. My organization centralizes company-wide build logic in one team, and one repository within that team. Due to JENKINS-43749, the Pipeline Multibranch Plugin doesn't work for my use case. In that bug Jesse recommended the use of Job DSL. So that is what I am doing. Is this an edge case situation? I'd be happy to move to a different setup if one was available, but I don't see any other options, I'm afraid. And this bug continues to affect us.

Hit this again today.

Constantly running into the same issue. Using both Job DSL+Pipeline and Pipeline directly to generate new jobs or update their configuration. Have no idea how to reproduce it as it seems very random.

It seems if I reload configuration from disk, the error goes away for some time.

mbochenk and therealwaldo, I made an interesting observation today. I've only seen this problem on jobs where Job DSL was setting concurrentBuild false. I've never seen the problem on jobs where concurrentBuild is set to true. Does this match your experience? It seems too uncanny to be a coincidence. And whenever I hit this bug, the two jobs seem to in fact be running concurrently, which "should" never happen because concurrentBuild is set to false. I wonder if this could actually be a Jenkins core issue, where two jobs are scheduled to run concurrently (even though they shouldn't be) and later trample on each other.

I verified my understanding of the above by checking config.xml after Job DSL had generated the job with concurrentBuild false. It had the <concurrentBuild>false</concurrentBuild> attribute. I then checked how this had been deserialized in the Script Console:

println job.@concurrentBuild
println job.concurrentBuild

This printed null and false, which was what I expected. The field had been set to null, and the getter was instead relying on the property. Next, I called job.save() from the Script Console and checked the contents of config.xml again. Now, <concurrentBuild>false</concurrentBuild> was gone. In its place, in the properties section was <org.jenkinsci.plugins.workflow.job.properties.DisableConcurrentBuildsJobProperty/>.

Based on the above, I think the fix should be for Job DSL to generate the property rather than setting the deprecated concurrentBuild field. In other words, there seems to be a race in converting the deprecated field to the new-style property. If Job DSL simply used the new-style property in the first place, we would avoid the race and therefore we wouldn't put Pipeline in a pathological state that ultimate results in a FileNotFoundException.

daspilker, do you have any concerns with this approach? If not, I will prepare a PR.

Thinking about this some more, the fix might be even simpler. The Dynamic DSL already supports the new-style property. So all that we need to do in Job DSL is deprecated concurrentBuilds for Pipeline jobs and encourage users to migrate to the new property via the Dynamic DSL. In other words, users should convert this syntax …

concurrentBuild false

… to this syntax …

properties {
  disableConcurrentBuilds()
}

I confirmed that this generates the new-style XML and that the "Do not allow concurrent builds" was checked when I viewed the generated job in the Jenkins UI.

basil thank you for extensive investigation. It seems highly plausible. I did observe multiple jobs running where there should not be, or a job stuck in running state although it has been completed. This also could explain why sometimes reload configuration helps to find "missing" jobs that are stuck in this state.

We do generate some jobs via DSL and some directly from the scripted pipeline, so I'd need to check how concurrentBuild is set up there, I suspect it is also using ye olde way of setting concurrent builds values.

One additional observation: it also happens on the jobs that are using Throttle Concurrent Builds plugin - the cause could be a similar one. Basically I just set something like these:

throttleConcurrentBuilds {
  categories(['some-category'])
}

throttleConcurrentBuilds {   
  maxPerNode(2)
  maxTotal(8)
}

I need to clarify: I believe we are explicitly setting concurrentBuild to either true or false for most, if not all, builds. Even the ones with throttleConcurrentBuilds - so it might not be related to the plugin itself. I will investigate this when I get some time - right now I am working on some projects that do not give me enough time to investigate this deeper.

P.S.

Had a quick look and in many places we are using setConcurrentProperty which is not marked deprecated in docs. This is not related to DSL, however it seems to me that this is the same bug, just in a different context.

P.P.S.

For freestyle jobs I have to set up {{concurrentBuild(true).}}I think by default freestyle jobs are not allowed concurrent execution. Not related to pipeline, of course, but feels inconsistent and may cause confusion?

I'm not surprised Throttle Concurrent Builds is on the scene of the crime here, but I don't want to jump to any conclusions yet without knowing more details about your configuration. It may or may not be related to the Pipeline race being described here. As an aside, if you're using Throttle Concurrent Builds with categories and Pipeline, you really should be using my patch from throttle-concurrent-builds-plugin#57, which improves CPU usage drastically and also has some correctness benefits. But that may or may not be related to this bug.

PS if you use that Throttle Concurrent Builds patch and find that it helps, please comment on the PR. I've been using it successfully for over 6 months and trying to get it merged/released (including asking for the privileges to merge/release it myself) but so far have received no response.

I've been running with the new syntax described in my previous comment for about a month, and this issue hasn't occurred again. Previously it occurred almost every time I deployed changes to a Job DSL pipeline.