I could reproduce it now on my local machine:
_myjob_trunk #6 main build action completed: SUCCESS
…
SCM changes detected in myjob_trunk. Triggering #8
…
myjob_trunk #7 main build action completed: FAILURE
SCM changes detected in myjob_trunk. Triggering #8
…
Nov 17, 2015 12:27:23 PM SCHWERWIEGEND hudson.model.Executor run
Unexpected executor death
java.lang.IllegalStateException: D:\jenkins\jobs\myjob_trunk\builds\8 already existed; will not overwrite with myjob_trunk #8
at hudson.model.RunMap.put(RunMap.java:189)
at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
at hudson.model.AbstractProject.newBuild(AbstractProject.java:1019)
at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1209)
at hudson.model.AbstractProject.createExecutable(AbstractProject.java:145)
at hudson.model.Executor$1.call(Executor.java:362)
at hudson.model.Executor$1.call(Executor.java:344)
at hudson.model.Queue._withLock(Queue.java:1345)
at hudson.model.Queue.withLock(Queue.java:1210)
at hudson.model.Executor.run(Executor.java:344)
Nov 17, 2015 12:27:40 PM INFORMATION hudson.model.Run execute
…
Nov 17, 2015 12:28:33 PM INFORMATION hudson.model.Run execute
myjob_trunk #8 main build action completed: FAILURE
Nov 17, 2015 12:34:01 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
…
Nov 17, 2015 12:56:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
SCM changes detected in myjob_trunk. Triggering #9
Nov 17, 2015 12:56:01 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
…
Nov 17, 2015 1:02:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
SCM changes detected in myjob_trunk. Triggering #10
…
Nov 17, 2015 1:04:14 PM INFORMATION hudson.model.Run execute
myjob_trunk #9 main build action completed: FAILURE
Nov 17, 2015 1:04:29 PM INFORMATION hudson.model.Run execute
AbwertungsGrund #2 main build action completed: SUCCESS
Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Started initialization
Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Listed all plugins
Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Prepared all plugins
Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Started all plugins
Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Augmented all extensions
Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Loaded all jobs
Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Completed initialization
Nov 17, 2015 1:06:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
…
SCM changes detected in myjob_trunk. Triggering #10
Nov 17, 2015 1:06:02 PM INFORMATION hudson.model.Run execute
…
Nov 17, 2015 1:08:28 PM INFORMATION hudson.model.Run execute
myjob_trunk #10 main build action completed: FAILURE
Nov 17, 2015 1:09:23 PM SCHWERWIEGEND hudson.model.Executor run
Unexpected executor death
java.lang.IllegalStateException: D:\jenkins\jobs\myjob_trunk\builds\10 already existed; will not overwrite with myjob_trunk #10
at hudson.model.RunMap.put(RunMap.java:189)
at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
at hudson.model.AbstractProject.newBuild(AbstractProject.java:1019)
at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1209)
at hudson.model.AbstractProject.createExecutable(AbstractProject.java:145)
at hudson.model.Executor$1.call(Executor.java:362)
at hudson.model.Executor$1.call(Executor.java:344)
at hudson.model.Queue._withLock(Queue.java:1345)
at hudson.model.Queue.withLock(Queue.java:1210)
at hudson.model.Executor.run(Executor.java:344)
Nov 17, 2015 1:10:02 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
…
Nov 17, 2015 1:38:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
SCM changes detected in myjob_trunk. Triggering #11
Nov 17, 2015 1:38:02 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
…
Nov 17, 2015 1:44:47 PM INFORMATION hudson.model.Run execute
myjob_trunk #11 main build action completed: FAILURE
Nov 17, 2015 1:45:03 PM INFORMATION hudson.model.Run execute
AbwertungsGrund #3 main build action completed: SUCCESS
Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Started initialization
Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Listed all plugins
Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Prepared all plugins
Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Started all plugins
Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Augmented all extensions
Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Loaded all jobs
Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Completed initialization
Nov 17, 2015 1:45:03 PM INFORMATION org.springframework.web.context.support.StaticWebApplicationContext prepareRefresh
Refreshing org.springframework.web.context.support.StaticWebApplicationContext@7d591a4: display name [Root WebApplicationContext]; startup date [Tue Nov 17 13:45:03 CET 2015]; root of context hierarchy
Nov 17, 2015 1:45:03 PM INFORMATION org.springframework.web.context.support.StaticWebApplicationContext obtainFreshBeanFactory
Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@7d591a4]: org.springframework.beans.factory.support.DefaultListableBeanFactory@3aeb7f67
Nov 17, 2015 1:45:03 PM INFORMATION org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@3aeb7f67: defining beans [initialDirContextFactory,bindAuthenticator,ldapUserSearch,authenticationManager,authoritiesPopulator]; root of factory hierarchy
…
Nov 17, 2015 2:12:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
SCM changes detected in myjob_trunk. Triggering #12
Nov 17, 2015 2:12:02 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
…
Nov 17, 2015 2:16:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
SCM changes detected in myjob_trunk. Triggering #13
…
Nov 17, 2015 2:18:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
SCM changes detected in myjob_trunk. Triggering #13
Nov 17, 2015 2:18:02 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
…
Nov 17, 2015 2:19:30 PM INFORMATION hudson.model.Run execute
myjob_trunk #12 main build action completed: FAILURE
Nov 17, 2015 2:19:42 PM INFORMATION hudson.model.Run execute
AbwertungsGrund #4 main build action completed: SUCCESS
Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Started initialization
Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Listed all plugins
Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Prepared all plugins
Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Started all plugins
Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Augmented all extensions
Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Loaded all jobs
Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
Completed initialization
_
I'm utilizing some jobs (e.g. "AbwertungsGrund") to show the job's reason for not beeing blue
as a comment within the build-history,
As Tomasz Sniatowski mentioned those jobs/skript do "hudson.model.Hudson.instance.doReload()" which means "Reload Configuration from Disk" I suppose. I don't know any other way to force those comments to be seen. To evaluate if this is the cause for those "dead executors" I now removed "hudson.model.Hudson.instance.doReload()" from those scripts...
no more dead executors since then!!
On my old ci-System running with Jenkins 1.596.2 (and more or less the same jobs) I included "hudson.model.Hudson.instance.doReload()" yesterday. No dead executors either.
But within 1.596.2 "hudson.model.Hudson.instance.doReload()" lead to "Out-of-order builds".
("Timestamp inconsistent")
"hudson.model.Hudson.instance.doReload()" or "Reload(ing) configuration from disk" while there are jobs still running lead to those issues.
I was able to reproduce this issue by changing the nextBuildNumber on disk from a valid number to a number which already has an associating build.
Steps to reproduce:
1. Trigger multiple builds (5 builds)
2. Change the JENKINS_HOME/job/JOBNAME/nextBuildNumber to one of the build numbers with an associating build (like 1), or remove the nextBuildNumber file.
3. Restart Jenkins
4. The UI will show correct information, however when triggering a new build the executor will throw the same exception.
Only workaround I know of is to change the nextBuildNumber to a valid build number.