• Issue is seen after upgrading to 1.609.1 from 1.596.2.
• Issue is seen on freestyle projects (fixes in https://issues.jenkins-ci.org/browse/JENKINS-26582 relate to matrix project)
• Issue maybe related to https://issues.jenkins-ci.org/browse/JENKINS-29268

Issue:

• For a few slaves, the threads go dead.
• These dead threads seem to be caused by builds on specific jobs.
• In these cases, the build numbers picked up for subsequent runs are incorrect / already used, and so they executor goes dead.

Couple of stack trace excerpts from Jenkins log, (there are others):
Aug 11, 2015 12:18:15 PM SEVERE hudson.model.Executor run
Unexpected executor death
java.lang.IllegalStateException: /mnt/jenkins/jenkins/data/jobs/MY_JOB2/builds/653 already existed; will not overwite with MY_JOB2 #653
at hudson.model.RunMap.put(RunMap.java:187)
at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
at hudson.model.AbstractProject.newBuild(AbstractProject.java:1006)
at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
at hudson.model.Executor$1.call(Executor.java:328)
at hudson.model.Executor$1.call(Executor.java:310)
at hudson.model.Queue._withLock(Queue.java:1246)
at hudson.model.Queue.withLock(Queue.java:1184)
at hudson.model.Executor.run(Executor.java:310)

–

Aug 11, 2015 12:59:00 PM SEVERE hudson.model.Executor run
Unexpected executor death
java.lang.IllegalStateException: /mnt/jenkins/jenkins/data/jobs/MY_JOB/builds/629 already existed; will not overwite with MY_JOB #629
at hudson.model.RunMap.put(RunMap.java:187)
at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
at hudson.model.AbstractProject.newBuild(AbstractProject.java:1006)
at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
at hudson.model.Executor$1.call(Executor.java:328)
at hudson.model.Executor$1.call(Executor.java:310)
at hudson.model.Queue._withLock(Queue.java:1246)
at hudson.model.Queue.withLock(Queue.java:1184)
at hudson.model.Executor.run(Executor.java:310)

• Job build record odd behaviour:
  Attached are two screen shots explaining the behaviour, 'odd_build_records_1.609.1.png' and 'odd_build_records_1.609.1_builds_dir'.
  The last build run is 633, the last build record shown on the Job page is 626.

• Work around used so far:
  Restart the dead threads as many number of times the build records have piled up on the backend, that aren't showing on the Job page / gui.
  A Jenkins restart brings the build records back on the gui in the expected way.

One case:
Build 633 exists.
New build number picked up by Jenkins is 633 again.
Triggering a new build causes a thread on the slave to go 'Dead'.
Corresponding message is:
java.lang.IllegalStateException: /mnt/jenkins/jenkins/data/jobs/MY_JOB/builds/633 already existed; will not overwite with MY_JOB #633
at hudson.model.RunMap.put(RunMap.java:187)
at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
at hudson.model.AbstractProject.newBuild(AbstractProject.java:1006)
at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
at hudson.model.Executor$1.call(Executor.java:328)
at hudson.model.Executor$1.call(Executor.java:310)
at hudson.model.Queue._withLock(Queue.java:1246)
at hudson.model.Queue.withLock(Queue.java:1184)
at hudson.model.Executor.run(Executor.java:310)

Fix: Manually 'Set Next Build Number' to the next one, and then trigger the build. (see attached image, 'odd_build_records_restart_thread.png')
Also restart the dead executors. (see attached image, odd_build_records_set_build_number.png)