Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29902

Unexpected executor death - java.lang.IllegalStateException: <build> already existed

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Major Major
    • core
    • None
    • Jenkins 1.609.1
      Master on RHEL 6.6, in in Tomcat 8.

      Issue:

      • For a few slaves, the threads go dead.
      • These dead threads seem to be caused by builds on specific jobs.
      • In these cases, the build numbers picked up for subsequent runs are incorrect / already used, and so they executor goes dead.

      Couple of stack trace excerpts from Jenkins log, (there are others):
      Aug 11, 2015 12:18:15 PM SEVERE hudson.model.Executor run
      Unexpected executor death
      java.lang.IllegalStateException: /mnt/jenkins/jenkins/data/jobs/MY_JOB2/builds/653 already existed; will not overwite with MY_JOB2 #653
      at hudson.model.RunMap.put(RunMap.java:187)
      at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
      at hudson.model.AbstractProject.newBuild(AbstractProject.java:1006)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
      at hudson.model.Executor$1.call(Executor.java:328)
      at hudson.model.Executor$1.call(Executor.java:310)
      at hudson.model.Queue._withLock(Queue.java:1246)
      at hudson.model.Queue.withLock(Queue.java:1184)
      at hudson.model.Executor.run(Executor.java:310)

      Aug 11, 2015 12:59:00 PM SEVERE hudson.model.Executor run
      Unexpected executor death
      java.lang.IllegalStateException: /mnt/jenkins/jenkins/data/jobs/MY_JOB/builds/629 already existed; will not overwite with MY_JOB #629
      at hudson.model.RunMap.put(RunMap.java:187)
      at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
      at hudson.model.AbstractProject.newBuild(AbstractProject.java:1006)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
      at hudson.model.Executor$1.call(Executor.java:328)
      at hudson.model.Executor$1.call(Executor.java:310)
      at hudson.model.Queue._withLock(Queue.java:1246)
      at hudson.model.Queue.withLock(Queue.java:1184)
      at hudson.model.Executor.run(Executor.java:310)

      • Job build record odd behaviour:
        Attached are two screen shots explaining the behaviour, 'odd_build_records_1.609.1.png' and 'odd_build_records_1.609.1_builds_dir'.
        The last build run is 633, the last build record shown on the Job page is 626.
      • Work around used so far:
        Restart the dead threads as many number of times the build records have piled up on the backend, that aren't showing on the Job page / gui.
        A Jenkins restart brings the build records back on the gui in the expected way.

      One case:
      Build 633 exists.
      New build number picked up by Jenkins is 633 again.
      Triggering a new build causes a thread on the slave to go 'Dead'.
      Corresponding message is:
      java.lang.IllegalStateException: /mnt/jenkins/jenkins/data/jobs/MY_JOB/builds/633 already existed; will not overwite with MY_JOB #633
      at hudson.model.RunMap.put(RunMap.java:187)
      at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
      at hudson.model.AbstractProject.newBuild(AbstractProject.java:1006)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
      at hudson.model.Executor$1.call(Executor.java:328)
      at hudson.model.Executor$1.call(Executor.java:310)
      at hudson.model.Queue._withLock(Queue.java:1246)
      at hudson.model.Queue.withLock(Queue.java:1184)
      at hudson.model.Executor.run(Executor.java:310)

      Fix: Manually 'Set Next Build Number' to the next one, and then trigger the build. (see attached image, 'odd_build_records_restart_thread.png')
      Also restart the dead executors. (see attached image, odd_build_records_set_build_number.png)

          [JENKINS-29902] Unexpected executor death - java.lang.IllegalStateException: <build> already existed

          Nils Oliver Adam added a comment - - edited

          I did update from 1.596.2 to 1.625.1 and I am having the same issue. Even if I start from scratch (starting Jenkins with just the jobs config.xml in their directories-no other directories or symbolic links present) Build #1 is ok in most cases - but after some builds the error described above appears:
          java.lang.IllegalStateException: C:\Jenkins\jobs\xyz_trunk\builds\25 already existed; will not overwrite with xyz_trunk #25
          at hudson.model.RunMap.put(RunMap.java:189)
          at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
          at hudson.model.AbstractProject.newBuild(AbstractProject.java:1019)
          at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1209)
          at hudson.model.AbstractProject.createExecutable(AbstractProject.java:145)
          at hudson.model.Executor$1.call(Executor.java:362)
          at hudson.model.Executor$1.call(Executor.java:344)
          at hudson.model.Queue._withLock(Queue.java:1345)
          at hudson.model.Queue.withLock(Queue.java:1210)
          at hudson.model.Executor.run(Executor.java:344)

          Nils Oliver Adam added a comment - - edited I did update from 1.596.2 to 1.625.1 and I am having the same issue. Even if I start from scratch (starting Jenkins with just the jobs config.xml in their directories-no other directories or symbolic links present) Build #1 is ok in most cases - but after some builds the error described above appears: java.lang.IllegalStateException: C:\Jenkins\jobs\xyz_trunk\builds\25 already existed; will not overwrite with xyz_trunk #25 at hudson.model.RunMap.put(RunMap.java:189) at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178) at hudson.model.AbstractProject.newBuild(AbstractProject.java:1019) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1209) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:145) at hudson.model.Executor$1.call(Executor.java:362) at hudson.model.Executor$1.call(Executor.java:344) at hudson.model.Queue._withLock(Queue.java:1345) at hudson.model.Queue.withLock(Queue.java:1210) at hudson.model.Executor.run(Executor.java:344)

          Jesse Glick added a comment -

          n_o_a if you can find self-contained instructions to reproduce from scratch, the world will be grateful.

          Jesse Glick added a comment - n_o_a if you can find self-contained instructions to reproduce from scratch, the world will be grateful.

          Besides I think I didn't see any nightly job with that issue. Just daily-jobs which use svn...

          "from scratch" referred just to the build-history. Of course I didn't do a complete installation from scratch.
          The original system is working with master & two slaves (each of them is based on Windows Server 2012 R2).
          At least 300 Jobs, 75 Plugins...
          The error happens on all three machines so I think it should also be reproduceable on a simple machine without slaves.

          I'll try to reproduce the behavior on my local machine (which of cause isn't Windows Server 2012).
          I've copied it, used 5 very simple jobs...no dead executor until now..

          But I must confess the groovy installation on my machine isn't working since I updatet the jdk.
          I have to fix this first...

          Nils Oliver Adam added a comment - Besides I think I didn't see any nightly job with that issue. Just daily-jobs which use svn... "from scratch" referred just to the build-history. Of course I didn't do a complete installation from scratch. The original system is working with master & two slaves (each of them is based on Windows Server 2012 R2). At least 300 Jobs, 75 Plugins... The error happens on all three machines so I think it should also be reproduceable on a simple machine without slaves. I'll try to reproduce the behavior on my local machine (which of cause isn't Windows Server 2012). I've copied it, used 5 very simple jobs...no dead executor until now.. But I must confess the groovy installation on my machine isn't working since I updatet the jdk. I have to fix this first...

          I just saw this issue on Jenkins ver. 1.625.1 in several unrelated jobs. In my case it doesn't happen often, just the 5 or so threads that died yesterday. Only thing interesting was that I used "Reload Configuration from Disk" after mass-editing some job configurations and the first error happened soon after looking at the logs.

          Tomasz Śniatowski added a comment - I just saw this issue on Jenkins ver. 1.625.1 in several unrelated jobs. In my case it doesn't happen often, just the 5 or so threads that died yesterday. Only thing interesting was that I used "Reload Configuration from Disk" after mass-editing some job configurations and the first error happened soon after looking at the logs.

          Nils Oliver Adam added a comment - - edited

          I could reproduce it now on my local machine:

          _myjob_trunk #6 main build action completed: SUCCESS

          SCM changes detected in myjob_trunk. Triggering #8

          myjob_trunk #7 main build action completed: FAILURE
          SCM changes detected in myjob_trunk. Triggering #8

          Nov 17, 2015 12:27:23 PM SCHWERWIEGEND hudson.model.Executor run
          Unexpected executor death
          java.lang.IllegalStateException: D:\jenkins\jobs\myjob_trunk\builds\8 already existed; will not overwrite with myjob_trunk #8
          at hudson.model.RunMap.put(RunMap.java:189)
          at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
          at hudson.model.AbstractProject.newBuild(AbstractProject.java:1019)
          at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1209)
          at hudson.model.AbstractProject.createExecutable(AbstractProject.java:145)
          at hudson.model.Executor$1.call(Executor.java:362)
          at hudson.model.Executor$1.call(Executor.java:344)
          at hudson.model.Queue._withLock(Queue.java:1345)
          at hudson.model.Queue.withLock(Queue.java:1210)
          at hudson.model.Executor.run(Executor.java:344)

          Nov 17, 2015 12:27:40 PM INFORMATION hudson.model.Run execute

          Nov 17, 2015 12:28:33 PM INFORMATION hudson.model.Run execute
          myjob_trunk #8 main build action completed: FAILURE
          Nov 17, 2015 12:34:01 PM INFORMATION hudson.triggers.SCMTrigger$Runner run

          Nov 17, 2015 12:56:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
          SCM changes detected in myjob_trunk. Triggering #9
          Nov 17, 2015 12:56:01 PM INFORMATION hudson.triggers.SCMTrigger$Runner run

          Nov 17, 2015 1:02:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
          SCM changes detected in myjob_trunk. Triggering #10

          Nov 17, 2015 1:04:14 PM INFORMATION hudson.model.Run execute
          myjob_trunk #9 main build action completed: FAILURE
          Nov 17, 2015 1:04:29 PM INFORMATION hudson.model.Run execute
          AbwertungsGrund #2 main build action completed: SUCCESS
          Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Started initialization
          Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Listed all plugins
          Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Prepared all plugins
          Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Started all plugins
          Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Augmented all extensions
          Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Loaded all jobs
          Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Completed initialization
          Nov 17, 2015 1:06:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run

          SCM changes detected in myjob_trunk. Triggering #10
          Nov 17, 2015 1:06:02 PM INFORMATION hudson.model.Run execute

          Nov 17, 2015 1:08:28 PM INFORMATION hudson.model.Run execute
          myjob_trunk #10 main build action completed: FAILURE
          Nov 17, 2015 1:09:23 PM SCHWERWIEGEND hudson.model.Executor run
          Unexpected executor death
          java.lang.IllegalStateException: D:\jenkins\jobs\myjob_trunk\builds\10 already existed; will not overwrite with myjob_trunk #10
          at hudson.model.RunMap.put(RunMap.java:189)
          at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
          at hudson.model.AbstractProject.newBuild(AbstractProject.java:1019)
          at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1209)
          at hudson.model.AbstractProject.createExecutable(AbstractProject.java:145)
          at hudson.model.Executor$1.call(Executor.java:362)
          at hudson.model.Executor$1.call(Executor.java:344)
          at hudson.model.Queue._withLock(Queue.java:1345)
          at hudson.model.Queue.withLock(Queue.java:1210)
          at hudson.model.Executor.run(Executor.java:344)
          Nov 17, 2015 1:10:02 PM INFORMATION hudson.triggers.SCMTrigger$Runner run

          Nov 17, 2015 1:38:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
          SCM changes detected in myjob_trunk. Triggering #11
          Nov 17, 2015 1:38:02 PM INFORMATION hudson.triggers.SCMTrigger$Runner run

          Nov 17, 2015 1:44:47 PM INFORMATION hudson.model.Run execute
          myjob_trunk #11 main build action completed: FAILURE
          Nov 17, 2015 1:45:03 PM INFORMATION hudson.model.Run execute
          AbwertungsGrund #3 main build action completed: SUCCESS
          Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Started initialization
          Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Listed all plugins
          Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Prepared all plugins
          Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Started all plugins
          Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Augmented all extensions
          Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Loaded all jobs
          Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Completed initialization
          Nov 17, 2015 1:45:03 PM INFORMATION org.springframework.web.context.support.StaticWebApplicationContext prepareRefresh
          Refreshing org.springframework.web.context.support.StaticWebApplicationContext@7d591a4: display name [Root WebApplicationContext]; startup date [Tue Nov 17 13:45:03 CET 2015]; root of context hierarchy
          Nov 17, 2015 1:45:03 PM INFORMATION org.springframework.web.context.support.StaticWebApplicationContext obtainFreshBeanFactory
          Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@7d591a4]: org.springframework.beans.factory.support.DefaultListableBeanFactory@3aeb7f67
          Nov 17, 2015 1:45:03 PM INFORMATION org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
          Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@3aeb7f67: defining beans [initialDirContextFactory,bindAuthenticator,ldapUserSearch,authenticationManager,authoritiesPopulator]; root of factory hierarchy

          Nov 17, 2015 2:12:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
          SCM changes detected in myjob_trunk. Triggering #12
          Nov 17, 2015 2:12:02 PM INFORMATION hudson.triggers.SCMTrigger$Runner run

          Nov 17, 2015 2:16:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
          SCM changes detected in myjob_trunk. Triggering #13

          Nov 17, 2015 2:18:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run
          SCM changes detected in myjob_trunk. Triggering #13
          Nov 17, 2015 2:18:02 PM INFORMATION hudson.triggers.SCMTrigger$Runner run

          Nov 17, 2015 2:19:30 PM INFORMATION hudson.model.Run execute
          myjob_trunk #12 main build action completed: FAILURE
          Nov 17, 2015 2:19:42 PM INFORMATION hudson.model.Run execute
          AbwertungsGrund #4 main build action completed: SUCCESS
          Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Started initialization
          Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Listed all plugins
          Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Prepared all plugins
          Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Started all plugins
          Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Augmented all extensions
          Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Loaded all jobs
          Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained
          Completed initialization
          _
          I'm utilizing some jobs (e.g. "AbwertungsGrund") to show the job's reason for not beeing blue as a comment within the build-history,
          As Tomasz Sniatowski mentioned those jobs/skript do "hudson.model.Hudson.instance.doReload()" which means "Reload Configuration from Disk" I suppose. I don't know any other way to force those comments to be seen. To evaluate if this is the cause for those "dead executors" I now removed "hudson.model.Hudson.instance.doReload()" from those scripts...
          no more dead executors since then!!
          On my old ci-System running with Jenkins 1.596.2 (and more or less the same jobs) I included "hudson.model.Hudson.instance.doReload()" yesterday. No dead executors either.
          But within 1.596.2 "hudson.model.Hudson.instance.doReload()" lead to "Out-of-order builds".
          ("Timestamp inconsistent")

          "hudson.model.Hudson.instance.doReload()" or "Reload(ing) configuration from disk" while there are jobs still running lead to those issues.

          Nils Oliver Adam added a comment - - edited I could reproduce it now on my local machine: _myjob_trunk #6 main build action completed: SUCCESS … SCM changes detected in myjob_trunk. Triggering #8 … myjob_trunk #7 main build action completed: FAILURE SCM changes detected in myjob_trunk. Triggering #8 … Nov 17, 2015 12:27:23 PM SCHWERWIEGEND hudson.model.Executor run Unexpected executor death java.lang.IllegalStateException: D:\jenkins\jobs\myjob_trunk\builds\8 already existed; will not overwrite with myjob_trunk #8 at hudson.model.RunMap.put(RunMap.java:189) at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178) at hudson.model.AbstractProject.newBuild(AbstractProject.java:1019) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1209) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:145) at hudson.model.Executor$1.call(Executor.java:362) at hudson.model.Executor$1.call(Executor.java:344) at hudson.model.Queue._withLock(Queue.java:1345) at hudson.model.Queue.withLock(Queue.java:1210) at hudson.model.Executor.run(Executor.java:344) Nov 17, 2015 12:27:40 PM INFORMATION hudson.model.Run execute … Nov 17, 2015 12:28:33 PM INFORMATION hudson.model.Run execute myjob_trunk #8 main build action completed: FAILURE Nov 17, 2015 12:34:01 PM INFORMATION hudson.triggers.SCMTrigger$Runner run … Nov 17, 2015 12:56:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run SCM changes detected in myjob_trunk. Triggering #9 Nov 17, 2015 12:56:01 PM INFORMATION hudson.triggers.SCMTrigger$Runner run … Nov 17, 2015 1:02:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run SCM changes detected in myjob_trunk. Triggering #10 … Nov 17, 2015 1:04:14 PM INFORMATION hudson.model.Run execute myjob_trunk #9 main build action completed: FAILURE Nov 17, 2015 1:04:29 PM INFORMATION hudson.model.Run execute AbwertungsGrund #2 main build action completed: SUCCESS Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Started initialization Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Listed all plugins Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Prepared all plugins Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Started all plugins Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Augmented all extensions Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Loaded all jobs Nov 17, 2015 1:04:29 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Completed initialization Nov 17, 2015 1:06:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run … SCM changes detected in myjob_trunk. Triggering #10 Nov 17, 2015 1:06:02 PM INFORMATION hudson.model.Run execute … Nov 17, 2015 1:08:28 PM INFORMATION hudson.model.Run execute myjob_trunk #10 main build action completed: FAILURE Nov 17, 2015 1:09:23 PM SCHWERWIEGEND hudson.model.Executor run Unexpected executor death java.lang.IllegalStateException: D:\jenkins\jobs\myjob_trunk\builds\10 already existed; will not overwrite with myjob_trunk #10 at hudson.model.RunMap.put(RunMap.java:189) at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178) at hudson.model.AbstractProject.newBuild(AbstractProject.java:1019) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1209) at hudson.model.AbstractProject.createExecutable(AbstractProject.java:145) at hudson.model.Executor$1.call(Executor.java:362) at hudson.model.Executor$1.call(Executor.java:344) at hudson.model.Queue._withLock(Queue.java:1345) at hudson.model.Queue.withLock(Queue.java:1210) at hudson.model.Executor.run(Executor.java:344) Nov 17, 2015 1:10:02 PM INFORMATION hudson.triggers.SCMTrigger$Runner run … Nov 17, 2015 1:38:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run SCM changes detected in myjob_trunk. Triggering #11 Nov 17, 2015 1:38:02 PM INFORMATION hudson.triggers.SCMTrigger$Runner run … Nov 17, 2015 1:44:47 PM INFORMATION hudson.model.Run execute myjob_trunk #11 main build action completed: FAILURE Nov 17, 2015 1:45:03 PM INFORMATION hudson.model.Run execute AbwertungsGrund #3 main build action completed: SUCCESS Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Started initialization Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Listed all plugins Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Prepared all plugins Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Started all plugins Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Augmented all extensions Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Loaded all jobs Nov 17, 2015 1:45:03 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Completed initialization Nov 17, 2015 1:45:03 PM INFORMATION org.springframework.web.context.support.StaticWebApplicationContext prepareRefresh Refreshing org.springframework.web.context.support.StaticWebApplicationContext@7d591a4: display name [Root WebApplicationContext] ; startup date [Tue Nov 17 13:45:03 CET 2015] ; root of context hierarchy Nov 17, 2015 1:45:03 PM INFORMATION org.springframework.web.context.support.StaticWebApplicationContext obtainFreshBeanFactory Bean factory for application context [org.springframework.web.context.support.StaticWebApplicationContext@7d591a4] : org.springframework.beans.factory.support.DefaultListableBeanFactory@3aeb7f67 Nov 17, 2015 1:45:03 PM INFORMATION org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@3aeb7f67: defining beans [initialDirContextFactory,bindAuthenticator,ldapUserSearch,authenticationManager,authoritiesPopulator] ; root of factory hierarchy … Nov 17, 2015 2:12:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run SCM changes detected in myjob_trunk. Triggering #12 Nov 17, 2015 2:12:02 PM INFORMATION hudson.triggers.SCMTrigger$Runner run … Nov 17, 2015 2:16:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run SCM changes detected in myjob_trunk. Triggering #13 … Nov 17, 2015 2:18:00 PM INFORMATION hudson.triggers.SCMTrigger$Runner run SCM changes detected in myjob_trunk. Triggering #13 Nov 17, 2015 2:18:02 PM INFORMATION hudson.triggers.SCMTrigger$Runner run … Nov 17, 2015 2:19:30 PM INFORMATION hudson.model.Run execute myjob_trunk #12 main build action completed: FAILURE Nov 17, 2015 2:19:42 PM INFORMATION hudson.model.Run execute AbwertungsGrund #4 main build action completed: SUCCESS Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Started initialization Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Listed all plugins Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Prepared all plugins Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Started all plugins Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Augmented all extensions Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Loaded all jobs Nov 17, 2015 2:19:42 PM INFORMATION jenkins.InitReactorRunner$1 onAttained Completed initialization _ I'm utilizing some jobs (e.g. "AbwertungsGrund") to show the job's reason for not beeing blue as a comment within the build-history, As Tomasz Sniatowski mentioned those jobs/skript do "hudson.model.Hudson.instance.doReload()" which means "Reload Configuration from Disk" I suppose. I don't know any other way to force those comments to be seen. To evaluate if this is the cause for those "dead executors" I now removed "hudson.model.Hudson.instance.doReload()" from those scripts... no more dead executors since then!! On my old ci-System running with Jenkins 1.596.2 (and more or less the same jobs) I included "hudson.model.Hudson.instance.doReload()" yesterday. No dead executors either. But within 1.596.2 "hudson.model.Hudson.instance.doReload()" lead to "Out-of-order builds". ("Timestamp inconsistent") "hudson.model.Hudson.instance.doReload()" or "Reload(ing) configuration from disk" while there are jobs still running lead to those issues.

          Jesse Glick added a comment -

          Reload Configuration from Disk is a plausible trigger for the bug.

          Jesse Glick added a comment - Reload Configuration from Disk is a plausible trigger for the bug.

          Steffen Breitbach added a comment - - edited

          This could be true. We have a job that reloads the configuration (scripted) but also triggers a downstream job. The downstream job ended in a dead executor the last time it ran....

          Would using a quiet period help? Are the build numbers determined/created before or after the quiet period?

          Steffen Breitbach added a comment - - edited This could be true. We have a job that reloads the configuration (scripted) but also triggers a downstream job. The downstream job ended in a dead executor the last time it ran.... Would using a quiet period help? Are the build numbers determined/created before or after the quiet period?

          Jesse Glick added a comment -

          Are the build numbers determined/created before or after the quiet period?

          After.

          We have a job that reloads the configuration (scripted)

          Probably a bad idea, but at any rate if you know how to reproduce from scratch please provide details so the bug can be fixed.

          Jesse Glick added a comment - Are the build numbers determined/created before or after the quiet period? After. We have a job that reloads the configuration (scripted) Probably a bad idea, but at any rate if you know how to reproduce from scratch please provide details so the bug can be fixed.

          I managed to reproduce this on a test instance running Jenkins ver. Jenkins ver. 1.631:

          • Create a freestyle project that takes a while to complete (execute shell: sleep 60)
          • Schedule a build
          • While it is building, schedule another so one is building (1) and one is queued (2)
          • While the first job is still building, trigger 'reload configuration from disk'
          • Wait for (1) to complete: OK
          • Wait for (2) to complete: seems off: build completes, but is not visible in build list of the job
          • Trigger another build: executor dies with an "2 already existed" error

          Reproduced 2/2 times for me and seems to match the logs I have from the "real" instance of the bug I hit.

          Tomasz Śniatowski added a comment - I managed to reproduce this on a test instance running Jenkins ver. Jenkins ver. 1.631: Create a freestyle project that takes a while to complete (execute shell: sleep 60) Schedule a build While it is building, schedule another so one is building (1) and one is queued (2) While the first job is still building, trigger 'reload configuration from disk' Wait for (1) to complete: OK Wait for (2) to complete: seems off: build completes, but is not visible in build list of the job Trigger another build: executor dies with an "2 already existed" error Reproduced 2/2 times for me and seems to match the logs I have from the "real" instance of the bug I hit.

          Jesse Glick added a comment -

          tsniatowski yes these steps form the basis of my upcoming functional test.

          Jesse Glick added a comment - tsniatowski yes these steps form the basis of my upcoming functional test.

            Unassigned Unassigned
            lata lata kopalle
            Votes:
            5 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: