Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29902

Unexpected executor death - java.lang.IllegalStateException: <build> already existed

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Major
    • core
    • None
    • Jenkins 1.609.1
      Master on RHEL 6.6, in in Tomcat 8.

    Description

      Issue:

      • For a few slaves, the threads go dead.
      • These dead threads seem to be caused by builds on specific jobs.
      • In these cases, the build numbers picked up for subsequent runs are incorrect / already used, and so they executor goes dead.

      Couple of stack trace excerpts from Jenkins log, (there are others):
      Aug 11, 2015 12:18:15 PM SEVERE hudson.model.Executor run
      Unexpected executor death
      java.lang.IllegalStateException: /mnt/jenkins/jenkins/data/jobs/MY_JOB2/builds/653 already existed; will not overwite with MY_JOB2 #653
      at hudson.model.RunMap.put(RunMap.java:187)
      at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
      at hudson.model.AbstractProject.newBuild(AbstractProject.java:1006)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
      at hudson.model.Executor$1.call(Executor.java:328)
      at hudson.model.Executor$1.call(Executor.java:310)
      at hudson.model.Queue._withLock(Queue.java:1246)
      at hudson.model.Queue.withLock(Queue.java:1184)
      at hudson.model.Executor.run(Executor.java:310)

      Aug 11, 2015 12:59:00 PM SEVERE hudson.model.Executor run
      Unexpected executor death
      java.lang.IllegalStateException: /mnt/jenkins/jenkins/data/jobs/MY_JOB/builds/629 already existed; will not overwite with MY_JOB #629
      at hudson.model.RunMap.put(RunMap.java:187)
      at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
      at hudson.model.AbstractProject.newBuild(AbstractProject.java:1006)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
      at hudson.model.Executor$1.call(Executor.java:328)
      at hudson.model.Executor$1.call(Executor.java:310)
      at hudson.model.Queue._withLock(Queue.java:1246)
      at hudson.model.Queue.withLock(Queue.java:1184)
      at hudson.model.Executor.run(Executor.java:310)

      • Job build record odd behaviour:
        Attached are two screen shots explaining the behaviour, 'odd_build_records_1.609.1.png' and 'odd_build_records_1.609.1_builds_dir'.
        The last build run is 633, the last build record shown on the Job page is 626.
      • Work around used so far:
        Restart the dead threads as many number of times the build records have piled up on the backend, that aren't showing on the Job page / gui.
        A Jenkins restart brings the build records back on the gui in the expected way.

      One case:
      Build 633 exists.
      New build number picked up by Jenkins is 633 again.
      Triggering a new build causes a thread on the slave to go 'Dead'.
      Corresponding message is:
      java.lang.IllegalStateException: /mnt/jenkins/jenkins/data/jobs/MY_JOB/builds/633 already existed; will not overwite with MY_JOB #633
      at hudson.model.RunMap.put(RunMap.java:187)
      at jenkins.model.lazy.LazyBuildMixIn.newBuild(LazyBuildMixIn.java:178)
      at hudson.model.AbstractProject.newBuild(AbstractProject.java:1006)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:1205)
      at hudson.model.AbstractProject.createExecutable(AbstractProject.java:144)
      at hudson.model.Executor$1.call(Executor.java:328)
      at hudson.model.Executor$1.call(Executor.java:310)
      at hudson.model.Queue._withLock(Queue.java:1246)
      at hudson.model.Queue.withLock(Queue.java:1184)
      at hudson.model.Executor.run(Executor.java:310)

      Fix: Manually 'Set Next Build Number' to the next one, and then trigger the build. (see attached image, 'odd_build_records_restart_thread.png')
      Also restart the dead executors. (see attached image, odd_build_records_set_build_number.png)

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              lata lata kopalle
              Votes:
              5 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: