Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15957

All build executors permanently "Dead!"

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Blocker Blocker
    • build-pipeline-plugin
    • None
    • RHEL5 x86_64

      I upgraded to Jenkins 1.491 a few days okay and performed a couple of builds.
      Today all nodes show all build executors as "Dead!" which I have never seen before. They refuse to come back even after restarting Jenkins.

      The assert (out of bounds exception) is the same for all threads.
      Jenkins is now completely unusable.
      I tried upgrading to 1.492 just released hoping it might be a fix
      but to no avail.

      Thread has died

      java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #7025 EXACT: lo=9,hi=5,size=19,size2=19
      at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:418)
      at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:502)
      at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:355)
      at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:207)
      at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:100)
      at hudson.model.Job.getLastBuildsOverThreshold(Job.java:856)
      at hudson.model.Job.getEstimatedDuration(Job.java:863)
      at hudson.model.queue.MappingWorksheet.<init>(MappingWorksheet.java:320)
      at hudson.model.queue.MappingWorksheet.<init>(MappingWorksheet.java:303)
      at hudson.model.Queue.maintain(Queue.java:1033)
      at hudson.model.Queue.pop(Queue.java:861)
      at hudson.model.Executor.grabJob(Executor.java:284)
      at hudson.model.Executor.run(Executor.java:205)

      Log:

      Caused by: java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #3090 EXACT: lo=1,hi=0,size=1,size2=1
      at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:418)
      at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:502)
      at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:355)
      at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:207)
      at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:100)
      at hudson.model.Job.getLastFailedBuild(Job.java:824)
      ... 135 more

      Nov 28, 2012 3:02:51 PM jenkins.model.lazy.AbstractLazyLoadRunMap search
      WARNING: Assertion error: failing to load #3090 EXACT: lo=1,hi=0,size=1,size2=1
      java.lang.Exception
      at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:417)
      at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:502)
      at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:355)
      at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:207)
      at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:100)
      at hudson.model.Job.getLastFailedBuild(Job.java:824)
      at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
      at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
      at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
      at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
      at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:72)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
      at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
      at org.kohsuke.stapler.jelly.groovy.JellyBuilder.doInvokeMethod(JellyBuilder.java:283)
      at org.kohsuke.stapler.jelly.groovy.Namespace$ProxyImpl.invoke(Namespace.java:92)
      at $Proxy32.projectView(Unknown Source)
      at lib.JenkinsTagLib$projectView.call(Unknown Source)
      at hudson.model.View.main.run(main.groovy:14)
      at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:66)
      at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:59)
      at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
      at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
      at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
      at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
      at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:107)
      at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:562)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:624)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:487)
      at org.kohsuke.stapler.Stapler.service(Stapler.java:164)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
      at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
      at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
      at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
      at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
      at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:679)

      Trivial by comparison but the message on the more details page is out of date (referring to Hudson):

      "Hudson dedicates a java.lang.Thread for each executor. Under normal operation, this thread is alive all the time (although unless a build is going on the thread is sleeping.) The "dead executor" sign (see the picture on the right) happens when this thread dies unexpectedly. When this happens, please check your Hudson system log (http://server/hudson/log) and look for a stack trace, then report that to Hudson users list or the issue tracker."

        1. systemInfo.html
          26 kB

          [JENKINS-15957] All build executors permanently "Dead!"

          Bruce Adams created issue -

          Bruce Adams added a comment -

          I should also add I have (or rather had) 4 nodes each with only 1 or 2 executor threads. All are dead
          (except the one which was already offline). They are all different flavours of linux.

          Build Executor Status

          1. Master
            1
            Dead

          2
          Dead

          RHEL6 temp
          1
          Dead

          tvdev01_brucea
          1
          Dead

          2
          Dead

          tvsys5 (offline)

          Bruce Adams added a comment - I should also add I have (or rather had) 4 nodes each with only 1 or 2 executor threads. All are dead (except the one which was already offline). They are all different flavours of linux. Build Executor Status Master 1 Dead 2 Dead RHEL6 temp 1 Dead tvdev01_brucea 1 Dead 2 Dead tvsys5 (offline)

          Bruce Adams added a comment -

          Of course because I tried to upgrade to 1.492 which also has this problem I now cannot easily downgrade to an earlier version to see if that fixes it.

          Bruce Adams added a comment - Of course because I tried to upgrade to 1.492 which also has this problem I now cannot easily downgrade to an earlier version to see if that fixes it.

          Bruce Adams added a comment -

          Attached SystemInfo.html

          Bruce Adams added a comment - Attached SystemInfo.html
          Bruce Adams made changes -
          Attachment New: systemInfo.html [ 22892 ]
          Environment Original: >lsb_release -a
          LSB Version: :core-4.0-amd64:core-4.0-ia32:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-ia32:printing-4.0-noarch
          Distributor ID: RedHatEnterpriseServer
          Description: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
          Release: 5.8
          Codename: Tikanga

          >java -version
          java version "1.6.0_34"
          Java(TM) SE Runtime Environment (build 1.6.0_34-b04)
          Java HotSpot(TM) 64-Bit Server VM (build 20.9-b04, mixed mode)
          		 New: RHEL5 x86_64

          Bruce Adams added a comment -

          >lsb_release -a
          LSB Version: :core-4.0-amd64:core-4.0-ia32:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-ia32:printing-4.0-noarch
          Distributor ID: RedHatEnterpriseServer
          Description: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
          Release: 5.8
          Codename: Tikanga

          >java -version
          java version "1.6.0_34"
          Java(TM) SE Runtime Environment (build 1.6.0_34-b04)
          Java HotSpot(TM) 64-Bit Server VM (build 20.9-b04, mixed mode)

          Bruce Adams added a comment - >lsb_release -a LSB Version: :core-4.0-amd64:core-4.0-ia32:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-ia32:printing-4.0-noarch Distributor ID: RedHatEnterpriseServer Description: Red Hat Enterprise Linux Server release 5.8 (Tikanga) Release: 5.8 Codename: Tikanga >java -version java version "1.6.0_34" Java(TM) SE Runtime Environment (build 1.6.0_34-b04) Java HotSpot(TM) 64-Bit Server VM (build 20.9-b04, mixed mode)

          candrews added a comment -

          I'm experiencing the same problem using Jenkin 1.492 on Windows 7 x86, Java 1.6.0_26-b03.

          candrews added a comment - I'm experiencing the same problem using Jenkin 1.492 on Windows 7 x86, Java 1.6.0_26-b03.

          candrews added a comment -

          This could be the same as JENKINS-15652

          candrews added a comment - This could be the same as JENKINS-15652

          Bruce Adams added a comment -

          I looked at JENKINS-15652 (and a few others) before I posted this.
          That is related to a bad build. I don't have an item.isStuck() message and removing the build history didn't make any difference.
          It is the executor threads themselves that fail to start.

          Any idea where I could find an older .war to downgrade to manually?

          Bruce Adams added a comment - I looked at JENKINS-15652 (and a few others) before I posted this. That is related to a bad build. I don't have an item.isStuck() message and removing the build history didn't make any difference. It is the executor threads themselves that fail to start. Any idea where I could find an older .war to downgrade to manually?

          Bruce Adams added a comment -

          I installed an earlier version (copied .war from this site manually) as a workaround and find that even after upgrading to 1.492 the executor threads are working.
          Though all my jobs seem to be sharing the same build history again (as per JENKINS-14583)

          Bruce Adams added a comment - I installed an earlier version (copied .war from this site manually) as a workaround and find that even after upgrading to 1.492 the executor threads are working. Though all my jobs seem to be sharing the same build history again (as per JENKINS-14583 )

            Unassigned Unassigned
            tortoise74 Bruce Adams
            Votes:
            5 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: