• Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Blocker Blocker
    • build-pipeline-plugin
    • None
    • RHEL5 x86_64

      I upgraded to Jenkins 1.491 a few days okay and performed a couple of builds.
      Today all nodes show all build executors as "Dead!" which I have never seen before. They refuse to come back even after restarting Jenkins.

      The assert (out of bounds exception) is the same for all threads.
      Jenkins is now completely unusable.
      I tried upgrading to 1.492 just released hoping it might be a fix
      but to no avail.

      Thread has died

      java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #7025 EXACT: lo=9,hi=5,size=19,size2=19
      at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:418)
      at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:502)
      at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:355)
      at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:207)
      at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:100)
      at hudson.model.Job.getLastBuildsOverThreshold(Job.java:856)
      at hudson.model.Job.getEstimatedDuration(Job.java:863)
      at hudson.model.queue.MappingWorksheet.<init>(MappingWorksheet.java:320)
      at hudson.model.queue.MappingWorksheet.<init>(MappingWorksheet.java:303)
      at hudson.model.Queue.maintain(Queue.java:1033)
      at hudson.model.Queue.pop(Queue.java:861)
      at hudson.model.Executor.grabJob(Executor.java:284)
      at hudson.model.Executor.run(Executor.java:205)

      Log:

      Caused by: java.lang.ArrayIndexOutOfBoundsException: Assertion error: failing to load #3090 EXACT: lo=1,hi=0,size=1,size2=1
      at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:418)
      at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:502)
      at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:355)
      at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:207)
      at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:100)
      at hudson.model.Job.getLastFailedBuild(Job.java:824)
      ... 135 more

      Nov 28, 2012 3:02:51 PM jenkins.model.lazy.AbstractLazyLoadRunMap search
      WARNING: Assertion error: failing to load #3090 EXACT: lo=1,hi=0,size=1,size2=1
      java.lang.Exception
      at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:417)
      at jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:502)
      at jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:355)
      at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:207)
      at hudson.model.AbstractBuild.getPreviousBuild(AbstractBuild.java:100)
      at hudson.model.Job.getLastFailedBuild(Job.java:824)
      at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
      at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
      at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
      at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
      at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
      at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
      at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
      at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
      at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:72)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
      at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
      at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
      at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
      at org.kohsuke.stapler.jelly.groovy.JellyBuilder.doInvokeMethod(JellyBuilder.java:283)
      at org.kohsuke.stapler.jelly.groovy.Namespace$ProxyImpl.invoke(Namespace.java:92)
      at $Proxy32.projectView(Unknown Source)
      at lib.JenkinsTagLib$projectView.call(Unknown Source)
      at hudson.model.View.main.run(main.groovy:14)
      at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:66)
      at org.kohsuke.stapler.jelly.groovy.GroovierJellyScript.run(GroovierJellyScript.java:59)
      at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:146)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
      at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:98)
      at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:99)
      at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
      at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
      at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:119)
      at org.kohsuke.stapler.jelly.CompressTag.doTag(CompressTag.java:44)
      at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
      at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
      at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
      at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
      at org.kohsuke.stapler.jelly.JellyClassTearOff.serveIndexJelly(JellyClassTearOff.java:107)
      at org.kohsuke.stapler.jelly.JellyFacet.handleIndexRequest(JellyFacet.java:127)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:562)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:624)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:658)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:487)
      at org.kohsuke.stapler.Stapler.service(Stapler.java:164)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:45)
      at winstone.ServletConfiguration.execute(ServletConfiguration.java:248)
      at winstone.RequestDispatcher.forward(RequestDispatcher.java:333)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:376)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:95)
      at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:66)
      at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:98)
      at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:87)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:47)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
      at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:166)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:173)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at jenkins.security.ApiTokenFilter.doFilter(ApiTokenFilter.java:63)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
      at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:66)
      at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
      at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
      at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:164)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:50)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
      at winstone.FilterConfiguration.execute(FilterConfiguration.java:194)
      at winstone.RequestDispatcher.doFilter(RequestDispatcher.java:366)
      at winstone.RequestDispatcher.forward(RequestDispatcher.java:331)
      at winstone.RequestHandlerThread.processRequest(RequestHandlerThread.java:215)
      at winstone.RequestHandlerThread.run(RequestHandlerThread.java:138)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:679)

      Trivial by comparison but the message on the more details page is out of date (referring to Hudson):

      "Hudson dedicates a java.lang.Thread for each executor. Under normal operation, this thread is alive all the time (although unless a build is going on the thread is sleeping.) The "dead executor" sign (see the picture on the right) happens when this thread dies unexpectedly. When this happens, please check your Hudson system log (http://server/hudson/log) and look for a stack trace, then report that to Hudson users list or the issue tracker."

          [JENKINS-15957] All build executors permanently "Dead!"

          Bruce Adams created issue -
          Bruce Adams made changes -
          Attachment New: systemInfo.html [ 22892 ]
          Environment Original: >lsb_release -a
          LSB Version: :core-4.0-amd64:core-4.0-ia32:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-ia32:printing-4.0-noarch
          Distributor ID: RedHatEnterpriseServer
          Description: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
          Release: 5.8
          Codename: Tikanga

          >java -version
          java version "1.6.0_34"
          Java(TM) SE Runtime Environment (build 1.6.0_34-b04)
          Java HotSpot(TM) 64-Bit Server VM (build 20.9-b04, mixed mode)
          New: RHEL5 x86_64
          Jesse Glick made changes -
          Link New: This issue duplicates JENKINS-15652 [ JENKINS-15652 ]
          Jesse Glick made changes -
          Resolution New: Duplicate [ 3 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 146759 ] New: JNJira + In-Review [ 192071 ]

            Unassigned Unassigned
            tortoise74 Bruce Adams
            Votes:
            5 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: