Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-9120

RingBufferLogHandler throws ArrayIndexOutOfBoundsException after int-overflow

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core

      On our jenkins (1.403) the following exception occured (only a restart can the system bring back to a normal behaviour):

      java.lang.ArrayIndexOutOfBoundsException: -255
      at hudson.util.RingBufferLogHandler.publish(RingBufferLogHandler.java:52)
      at java.util.logging.Logger.log(Unknown Source)
      at java.util.logging.Logger.doLog(Unknown Source)
      at java.util.logging.Logger.log(Unknown Source)
      at java.util.logging.Logger.fine(Unknown Source)
      at hudson.security.SidACL.hasPermission(SidACL.java:54)
      at hudson.security.ACL.checkPermission(ACL.java:52)
      at hudson.model.Node.checkPermission(Node.java:316)
      at hudson.model.Hudson.getTarget(Hudson.java:3409)
      at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:497)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:640)
      at org.kohsuke.stapler.Stapler.invoke(Stapler.java:478)
      at org.kohsuke.stapler.Stapler.service(Stapler.java:160)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
      ... more

      After that all requests to jenkins fail with the same exception.
      The source of RingBufferLogHandler shows the usage of the primitive int's start and size which are incremented but never decremented ...

      private int start = 0;
      private final LogRecord[] records;
      private volatile int size = 0;

      ...

      public synchronized void publish(LogRecord record) {
      int len = records.length;
      records[(start+size)%len]=record;
      if(size==len)

      { start++; }

      else

      { size++; }

      }

      So after a time an overflow occures and cause the ArrayIndexOutOfBoundsException.

          [JENKINS-9120] RingBufferLogHandler throws ArrayIndexOutOfBoundsException after int-overflow

          Using 1.516 with Windows Server 2008 R2, JDK 1.7 Update 21 64Bit.

          Maybe its a problem of excessive jenkins usage? We currently use a setup of 1 master and 12 slaves. All with the same jdk. We use a jnlp connection to connect slave and master.

          This problem occurs nearly every day, although that we restart all master and slave VMs once a day.

          Is there a possible fix or workaround?

          Sebastian Laag added a comment - Using 1.516 with Windows Server 2008 R2, JDK 1.7 Update 21 64Bit. Maybe its a problem of excessive jenkins usage? We currently use a setup of 1 master and 12 slaves. All with the same jdk. We use a jnlp connection to connect slave and master. This problem occurs nearly every day, although that we restart all master and slave VMs once a day. Is there a possible fix or workaround?

          Marco Rothe added a comment -

          > Maybe its a problem of excessive jenkins usage ?

          IMO it is a "problem" of excessive logging. We encountered the problem in 2011 with DEBUG logging enabled on some categories.

          I assume if you do only warn/error logging the problem may occures nearly never ... but: The bug exists and is an accident that waits to happen!

          Marco Rothe added a comment - > Maybe its a problem of excessive jenkins usage ? IMO it is a "problem" of excessive logging. We encountered the problem in 2011 with DEBUG logging enabled on some categories. I assume if you do only warn/error logging the problem may occures nearly never ... but: The bug exists and is an accident that waits to happen!

          Richard Pon added a comment -

          We just encountered this issue this week. It's happened twice now, about every three days. Any ideas on which logs contribute to the problem the most?

          I changed my log levels to warning and info, hoping that will be a good work around. Any ideas would be great. Thanks.

          Richard Pon added a comment - We just encountered this issue this week. It's happened twice now, about every three days. Any ideas on which logs contribute to the problem the most? I changed my log levels to warning and info, hoping that will be a good work around. Any ideas would be great. Thanks.

          Richard Pon added a comment - - edited

          After some investigation, we found that a large number of calls to the Jenkins API caused this issue to occur in our system. Not sure how API calls are logged. We estimate that this failure occurred after about 37 million api requests.

          Richard Pon added a comment - - edited After some investigation, we found that a large number of calls to the Jenkins API caused this issue to occur in our system. Not sure how API calls are logged. We estimate that this failure occurred after about 37 million api requests.

          Code changed in jenkins
          User: Josh Gibbs
          Path:
          core/src/main/java/hudson/util/RingBufferLogHandler.java
          http://jenkins-ci.org/commit/jenkins/67838866c6c851adc8e39dbd20ff81a434cf6cc6
          Log:
          [FIXED JENKINS-9120] RingBufferLogHandler has int that needs to be reset.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Josh Gibbs Path: core/src/main/java/hudson/util/RingBufferLogHandler.java http://jenkins-ci.org/commit/jenkins/67838866c6c851adc8e39dbd20ff81a434cf6cc6 Log: [FIXED JENKINS-9120] RingBufferLogHandler has int that needs to be reset.

          Code changed in jenkins
          User: Oliver Gondža
          Path:
          test/src/test/java/hudson/util/RingBufferLogHandlerTest.java
          http://jenkins-ci.org/commit/jenkins/777d232053488b2dcedc122529283c6e69e99c95
          Log:
          JENKINS-9120 Add test

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: test/src/test/java/hudson/util/RingBufferLogHandlerTest.java http://jenkins-ci.org/commit/jenkins/777d232053488b2dcedc122529283c6e69e99c95 Log: JENKINS-9120 Add test

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3086
          [FIXED JENKINS-9120] RingBufferLogHandler has int that needs to be reset. (Revision 67838866c6c851adc8e39dbd20ff81a434cf6cc6)
          JENKINS-9120 Add test (Revision 777d232053488b2dcedc122529283c6e69e99c95)

          Result = SUCCESS
          ogondza : 67838866c6c851adc8e39dbd20ff81a434cf6cc6
          Files :

          • core/src/main/java/hudson/util/RingBufferLogHandler.java

          ogondza : 777d232053488b2dcedc122529283c6e69e99c95
          Files :

          • test/src/test/java/hudson/util/RingBufferLogHandlerTest.java

          dogfood added a comment - Integrated in jenkins_main_trunk #3086 [FIXED JENKINS-9120] RingBufferLogHandler has int that needs to be reset. (Revision 67838866c6c851adc8e39dbd20ff81a434cf6cc6) JENKINS-9120 Add test (Revision 777d232053488b2dcedc122529283c6e69e99c95) Result = SUCCESS ogondza : 67838866c6c851adc8e39dbd20ff81a434cf6cc6 Files : core/src/main/java/hudson/util/RingBufferLogHandler.java ogondza : 777d232053488b2dcedc122529283c6e69e99c95 Files : test/src/test/java/hudson/util/RingBufferLogHandlerTest.java

          Code changed in jenkins
          User: Oliver Gondža
          Path:
          test/src/test/java/hudson/util/RingBufferLogHandlerTest.java
          http://jenkins-ci.org/commit/jenkins/754ef75447d83767b4485ee8b15bb097f65838c4
          Log:
          JENKINS-9120 Add test

          (cherry picked from commit 777d232053488b2dcedc122529283c6e69e99c95)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: test/src/test/java/hudson/util/RingBufferLogHandlerTest.java http://jenkins-ci.org/commit/jenkins/754ef75447d83767b4485ee8b15bb097f65838c4 Log: JENKINS-9120 Add test (cherry picked from commit 777d232053488b2dcedc122529283c6e69e99c95)

          Code changed in jenkins
          User: Josh Gibbs
          Path:
          core/src/main/java/hudson/util/RingBufferLogHandler.java
          http://jenkins-ci.org/commit/jenkins/9cc2bc21307326fd8f92a0c67401d17357ab0390
          Log:
          [FIXED JENKINS-9120] RingBufferLogHandler has int that needs to be reset.

          (cherry picked from commit 67838866c6c851adc8e39dbd20ff81a434cf6cc6)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Josh Gibbs Path: core/src/main/java/hudson/util/RingBufferLogHandler.java http://jenkins-ci.org/commit/jenkins/9cc2bc21307326fd8f92a0c67401d17357ab0390 Log: [FIXED JENKINS-9120] RingBufferLogHandler has int that needs to be reset. (cherry picked from commit 67838866c6c851adc8e39dbd20ff81a434cf6cc6)

          pavan kumar added a comment -

          Hi ALex,

          I am encountering a pretty similar issue as many of our jenkins builds stopped triggering at scheduled times and we have no idea about what could be the root cause of it. We restarted our Jenkins master many times but no luck and i dont see anything unusual errors too in the logs. Please help us to overcome this issue

          pavan kumar added a comment - Hi ALex, I am encountering a pretty similar issue as many of our jenkins builds stopped triggering at scheduled times and we have no idea about what could be the root cause of it. We restarted our Jenkins master many times but no luck and i dont see anything unusual errors too in the logs. Please help us to overcome this issue

            Unassigned Unassigned
            marco_rothe Marco Rothe
            Votes:
            18 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: