Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19743

Massive parallel builds sometimes cause errors in LogRotation

      We submit about 120 parallel builds with 15sec execution time => issue may be caused by frequent log rotation without locks.

      Jobs seem to be OK, but there is a lot of SEVERE messages in the Jenkins log.

      SEVERE: Executor threw an exception
      java.util.NoSuchElementException
      at com.google.common.collect.AbstractIterator.next(AbstractIterator.java:154)
      at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1067)
      at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
      at hudson.util.RunList.subList(RunList.java:143)
      at hudson.tasks.LogRotator.perform(LogRotator.java:119)
      at hudson.model.Job.logRotate(Job.java:404)
      at hudson.model.Run.execute(Run.java:1655)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:237)ct.AbstractIterator.next(AbstractIterator.java:154)

        1. BuildFlowPluginTest.cs
          0.7 kB
        2. LogRotator.txt
          1 kB
        3. Run.txt
          1 kB
        4. StressTest_Job_config_1.png
          StressTest_Job_config_1.png
          116 kB
        5. StressTest_Job_config_2.png
          StressTest_Job_config_2.png
          68 kB
        6. StressTest_Job_config_3.png
          StressTest_Job_config_3.png
          85 kB
        7. StressTest_SubJob_config_1.png
          StressTest_SubJob_config_1.png
          102 kB
        8. StressTest_SubJob_config_2.png
          StressTest_SubJob_config_2.png
          87 kB
        9. StressTest_SubJob_config_3.png
          StressTest_SubJob_config_3.png
          57 kB

          [JENKINS-19743] Massive parallel builds sometimes cause errors in LogRotation

          Oleg Nenashev created issue -
          Oleg Nenashev made changes -
          Description Original: We submit about 120 parallel builds with 15sec execution time => issue may be caused by frequent log rotation without locks.

          {quote}
          SEVERE: Executor threw an exception
          java.util.NoSuchElementException
                  at com.google.common.collect.AbstractIterator.next(AbstractIterator.java:154)
                  at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1067)
                  at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
                  at hudson.util.RunList.subList(RunList.java:143)
                  at hudson.tasks.LogRotator.perform(LogRotator.java:119)
                  at hudson.model.Job.logRotate(Job.java:404)
                  at hudson.model.Run.execute(Run.java:1655)
                  at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
                  at hudson.model.ResourceController.execute(ResourceController.java:88)
                  at hudson.model.Executor.run(Executor.java:237)ct.AbstractIterator.next(AbstractIterator.java:154)
          {quote}
          New: We submit about 120 parallel builds with 15sec execution time => issue may be caused by frequent log rotation without locks.

          Jobs seem to be OK, but there is a lot of SEVERE messages in the Jenkins log.
          {quote}
          SEVERE: Executor threw an exception
          java.util.NoSuchElementException
                  at com.google.common.collect.AbstractIterator.next(AbstractIterator.java:154)
                  at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1067)
                  at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
                  at hudson.util.RunList.subList(RunList.java:143)
                  at hudson.tasks.LogRotator.perform(LogRotator.java:119)
                  at hudson.model.Job.logRotate(Job.java:404)
                  at hudson.model.Run.execute(Run.java:1655)
                  at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
                  at hudson.model.ResourceController.execute(ResourceController.java:88)
                  at hudson.model.Executor.run(Executor.java:237)ct.AbstractIterator.next(AbstractIterator.java:154)
          {quote}
          Oleg Nenashev made changes -
          Resolution New: Cannot Reproduce [ 5 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]
          Astro Ashtar made changes -
          Resolution Original: Cannot Reproduce [ 5 ]
          Status Original: Resolved [ 5 ] New: Reopened [ 4 ]
          Nebu Kadnezar made changes -
          Attachment New: BuildFlowPluginTest.cs [ 28788 ]
          Attachment New: StressTest_Job_config_1.png [ 28789 ]
          Attachment New: StressTest_Job_config_2.png [ 28790 ]
          Attachment New: StressTest_Job_config_3.png [ 28791 ]
          Attachment New: StressTest_SubJob_config_1.png [ 28792 ]
          Attachment New: StressTest_SubJob_config_2.png [ 28793 ]
          Attachment New: StressTest_SubJob_config_3.png [ 28794 ]
          Nebu Kadnezar made changes -
          Attachment New: LogRotator.txt [ 28900 ]
          Attachment New: Run.txt [ 28901 ]
          R. Tyler Croy made changes -
          Workflow Original: JNJira [ 151261 ] New: JNJira + In-Review [ 186156 ]

            Unassigned Unassigned
            oleg_nenashev Oleg Nenashev
            Votes:
            14 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated: