Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29888

Jenkins stuck when concurrent builds of a same job due to log rotator

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • core
    • None

      Hello,

      I use Jenkins with parallel builds to checkout many packages.

      Example:

      parallel (
      build(build("CheckOut-Package", PACKAGE_NAME: "Package1");
      build(build("CheckOut-Package", PACKAGE_NAME: "Package2");
      ...
      build(build("CheckOut-Package", PACKAGE_NAME: "Package10");
      )
      

      Each of these jobs calls the log rotator when finished.
      Some time, next log appears:

      SEVERE: Executor threw an exception
      java.util.NoSuchElementException
              at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76)
              at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63)
              at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
              at hudson.util.RunList.subList(RunList.java:139)
              at hudson.tasks.LogRotator.perform(LogRotator.java:125)
              at hudson.model.Job.logRotate(Job.java:467)
              at hudson.model.Run.execute(Run.java:1808)
              at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
              at hudson.model.ResourceController.execute(ResourceController.java:98)
              at hudson.model.Executor.run(Executor.java:374)
      

      In this case, Jenkins does not detect that the job is finished even if it is indeed finished, and stay stucked.
      Since log Rotator should not be a blocking task, I added an exception catch in this PR:
      https://github.com/jenkinsci/jenkins/pull/1790

      That was enough on my side to handle this very recurrent problem.

          [JENKINS-29888] Jenkins stuck when concurrent builds of a same job due to log rotator

          Otmane Tazi created issue -
          Oliver Gondža made changes -
          Description Original: Hello,

          I use Jenkins with parallel builds to checkout many packages.

          Example:
          parallel (
          build(build("CheckOut-Package", PACKAGE_NAME: "Package1");
          build(build("CheckOut-Package", PACKAGE_NAME: "Package2");
          ...
          build(build("CheckOut-Package", PACKAGE_NAME: "Package10");
          )

          Each of these jobs calls the log rotator when finished.
          Some time, next log appears:

          SEVERE: Executor threw an exception
          java.util.NoSuchElementException
                  at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76)
                  at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63)
                  at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
                  at hudson.util.RunList.subList(RunList.java:139)
                  at hudson.tasks.LogRotator.perform(LogRotator.java:125)
                  at hudson.model.Job.logRotate(Job.java:467)
                  at hudson.model.Run.execute(Run.java:1808)
                  at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                  at hudson.model.ResourceController.execute(ResourceController.java:98)
                  at hudson.model.Executor.run(Executor.java:374)

          In this case, Jenkins does not detect that the job is finished even if it is indeed finished, and stay stucked.
          Since log Rotator should not be a blocking task, I added an exception catch in this PR:
          https://github.com/jenkinsci/jenkins/pull/1790

          That was enough on my side to handle this very recurrent problem.
          New: Hello,

          I use Jenkins with parallel builds to checkout many packages.

          Example:
          {noformat}
          parallel (
          build(build("CheckOut-Package", PACKAGE_NAME: "Package1");
          build(build("CheckOut-Package", PACKAGE_NAME: "Package2");
          ...
          build(build("CheckOut-Package", PACKAGE_NAME: "Package10");
          )
          {noformat}

          Each of these jobs calls the log rotator when finished.
          Some time, next log appears:

          {noformat}
          SEVERE: Executor threw an exception
          java.util.NoSuchElementException
                  at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76)
                  at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63)
                  at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
                  at hudson.util.RunList.subList(RunList.java:139)
                  at hudson.tasks.LogRotator.perform(LogRotator.java:125)
                  at hudson.model.Job.logRotate(Job.java:467)
                  at hudson.model.Run.execute(Run.java:1808)
                  at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
                  at hudson.model.ResourceController.execute(ResourceController.java:98)
                  at hudson.model.Executor.run(Executor.java:374)
          {noformat}

          In this case, Jenkins does not detect that the job is finished even if it is indeed finished, and stay stucked.
          Since log Rotator should not be a blocking task, I added an exception catch in this PR:
          https://github.com/jenkinsci/jenkins/pull/1790

          That was enough on my side to handle this very recurrent problem.

          Code changed in jenkins
          User: tfennelly
          Path:
          .gitignore
          changelog.html
          cli/pom.xml
          core/pom.xml
          core/src/main/java/hudson/model/AbstractProject.java
          core/src/main/java/hudson/model/Run.java
          core/src/main/resources/hudson/widgets/HistoryWidget/index.jelly
          core/src/main/resources/lib/form/confirm.js
          core/src/main/resources/lib/form/select/select.js
          core/src/main/resources/lib/layout/layout.jelly
          plugins/pom.xml
          pom.xml
          test/pom.xml
          test/src/test/groovy/hudson/model/AbstractProjectTest.groovy
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/95ca3da67d217c90d31819ec92e521e2072acd5a
          Log:
          Merge branch 'master' into plugin-manager-dependants

          • master:
            Update the changelog by new merges:
            [FIXED JENKINS-30569] HistoryWidget: fix JS syntax error
            [FIXED JENKINS-29014] render API link conditional on getApi() presence
            JENKINS-21720 JS alert preventig to leave a configuration page even without formulary changes
            add ctags file 'tags' to .gitignore
            [maven-release-plugin] prepare release jenkins-1.632
            [FIXED JENKINS-29888] Handling all exceptions returned by logRotator
            JENKINS-30742 Fixed possible NPE in AbstractProject.resolveForCLI()

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: tfennelly Path: .gitignore changelog.html cli/pom.xml core/pom.xml core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/Run.java core/src/main/resources/hudson/widgets/HistoryWidget/index.jelly core/src/main/resources/lib/form/confirm.js core/src/main/resources/lib/form/select/select.js core/src/main/resources/lib/layout/layout.jelly plugins/pom.xml pom.xml test/pom.xml test/src/test/groovy/hudson/model/AbstractProjectTest.groovy war/pom.xml http://jenkins-ci.org/commit/jenkins/95ca3da67d217c90d31819ec92e521e2072acd5a Log: Merge branch 'master' into plugin-manager-dependants master: Update the changelog by new merges: [FIXED JENKINS-30569] HistoryWidget: fix JS syntax error [FIXED JENKINS-29014] render API link conditional on getApi() presence JENKINS-21720 JS alert preventig to leave a configuration page even without formulary changes add ctags file 'tags' to .gitignore [maven-release-plugin] prepare release jenkins-1.632 [FIXED JENKINS-29888] Handling all exceptions returned by logRotator JENKINS-30742 Fixed possible NPE in AbstractProject.resolveForCLI()

          Code changed in jenkins
          User: Otmane TAZI
          Path:
          core/src/main/java/hudson/model/Run.java
          http://jenkins-ci.org/commit/jenkins/1a5d57fdda3dba92543b8f486d3dce9cfb9c3811
          Log:
          [FIXED JENKINS-29888] Handling all exceptions returned by logRotator

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Otmane TAZI Path: core/src/main/java/hudson/model/Run.java http://jenkins-ci.org/commit/jenkins/1a5d57fdda3dba92543b8f486d3dce9cfb9c3811 Log: [FIXED JENKINS-29888] Handling all exceptions returned by logRotator
          SCM/JIRA link daemon made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: Open [ 1 ] New: Resolved [ 5 ]

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          core/src/main/java/hudson/model/Run.java
          http://jenkins-ci.org/commit/jenkins/876e8e20505b705d07b561b34863938872240369
          Log:
          Merge pull request #1790 from tototoman/master

          [FIXED JENKINS-29888] - Handling all LogRotator exceptions

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/hudson/model/Run.java http://jenkins-ci.org/commit/jenkins/876e8e20505b705d07b561b34863938872240369 Log: Merge pull request #1790 from tototoman/master [FIXED JENKINS-29888] - Handling all LogRotator exceptions

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: changelog.html http://jenkins-ci.org/commit/jenkins/a79e2f144a932fe0a97b524f79133d1ea5d0fef0 Log: Update the changelog by new merges: JENKINS-29888 - https://github.com/jenkinsci/jenkins/pull/1790 JENKINS-30742 - https://github.com/jenkinsci/jenkins/pull/1849 JENKINS-29014 - https://github.com/jenkinsci/jenkins/pull/1850 JENKINS-21720 - https://github.com/jenkinsci/jenkins/pull/1854 JENKINS-30569 - https://github.com/jenkinsci/jenkins/pull/1856

          Same issue occured two times just today on our Jenkins server.
          Version: 1.625.3 (LTS)

          From jenkins log:
          Jan 15, 2016 1:33:38 PM hudson.model.Executor finish1
          SEVERE: Executor threw an exception
          java.util.NoSuchElementException
          at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76)
          at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63)
          at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
          at hudson.util.RunList.subList(RunList.java:139)
          at hudson.tasks.LogRotator.perform(LogRotator.java:125)
          at hudson.model.Job.logRotate(Job.java:467)
          at hudson.model.Run.execute(Run.java:1805)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:408)

          Jan 15, 2016 11:27:15 AM hudson.model.Executor finish1
          SEVERE: Executor threw an exception
          java.util.NoSuchElementException
          at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76)
          at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63)
          at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
          at hudson.util.RunList.subList(RunList.java:139)
          at hudson.tasks.LogRotator.perform(LogRotator.java:125)
          at hudson.model.Job.logRotate(Job.java:467)
          at hudson.model.Run.execute(Run.java:1805)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:408)

          Thorsten Möllers added a comment - Same issue occured two times just today on our Jenkins server. Version: 1.625.3 (LTS) From jenkins log: Jan 15, 2016 1:33:38 PM hudson.model.Executor finish1 SEVERE: Executor threw an exception java.util.NoSuchElementException at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63) at java.util.AbstractMap$2$1.next(AbstractMap.java:385) at hudson.util.RunList.subList(RunList.java:139) at hudson.tasks.LogRotator.perform(LogRotator.java:125) at hudson.model.Job.logRotate(Job.java:467) at hudson.model.Run.execute(Run.java:1805) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:408) Jan 15, 2016 11:27:15 AM hudson.model.Executor finish1 SEVERE: Executor threw an exception java.util.NoSuchElementException at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63) at java.util.AbstractMap$2$1.next(AbstractMap.java:385) at hudson.util.RunList.subList(RunList.java:139) at hudson.tasks.LogRotator.perform(LogRotator.java:125) at hudson.model.Job.logRotate(Job.java:467) at hudson.model.Run.execute(Run.java:1805) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:408)
          Thorsten Möllers made changes -
          Assignee New: Régis Desgroppes [ rdesgroppes ]
          Resolution Original: Fixed [ 1 ]
          Status Original: Resolved [ 5 ] New: Reopened [ 4 ]
          Régis Desgroppes made changes -
          Assignee Original: Régis Desgroppes [ rdesgroppes ]

            Unassigned Unassigned
            totoman Otmane Tazi
            Votes:
            4 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: