Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-29888

Jenkins stuck when concurrent builds of a same job due to log rotator

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Reopened (View Workflow)
    • Priority: Critical
    • Resolution: Unresolved
    • Component/s: core
    • Labels:
      None
    • Similar Issues:

      Description

      Hello,

      I use Jenkins with parallel builds to checkout many packages.

      Example:

      parallel (
      build(build("CheckOut-Package", PACKAGE_NAME: "Package1");
      build(build("CheckOut-Package", PACKAGE_NAME: "Package2");
      ...
      build(build("CheckOut-Package", PACKAGE_NAME: "Package10");
      )
      

      Each of these jobs calls the log rotator when finished.
      Some time, next log appears:

      SEVERE: Executor threw an exception
      java.util.NoSuchElementException
              at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76)
              at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63)
              at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
              at hudson.util.RunList.subList(RunList.java:139)
              at hudson.tasks.LogRotator.perform(LogRotator.java:125)
              at hudson.model.Job.logRotate(Job.java:467)
              at hudson.model.Run.execute(Run.java:1808)
              at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
              at hudson.model.ResourceController.execute(ResourceController.java:98)
              at hudson.model.Executor.run(Executor.java:374)
      

      In this case, Jenkins does not detect that the job is finished even if it is indeed finished, and stay stucked.
      Since log Rotator should not be a blocking task, I added an exception catch in this PR:
      https://github.com/jenkinsci/jenkins/pull/1790

      That was enough on my side to handle this very recurrent problem.

        Attachments

          Activity

          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: tfennelly
          Path:
          .gitignore
          changelog.html
          cli/pom.xml
          core/pom.xml
          core/src/main/java/hudson/model/AbstractProject.java
          core/src/main/java/hudson/model/Run.java
          core/src/main/resources/hudson/widgets/HistoryWidget/index.jelly
          core/src/main/resources/lib/form/confirm.js
          core/src/main/resources/lib/form/select/select.js
          core/src/main/resources/lib/layout/layout.jelly
          plugins/pom.xml
          pom.xml
          test/pom.xml
          test/src/test/groovy/hudson/model/AbstractProjectTest.groovy
          war/pom.xml
          http://jenkins-ci.org/commit/jenkins/95ca3da67d217c90d31819ec92e521e2072acd5a
          Log:
          Merge branch 'master' into plugin-manager-dependants

          • master:
            Update the changelog by new merges:
            [FIXED JENKINS-30569] HistoryWidget: fix JS syntax error
            [FIXED JENKINS-29014] render API link conditional on getApi() presence
            JENKINS-21720 JS alert preventig to leave a configuration page even without formulary changes
            add ctags file 'tags' to .gitignore
            [maven-release-plugin] prepare release jenkins-1.632
            [FIXED JENKINS-29888] Handling all exceptions returned by logRotator
            JENKINS-30742 Fixed possible NPE in AbstractProject.resolveForCLI()
          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: tfennelly Path: .gitignore changelog.html cli/pom.xml core/pom.xml core/src/main/java/hudson/model/AbstractProject.java core/src/main/java/hudson/model/Run.java core/src/main/resources/hudson/widgets/HistoryWidget/index.jelly core/src/main/resources/lib/form/confirm.js core/src/main/resources/lib/form/select/select.js core/src/main/resources/lib/layout/layout.jelly plugins/pom.xml pom.xml test/pom.xml test/src/test/groovy/hudson/model/AbstractProjectTest.groovy war/pom.xml http://jenkins-ci.org/commit/jenkins/95ca3da67d217c90d31819ec92e521e2072acd5a Log: Merge branch 'master' into plugin-manager-dependants master: Update the changelog by new merges: [FIXED JENKINS-30569] HistoryWidget: fix JS syntax error [FIXED JENKINS-29014] render API link conditional on getApi() presence JENKINS-21720 JS alert preventig to leave a configuration page even without formulary changes add ctags file 'tags' to .gitignore [maven-release-plugin] prepare release jenkins-1.632 [FIXED JENKINS-29888] Handling all exceptions returned by logRotator JENKINS-30742 Fixed possible NPE in AbstractProject.resolveForCLI()
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Otmane TAZI
          Path:
          core/src/main/java/hudson/model/Run.java
          http://jenkins-ci.org/commit/jenkins/1a5d57fdda3dba92543b8f486d3dce9cfb9c3811
          Log:
          [FIXED JENKINS-29888] Handling all exceptions returned by logRotator

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Otmane TAZI Path: core/src/main/java/hudson/model/Run.java http://jenkins-ci.org/commit/jenkins/1a5d57fdda3dba92543b8f486d3dce9cfb9c3811 Log: [FIXED JENKINS-29888] Handling all exceptions returned by logRotator
          Hide
          scm_issue_link SCM/JIRA link daemon added a comment -

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          core/src/main/java/hudson/model/Run.java
          http://jenkins-ci.org/commit/jenkins/876e8e20505b705d07b561b34863938872240369
          Log:
          Merge pull request #1790 from tototoman/master

          [FIXED JENKINS-29888] - Handling all LogRotator exceptions

          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/hudson/model/Run.java http://jenkins-ci.org/commit/jenkins/876e8e20505b705d07b561b34863938872240369 Log: Merge pull request #1790 from tototoman/master [FIXED JENKINS-29888] - Handling all LogRotator exceptions
          Show
          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: changelog.html http://jenkins-ci.org/commit/jenkins/a79e2f144a932fe0a97b524f79133d1ea5d0fef0 Log: Update the changelog by new merges: JENKINS-29888 - https://github.com/jenkinsci/jenkins/pull/1790 JENKINS-30742 - https://github.com/jenkinsci/jenkins/pull/1849 JENKINS-29014 - https://github.com/jenkinsci/jenkins/pull/1850 JENKINS-21720 - https://github.com/jenkinsci/jenkins/pull/1854 JENKINS-30569 - https://github.com/jenkinsci/jenkins/pull/1856
          Hide
          thors Thorsten Möllers added a comment -

          Same issue occured two times just today on our Jenkins server.
          Version: 1.625.3 (LTS)

          From jenkins log:
          Jan 15, 2016 1:33:38 PM hudson.model.Executor finish1
          SEVERE: Executor threw an exception
          java.util.NoSuchElementException
          at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76)
          at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63)
          at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
          at hudson.util.RunList.subList(RunList.java:139)
          at hudson.tasks.LogRotator.perform(LogRotator.java:125)
          at hudson.model.Job.logRotate(Job.java:467)
          at hudson.model.Run.execute(Run.java:1805)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:408)

          Jan 15, 2016 11:27:15 AM hudson.model.Executor finish1
          SEVERE: Executor threw an exception
          java.util.NoSuchElementException
          at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76)
          at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63)
          at java.util.AbstractMap$2$1.next(AbstractMap.java:385)
          at hudson.util.RunList.subList(RunList.java:139)
          at hudson.tasks.LogRotator.perform(LogRotator.java:125)
          at hudson.model.Job.logRotate(Job.java:467)
          at hudson.model.Run.execute(Run.java:1805)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
          at hudson.model.ResourceController.execute(ResourceController.java:98)
          at hudson.model.Executor.run(Executor.java:408)

          Show
          thors Thorsten Möllers added a comment - Same issue occured two times just today on our Jenkins server. Version: 1.625.3 (LTS) From jenkins log: Jan 15, 2016 1:33:38 PM hudson.model.Executor finish1 SEVERE: Executor threw an exception java.util.NoSuchElementException at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63) at java.util.AbstractMap$2$1.next(AbstractMap.java:385) at hudson.util.RunList.subList(RunList.java:139) at hudson.tasks.LogRotator.perform(LogRotator.java:125) at hudson.model.Job.logRotate(Job.java:467) at hudson.model.Run.execute(Run.java:1805) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:408) Jan 15, 2016 11:27:15 AM hudson.model.Executor finish1 SEVERE: Executor threw an exception java.util.NoSuchElementException at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:76) at jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63) at java.util.AbstractMap$2$1.next(AbstractMap.java:385) at hudson.util.RunList.subList(RunList.java:139) at hudson.tasks.LogRotator.perform(LogRotator.java:125) at hudson.model.Job.logRotate(Job.java:467) at hudson.model.Run.execute(Run.java:1805) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) at hudson.model.ResourceController.execute(ResourceController.java:98) at hudson.model.Executor.run(Executor.java:408)
          Hide
          tib Timmy Brolin added a comment -

          Happens somewhat frequently for us as well. Very annoying.
          (Jenkins ver. 1.625.2)

          Show
          tib Timmy Brolin added a comment - Happens somewhat frequently for us as well. Very annoying. (Jenkins ver. 1.625.2)
          Hide
          lcary Luc Cary added a comment -

          For a long time, this issue was making our CI cluster less reliable. We were on jenkins 1.625.2, and our build flow jobs would often hang when running in parallel. This was a bad user experience that made our CI pipelines flaky, since it either caused timeouts or required that we abort the build.

          This was fixed for my team by upgrading to the latest stable version of jenkins core (1.651.1). I believe any version of jenkins before https://github.com/jenkinsci/jenkins/pull/1790/files was merged will have this bug, but upgrading to a newer jenkins version (any version after 1.633) should make the exception non-fatal and thus address the issue.

          Show
          lcary Luc Cary added a comment - For a long time, this issue was making our CI cluster less reliable. We were on jenkins 1.625.2, and our build flow jobs would often hang when running in parallel. This was a bad user experience that made our CI pipelines flaky, since it either caused timeouts or required that we abort the build. This was fixed for my team by upgrading to the latest stable version of jenkins core (1.651.1). I believe any version of jenkins before https://github.com/jenkinsci/jenkins/pull/1790/files was merged will have this bug, but upgrading to a newer jenkins version (any version after 1.633) should make the exception non-fatal and thus address the issue.
          Hide
          ptha Peter Hannon added a comment -

          Thanks Luc Cary upgrading from 1.587 to 1.643 resolved this for me.

          Show
          ptha Peter Hannon added a comment - Thanks Luc Cary upgrading from 1.587 to 1.643 resolved this for me.

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            totoman Otmane Tazi
            Votes:
            4 Vote for this issue
            Watchers:
            8 Start watching this issue

              Dates

              Created:
              Updated: