Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10234

Junit result archiver getting stuck for a long time in concurrent builds

    XMLWordPrintable

Details

    Description

      When reaching the end of a build with jUnit results, possibly when the job is allowed to run concurrently, we are frequently seeing our system get stuck on "Recording test results".

      Looking at the thread list, I see the following:
      "Executor #9 for master : executing Run_Manual_SOAK #242 : waiting for Check point JUnit result archiving on Run_Manual_SOAK #241
      java.lang.Object.wait(Native Method)
      java.lang.Object.wait(Object.java:502)
      hudson.model.Run$Runner$CheckpointSet.waitForCheckPoint(Run.java:1266)
      hudson.model.Run.waitForCheckpoint(Run.java:1234)
      hudson.model.CheckPoint.block(CheckPoint.java:144)
      hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:159)
      hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
      hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:663)
      hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:638)
      hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:616)
      hudson.model.Build$RunnerImpl.post2(Build.java:161)
      hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:585)
      hudson.model.Run.run(Run.java:1399)
      hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      hudson.model.ResourceController.execute(ResourceController.java:88)
      hudson.model.Executor.run(Executor.java:145)
      Executor #9 for master : executing Run_Manual_SOAK #242 : waiting for Check point JUnit result archiving on Run_Manual_SOAK #241"

      All the stuck jobs are in the same place. They do eventually come unstuck, but can spend a long time (hours and sometimes a day or so) in this state.
      Machine load average is at 0.23 0.22 0.21.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-9913 Not obvious why some post-build tasks enforce serial behavior even when builds are concurrent

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-42727 Cppcheck plugin waiting for checkpoint erroneously when concurrent builds are enabled

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-24450 JacocoPublisher serializes concurrent builds waiting for checkpoint

          • Major - Major loss of function.
          • Closed

          Activity

            Ascending order - Click to sort in descending order
            View 17 older comments
            whimboo Henrik Skupin added a comment -

            There is version 1.565.3 LTS to be scheduled for Oct 1st. Not sure when fixes are taking into consideration for LTS releases. Anything else beside the keyword we could try to get it in? The next major version bump for LTS will happen end of Oct, where this might be fixed.

            whimboo Henrik Skupin added a comment - There is version 1.565.3 LTS to be scheduled for Oct 1st. Not sure when fixes are taking into consideration for LTS releases. Anything else beside the keyword we could try to get it in? The next major version bump for LTS will happen end of Oct, where this might be fixed.
            danielbeck Daniel Beck added a comment -

            Next LTS will be based on 1.580, and the 1.565.x line is done.

            danielbeck Daniel Beck added a comment - Next LTS will be based on 1.580, and the 1.565.x line is done.
            lan_wu Lan Wu added a comment - - edited

            Sorry new to Jenkins jira. We are hitting this bug, but I can't see from this listing which version this was fixed in. On this page, http://jenkins-ci.org/changelog-stable, I couldn't find issue 10234. Does that mean it's not in one of the LTS builds? Thanks!

            lan_wu Lan Wu added a comment - - edited Sorry new to Jenkins jira. We are hitting this bug, but I can't see from this listing which version this was fixed in. On this page, http://jenkins-ci.org/changelog-stable , I couldn't find issue 10234. Does that mean it's not in one of the LTS builds? Thanks!
            danielbeck Daniel Beck added a comment -

            Does that mean it's not in one of the LTS builds?

            No, it just means it has not specifically been fixed/backported for one of the LTS releases. It was fixed for 1.575 and is therefore in 1.580 and and the LTS releases based on that.

            danielbeck Daniel Beck added a comment - Does that mean it's not in one of the LTS builds? No, it just means it has not specifically been fixed/backported for one of the LTS releases. It was fixed for 1.575 and is therefore in 1.580 and and the LTS releases based on that.
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Martin Bektchiev
            Path:
            src/main/java/hudson/plugins/nunit/NUnitPublisher.java
            http://jenkins-ci.org/commit/nunit-plugin/1e89e0267814d280b7051d7d70d5d2939326b182
            Log:
            Do not wait for checkpoint from previous build

            Fixes an issue with result archiver getting stuck for a long time in concurrent builds.
            A similar issue has been fixed in the JUnit Jenkins plugin:

            https://issues.jenkins-ci.org/browse/JENKINS-10234
            https://github.com/jenkinsci/jenkins/commit/90ff9f806fcac1a58f4bd40bfcc4ed5273ff116a

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Martin Bektchiev Path: src/main/java/hudson/plugins/nunit/NUnitPublisher.java http://jenkins-ci.org/commit/nunit-plugin/1e89e0267814d280b7051d7d70d5d2939326b182 Log: Do not wait for checkpoint from previous build Fixes an issue with result archiver getting stuck for a long time in concurrent builds. A similar issue has been fixed in the JUnit Jenkins plugin: https://issues.jenkins-ci.org/browse/JENKINS-10234 https://github.com/jenkinsci/jenkins/commit/90ff9f806fcac1a58f4bd40bfcc4ed5273ff116a

            People

              jglick Jesse Glick
              dannystaple Danny Staple
              Votes:
              15 Vote for this issue
              Watchers:
              30 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: