Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-15931

Build is hanging in xUnit plugin processing

    XMLWordPrintable

Details

    Description

      When using Amazon EC2 plugin, builds on newly created EC2 instances sometimes hang.
      We have two permanent slaves for a certain label (e.g. 'unittest'),
      which is generated from an AMI.
      The same AMI is specified within Amazon EC2 plugin settings.

      We have a job which can be executed concurrently
      When we invoke three builds at one time, two permanent slaves are exhausted
      and the new one is created.

      The problem is that the build on the new slave hangs at the end of it
      where xUnit plugin is aggregating the test result.

      [CHECKSTYLE] Collecting checkstyle analysis files...
      [CHECKSTYLE] Computing warning deltas based on reference build #850
      [FINDBUGS] Collecting findbugs analysis files...
      [FINDBUGS] Computing warning deltas based on reference build #850
      Archiving artifacts
      [xUnit] [INFO] - Starting to record.
      [xUnit] [INFO] - Processing JUnit
      [xUnit] [INFO] - [JUnit] - 581 test report file(s) were found with the pattern '**/testresult/**/*.xml' relative to '/var/lib/jenkins/workspace/400_Precommit_Check_Branch' for the testing framework 'JUnit'.
      

      After aborting the build, the following error is shown.

      ERROR: Publisher org.jenkinsci.plugins.xunit.XUnitPublisher aborted due to exception
      java.lang.InterruptedException
      	at java.lang.Object.wait(Native Method)
      	at hudson.remoting.Request.call(Request.java:146)
      	at hudson.remoting.Channel.call(Channel.java:665)
      	at hudson.FilePath.act(FilePath.java:841)
      	at hudson.FilePath.act(FilePath.java:825)
      	at org.jenkinsci.plugins.xunit.XUnitPublisher.performTests(XUnitPublisher.java:170)
      	at org.jenkinsci.plugins.xunit.XUnitPublisher.performXUnit(XUnitPublisher.java:115)
      	at org.jenkinsci.plugins.xunit.XUnitPublisher.perform(XUnitPublisher.java:92)
      	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:779)
      	at hudson.model.Build$BuildExecution.post2(Build.java:183)
      	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:726)
      	at hudson.model.Run.execute(Run.java:1541)
      	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      	at hudson.model.ResourceController.execute(ResourceController.java:88)
      	at hudson.model.Executor.run(Executor.java:236)
      Email was triggered for: Failure
      Sending email for trigger: Failure
      

      I tried to capture the thread dump, but both master and the target slave had EMPTY thread dump while another slave had its own.
      I'd appreciate if someone give me advice.

      Jenkins ver. 1.491
      xUnit plugin 1.51
      Amazon EC2 plugin 1.17

      Attachments

        1. hs_err_pid15419.log
          92 kB
        2. jstack_master.txt
          105 kB
        3. jstack_slave.out
          13 kB
        4. lsof_slave.out
          6 kB

        Activity

          Here i added the full Java trace from core dump of the hanging Jenkins Jobs.

          voellm Eckhard Völlm added a comment - Here i added the full Java trace from core dump of the hanging Jenkins Jobs.
          voellm Eckhard Völlm added a comment - - edited

          Jenkins version 1.6.43
          xUnit version 1.99

          java version "1.7.0_91"
          OpenJDK Runtime Environment (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.14.04.1)
          OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode)
          Linux slave 4.2.0-19-lowlatency #23~14.04.1-Ubuntu SMP PREEMPT Thu Nov 12 13:19:01 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

          Here a thread where Java threads also were hanging at "java.util.zip.ZipFile.getEntry":
          https://github.com/aws/aws-sdk-java/issues/238
          same as we observe here. They also found a solution, IMHO:
          "And yes, using this system property gets rid of the thread blocks :+1: Thank you so much for the suggestion!"

          voellm Eckhard Völlm added a comment - - edited Jenkins version 1.6.43 xUnit version 1.99 java version "1.7.0_91" OpenJDK Runtime Environment (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.14.04.1) OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode) Linux slave 4.2.0-19-lowlatency #23~14.04.1-Ubuntu SMP PREEMPT Thu Nov 12 13:19:01 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux Here a thread where Java threads also were hanging at "java.util.zip.ZipFile.getEntry": https://github.com/aws/aws-sdk-java/issues/238 same as we observe here. They also found a solution, IMHO: "And yes, using this system property gets rid of the thread blocks :+1: Thank you so much for the suggestion!"
          francisu Francis Upton added a comment -

          Does not appear to be an ec2 plugin issue.

          francisu Francis Upton added a comment - Does not appear to be an ec2 plugin issue.
          voellm Eckhard Völlm added a comment - - edited

          Yes, we do not have ec2 plugin in use, i can confirm that.
          And we decided to try an upgrade of Java version now, using oracle 1.8.0_66-b17 since last week,
          Since we did that, there was no hanging, but time is too short to say something, not many test were running since snowzilla hit eastcost.

          Propose to switch Subject, to "Build is hanging in xUnit plugin processing"

          voellm Eckhard Völlm added a comment - - edited Yes, we do not have ec2 plugin in use, i can confirm that. And we decided to try an upgrade of Java version now, using oracle 1.8.0_66-b17 since last week, Since we did that, there was no hanging, but time is too short to say something, not many test were running since snowzilla hit eastcost. Propose to switch Subject, to "Build is hanging in xUnit plugin processing"
          nfalco Nikolas Falco added a comment -

          Please try with lastest version of Jenkins and the plugin, i could not reproduce this scenario also if detailed with thread dump.

          nfalco Nikolas Falco added a comment - Please try with lastest version of Jenkins and the plugin, i could not reproduce this scenario also if detailed with thread dump.

          People

            nfalco Nikolas Falco
            tmgw165 Hiroko Tamagawa
            Votes:
            4 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: