Details
-
Bug
-
Status: Resolved (View Workflow)
-
Major
-
Resolution: Cannot Reproduce
Description
When using Amazon EC2 plugin, builds on newly created EC2 instances sometimes hang.
We have two permanent slaves for a certain label (e.g. 'unittest'),
which is generated from an AMI.
The same AMI is specified within Amazon EC2 plugin settings.
We have a job which can be executed concurrently
When we invoke three builds at one time, two permanent slaves are exhausted
and the new one is created.
The problem is that the build on the new slave hangs at the end of it
where xUnit plugin is aggregating the test result.
[CHECKSTYLE] Collecting checkstyle analysis files... [CHECKSTYLE] Computing warning deltas based on reference build #850 [FINDBUGS] Collecting findbugs analysis files... [FINDBUGS] Computing warning deltas based on reference build #850 Archiving artifacts [xUnit] [INFO] - Starting to record. [xUnit] [INFO] - Processing JUnit [xUnit] [INFO] - [JUnit] - 581 test report file(s) were found with the pattern '**/testresult/**/*.xml' relative to '/var/lib/jenkins/workspace/400_Precommit_Check_Branch' for the testing framework 'JUnit'.
After aborting the build, the following error is shown.
ERROR: Publisher org.jenkinsci.plugins.xunit.XUnitPublisher aborted due to exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at hudson.remoting.Request.call(Request.java:146) at hudson.remoting.Channel.call(Channel.java:665) at hudson.FilePath.act(FilePath.java:841) at hudson.FilePath.act(FilePath.java:825) at org.jenkinsci.plugins.xunit.XUnitPublisher.performTests(XUnitPublisher.java:170) at org.jenkinsci.plugins.xunit.XUnitPublisher.performXUnit(XUnitPublisher.java:115) at org.jenkinsci.plugins.xunit.XUnitPublisher.perform(XUnitPublisher.java:92) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804) at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:779) at hudson.model.Build$BuildExecution.post2(Build.java:183) at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:726) at hudson.model.Run.execute(Run.java:1541) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:236) Email was triggered for: Failure Sending email for trigger: Failure
I tried to capture the thread dump, but both master and the target slave had EMPTY thread dump while another slave had its own.
I'd appreciate if someone give me advice.
Jenkins ver. 1.491
xUnit plugin 1.51
Amazon EC2 plugin 1.17
Attachments
Activity
Jenkins version 1.6.43
xUnit version 1.99
java version "1.7.0_91"
OpenJDK Runtime Environment (IcedTea 2.6.3) (7u91-2.6.3-0ubuntu0.14.04.1)
OpenJDK 64-Bit Server VM (build 24.91-b01, mixed mode)
Linux slave 4.2.0-19-lowlatency #23~14.04.1-Ubuntu SMP PREEMPT Thu Nov 12 13:19:01 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Here a thread where Java threads also were hanging at "java.util.zip.ZipFile.getEntry":
https://github.com/aws/aws-sdk-java/issues/238
same as we observe here. They also found a solution, IMHO:
"And yes, using this system property gets rid of the thread blocks :+1: Thank you so much for the suggestion!"
Francis Upton
added a comment - Does not appear to be an ec2 plugin issue. Yes, we do not have ec2 plugin in use, i can confirm that.
And we decided to try an upgrade of Java version now, using oracle 1.8.0_66-b17 since last week,
Since we did that, there was no hanging, but time is too short to say something, not many test were running since snowzilla hit eastcost.
Propose to switch Subject, to "Build is hanging in xUnit plugin processing"
Please try with lastest version of Jenkins and the plugin, i could not reproduce this scenario also if detailed with thread dump.
Here i added the full Java trace from core dump of the hanging Jenkins Jobs.