-
Bug
-
Resolution: Fixed
-
Major
-
None
-
Jenkins 1.450
Perforce plugin 1.3.7
Red Hat Enterprise Linux 4.5
We've noticed that sometimes our jobs get in a state where they're taking a long time to complete. Investigation reveals that the maven build has completing successfully, and it's "stuck" trying to send a success email.
Left alone, the build will eventually complete but a 10 minute build may take several hours to succeed and blocks other jobs since the executors are busy. Manual monitoring and intervention is usually required to free up the jobs.
Here's an example thread taken from a Jenkins thread dump that shows the "stuck" job. It appears to be processing the change list to retrieve the checkin user(s) that checked in code. Why is this taking so long?
The exact stack changes with each thread dump, but it is consistent up to PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111).
"Executor #2 for master : executing markview-7.0.x_document-server #103" prio=10 tid=0x48865400 nid=0x5109 runnable [0x49bfa000]
java.lang.Thread.State: RUNNABLE
at java.util.ArrayList.size(ArrayList.java:177)
at org.dom4j.tree.BackedList.<init>(BackedList.java:36)
at org.dom4j.tree.AbstractBranch.createResultList(AbstractBranch.java:373)
at org.dom4j.tree.AbstractElement.elements(AbstractElement.java:392)
at org.dom4j.tree.AbstractElement.elementIterator(AbstractElement.java:428)
at org.jaxen.dom4j.DocumentNavigator.getChildAxisIterator(DocumentNavigator.java:233)
at org.jaxen.expr.iter.IterableChildAxis.namedAccessIterator(IterableChildAxis.java:98)
at org.jaxen.expr.DefaultNameStep.evaluate(DefaultNameStep.java:180)
at org.jaxen.expr.DefaultLocationPath.evaluate(DefaultLocationPath.java:140)
at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102)
at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674)
at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213)
at org.jaxen.BaseXPath.selectSingleNode(BaseXPath.java:234)
at org.dom4j.xpath.DefaultXPath.selectSingleNode(DefaultXPath.java:159)
at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:185)
at hudson.plugins.perforce.PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111)
at hudson.plugins.perforce.PerforceChangeLogParser.parse(PerforceChangeLogParser.java:18)
at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:808)
at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:782)
at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:356)
at hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1366)
at hudson.model.User.getProjects(User.java:402)
at hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:38)
at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100)
at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514)
at hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:331)
at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:251)
at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:243)
at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:203)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:700)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:675)
at hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:1022)
at hudson.model.Run.run(Run.java:1453)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:481)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:238)
[JENKINS-12672] Jobs take a long time to complete - get stuck sending success/failure email
Code changed in jenkins
User: takeuchi
Path:
src/main/java/hudson/scm/SubversionMailAddressResolverImpl.java
http://jenkins-ci.org/commit/subversion-plugin/1fe9344144d3c95ed473a274ab40ab347c09bffa
Log:
JENKINS-14966 Aplying the same fix as in https://github.com/jenkinsci/perforce-plugin/commit/414cd9f0a6409b46eef58d1229953e5093daa940
which fixed issue JENKINS-12672
sharon xia
added a comment - This issue is still reported on 1.532.2 sharon xia
added a comment - This issue is reported on :
Jenkins: 1.532.2 LFS version.
The build has finished but hung at sending email out. The side effect is: No more other build can be started from this job, as this job is been blocked by this unfinished job.
Recording fingerprints
Email was triggered for: Success
Sending email for trigger: Success
...(just hung here)
...
sharon xia
added a comment - This issue is reported on :
Jenkins: 1.532.2 LFS version.
The build has finished but hung at sending email out. The side effect is: No more other build can be started from this job, as this job is been blocked by this unfinished job.
Recording fingerprints
Email was triggered for: Success
Sending email for trigger: Success
...(just hung here)
... sharon xia
added a comment - We are using Editable Email Notification plugin. sharon xia
added a comment - What's worse is: This build cannot be deleted as it says java.exe is using the log file.
sharon xia
added a comment - What's worse is: This build cannot be deleted as it says java.exe is using the log file. sharon xia
added a comment - perforce plugin? We don't use perforce plugin.
We were using email-ext 2.14 plugin. I just updated it to the latest 2.37.2.2. Hope this could be resolved.
sharon xia
added a comment - perforce plugin? We don't use perforce plugin.
We were using email-ext 2.14 plugin. I just updated it to the latest 2.37.2.2. Hope this could be resolved. I suppose that the high-priority handling of user-specified e-mail address (JENKINS-19433, https://github.com/jenkinsci/mailer-plugin/pull/12) will resolve performance issues of resolvers for users, who have registered in Jenkins.
Resolving as fixed. Feel free to reopen if anything weird happens again.