We've noticed that sometimes our jobs get in a state where they're taking a long time to complete. Investigation reveals that the maven build has completing successfully, and it's "stuck" trying to send a success email.

Left alone, the build will eventually complete but a 10 minute build may take several hours to succeed and blocks other jobs since the executors are busy. Manual monitoring and intervention is usually required to free up the jobs.

Here's an example thread taken from a Jenkins thread dump that shows the "stuck" job. It appears to be processing the change list to retrieve the checkin user(s) that checked in code. Why is this taking so long?

The exact stack changes with each thread dump, but it is consistent up to PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111).

"Executor #2 for master : executing markview-7.0.x_document-server #103" prio=10 tid=0x48865400 nid=0x5109 runnable [0x49bfa000]
java.lang.Thread.State: RUNNABLE
at java.util.ArrayList.size(ArrayList.java:177)
at org.dom4j.tree.BackedList.<init>(BackedList.java:36)
at org.dom4j.tree.AbstractBranch.createResultList(AbstractBranch.java:373)
at org.dom4j.tree.AbstractElement.elements(AbstractElement.java:392)
at org.dom4j.tree.AbstractElement.elementIterator(AbstractElement.java:428)
at org.jaxen.dom4j.DocumentNavigator.getChildAxisIterator(DocumentNavigator.java:233)
at org.jaxen.expr.iter.IterableChildAxis.namedAccessIterator(IterableChildAxis.java:98)
at org.jaxen.expr.DefaultNameStep.evaluate(DefaultNameStep.java:180)
at org.jaxen.expr.DefaultLocationPath.evaluate(DefaultLocationPath.java:140)
at org.jaxen.expr.DefaultXPathExpr.asList(DefaultXPathExpr.java:102)
at org.jaxen.BaseXPath.selectNodesForContext(BaseXPath.java:674)
at org.jaxen.BaseXPath.selectNodes(BaseXPath.java:213)
at org.jaxen.BaseXPath.selectSingleNode(BaseXPath.java:234)
at org.dom4j.xpath.DefaultXPath.selectSingleNode(DefaultXPath.java:159)
at org.dom4j.tree.AbstractNode.selectSingleNode(AbstractNode.java:185)
at hudson.plugins.perforce.PerforceChangeLogSet.parse(PerforceChangeLogSet.java:111)
at hudson.plugins.perforce.PerforceChangeLogParser.parse(PerforceChangeLogParser.java:18)
at hudson.model.AbstractBuild.calcChangeSet(AbstractBuild.java:808)
at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:782)
at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:356)
at hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1366)
at hudson.model.User.getProjects(User.java:402)
at hudson.plugins.perforce.PerforceMailResolver.findMailAddressFor(PerforceMailResolver.java:38)
at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:100)
at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:514)
at hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:331)
at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:251)
at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:243)
at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:203)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:700)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:675)
at hudson.maven.MavenModuleSetBuild$RunnerImpl.cleanUp(MavenModuleSetBuild.java:1022)
at hudson.model.Run.run(Run.java:1453)
at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:481)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:238)