Our cron box, a Jenkins slave which runs most of our tied jobs, got overwhelmed with CPU and jobs were failing. This made sense, but it didn't send failure emails even though the jobs were failing, which was quite unfortunate as it meant we were failing to update production data for a few hours until we noticed. It seems to be relying on a healthy connection to the slave in the case of a failure, which might be the cause of the failure in the first place.

I'm not actually sure if email-ext is the root culprit or not, though it is configured for affected jobs.

The full log follows:

Started by user michael
Building remotely on Cron
hudson.util.IOException2: remote file operation failed: /tmp/workspace/alerts_low_priority at hudson.remoting.Channel@317dc9:Cron
at hudson.FilePath.act(FilePath.java:754)
at hudson.FilePath.act(FilePath.java:740)
at hudson.FilePath.mkdirs(FilePath.java:806)
at hudson.model.AbstractProject.checkout(AbstractProject.java:1191)
at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:555)
at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:443)
at hudson.model.Run.run(Run.java:1376)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:230)
Caused by: hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:486)
at hudson.remoting.Request.call(Request.java:110)
at hudson.remoting.Channel.call(Channel.java:668)
at hudson.FilePath.act(FilePath.java:747)
... 9 more
Caused by: java.io.IOException
at hudson.remoting.Channel.close(Channel.java:855)
at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:112)
at hudson.remoting.PingThread.ping(PingThread.java:107)
at hudson.remoting.PingThread.run(PingThread.java:81)
Email was triggered for: Failure
Email was triggered for: Still Failing
Trigger Failure was overridden by another trigger and will not send an email.
Sending email for trigger: Still Failing
ERROR: Could not send email as a part of the post-build publishers.
hudson.remoting.ChannelClosedException: channel is already closed
at hudson.remoting.Channel.send(Channel.java:486)
at hudson.remoting.Request.call(Request.java:110)
at hudson.remoting.Channel.call(Channel.java:668)
at hudson.EnvVars.getRemote(EnvVars.java:202)
at hudson.model.Computer.getEnvironment(Computer.java:821)
at hudson.model.Run.getEnvironment(Run.java:1794)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:783)
at hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:295)
at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:235)
at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:227)
at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:189)
at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:657)
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:635)
at hudson.model.Build$RunnerImpl.cleanUp(Build.java:171)
at hudson.model.Run.run(Run.java:1420)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
at hudson.model.ResourceController.execute(ResourceController.java:88)
at hudson.model.Executor.run(Executor.java:230)
Caused by: java.io.IOException
at hudson.remoting.Channel.close(Channel.java:855)
at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:112)
at hudson.remoting.PingThread.ping(PingThread.java:107)
at hudson.remoting.PingThread.run(PingThread.java:81)
Finished: FAILURE