Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12047

ERROR: Could not send email as a part of the post-build publishers.

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed
    • email-ext-plugin
    • None
    • Ubuntu 11.04 host and slave on AWS.

    Description

      Our cron box, a Jenkins slave which runs most of our tied jobs, got overwhelmed with CPU and jobs were failing. This made sense, but it didn't send failure emails even though the jobs were failing, which was quite unfortunate as it meant we were failing to update production data for a few hours until we noticed. It seems to be relying on a healthy connection to the slave in the case of a failure, which might be the cause of the failure in the first place.

      I'm not actually sure if email-ext is the root culprit or not, though it is configured for affected jobs.

      The full log follows:

      Started by user michael
      Building remotely on Cron
      hudson.util.IOException2: remote file operation failed: /tmp/workspace/alerts_low_priority at hudson.remoting.Channel@317dc9:Cron
      at hudson.FilePath.act(FilePath.java:754)
      at hudson.FilePath.act(FilePath.java:740)
      at hudson.FilePath.mkdirs(FilePath.java:806)
      at hudson.model.AbstractProject.checkout(AbstractProject.java:1191)
      at hudson.model.AbstractBuild$AbstractRunner.checkout(AbstractBuild.java:555)
      at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:443)
      at hudson.model.Run.run(Run.java:1376)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:230)
      Caused by: hudson.remoting.ChannelClosedException: channel is already closed
      at hudson.remoting.Channel.send(Channel.java:486)
      at hudson.remoting.Request.call(Request.java:110)
      at hudson.remoting.Channel.call(Channel.java:668)
      at hudson.FilePath.act(FilePath.java:747)
      ... 9 more
      Caused by: java.io.IOException
      at hudson.remoting.Channel.close(Channel.java:855)
      at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:112)
      at hudson.remoting.PingThread.ping(PingThread.java:107)
      at hudson.remoting.PingThread.run(PingThread.java:81)
      Email was triggered for: Failure
      Email was triggered for: Still Failing
      Trigger Failure was overridden by another trigger and will not send an email.
      Sending email for trigger: Still Failing
      ERROR: Could not send email as a part of the post-build publishers.
      hudson.remoting.ChannelClosedException: channel is already closed
      at hudson.remoting.Channel.send(Channel.java:486)
      at hudson.remoting.Request.call(Request.java:110)
      at hudson.remoting.Channel.call(Channel.java:668)
      at hudson.EnvVars.getRemote(EnvVars.java:202)
      at hudson.model.Computer.getEnvironment(Computer.java:821)
      at hudson.model.Run.getEnvironment(Run.java:1794)
      at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:783)
      at hudson.plugins.emailext.ExtendedEmailPublisher.createMail(ExtendedEmailPublisher.java:295)
      at hudson.plugins.emailext.ExtendedEmailPublisher.sendMail(ExtendedEmailPublisher.java:235)
      at hudson.plugins.emailext.ExtendedEmailPublisher._perform(ExtendedEmailPublisher.java:227)
      at hudson.plugins.emailext.ExtendedEmailPublisher.perform(ExtendedEmailPublisher.java:189)
      at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:36)
      at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:682)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:657)
      at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:635)
      at hudson.model.Build$RunnerImpl.cleanUp(Build.java:171)
      at hudson.model.Run.run(Run.java:1420)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:230)
      Caused by: java.io.IOException
      at hudson.remoting.Channel.close(Channel.java:855)
      at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:112)
      at hudson.remoting.PingThread.ping(PingThread.java:107)
      at hudson.remoting.PingThread.run(PingThread.java:81)
      Finished: FAILURE

      Attachments

        Activity

          slide_o_mix Alex Earl added a comment -

          It's trying to get the environment for the build, which apparently happened on a slave. I can wrap this call up in a try/catch, but if it doesn't succeed, other parts of the email sending process may fail still as the EnvVars are used in several places.

          slide_o_mix Alex Earl added a comment - It's trying to get the environment for the build, which apparently happened on a slave. I can wrap this call up in a try/catch, but if it doesn't succeed, other parts of the email sending process may fail still as the EnvVars are used in several places.

          Code changed in jenkins
          User: Alex Earl
          Path:
          src/main/java/hudson/plugins/emailext/ExtendedEmailPublisher.java
          http://jenkins-ci.org/commit/email-ext-plugin/5f1e0c88c4be3d4a8e1e88024cd6d464a525886c
          Log:
          Fix JENKINS-12047

          Added try/catch around environment gathering.

          scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Alex Earl Path: src/main/java/hudson/plugins/emailext/ExtendedEmailPublisher.java http://jenkins-ci.org/commit/email-ext-plugin/5f1e0c88c4be3d4a8e1e88024cd6d464a525886c Log: Fix JENKINS-12047 Added try/catch around environment gathering.

          People

            slide_o_mix Alex Earl
            mcrooney mcrooney
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: