Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46961

Pipelines interrupted while starting incorrectly resume after Jenkins restarts and cannot be killed

XMLWordPrintable

    • workflow-job 2.40, workflow-cps 2.83

      I have a Multibranch Pipeline job that failed on August 30th 2017 due to a restart on the master or the slave (they're in separate servers).

      I've tried to abort it multiple times (and the log shows "Aborted by <user-at-company>" every time I try to abort), but it doesn't work.

      I've tried restarting both master and slave, but when I do that, I get a message like this on the log:

      "Resuming build at Tue Sep 12 17:49:04 BRT 2017 after Jenkins restart"

      Here are some more log lines:

       

      > git checkout -f a4ab3c46a97093925f401a391b238821f1099417
      First time build. Skipping changelog.
      java.nio.channels.ClosedByInterruptException
      at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
      at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
      at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
      at java.nio.channels.Channels.writeFully(Channels.java:101)
      at java.nio.channels.Channels.access$000(Channels.java:61)
      at java.nio.channels.Channels$1.write(Channels.java:174)
      at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
      at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
      at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
      at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
      at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
      at java.io.BufferedWriter.flush(BufferedWriter.java:254)
      at hudson.util.AtomicFileWriter.flush(AtomicFileWriter.java:97)
      at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:75)
      Caused: com.thoughtworks.xstream.io.StreamException: : null
      at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:77)
      at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:322)
      at com.thoughtworks.xstream.io.WriterWrapper.endNode(WriterWrapper.java:37)
      at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:48)
      at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83)
      at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
      at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
      at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
      at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
      at hudson.XmlFile.write(XmlFile.java:171)
      Caused: java.io.IOException
      at hudson.XmlFile.write(XmlFile.java:174)
      at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.storeNode(SimpleXStreamFlowNodeStorage.java:93)
      at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.storeNode(CpsFlowExecution.java:1481)
      at org.jenkinsci.plugins.workflow.cps.FlowHead.newStartNode(FlowHead.java:109)
      at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:487)
      at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269)
      at hudson.model.ResourceController.execute(ResourceController.java:97)
      at hudson.model.Executor.run(Executor.java:405)
      Finished: FAILURE
      Resuming build at Wed Aug 30 18:32:00 BRT 2017 after Jenkins restart
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Aborted by user-at-company
      Resuming build at Tue Sep 12 17:49:04 BRT 2017 after Jenkins restart
      Resuming build at Wed Sep 13 18:05:24 BRT 2017 after Jenkins restart
      Resuming build at Mon Sep 18 11:07:36 BRT 2017 after Jenkins restart

          [JENKINS-46961] Pipelines interrupted while starting incorrectly resume after Jenkins restarts and cannot be killed

          Elifarley created issue -
          Elifarley made changes -
          Link New: This issue is related to JENKINS-43587 [ JENKINS-43587 ]
          Elifarley made changes -
          Description Original: I have a Multibranch Pipeline job that failed on August 30th 2017 due to a restart on the master or the slave (they're in separate servers).

          I've tried to abort it multiple times, but it doesn't work.

          I've tried restarting both master and slave, but when I do that, I get a message like this on the log:

          "Resuming build at Tue Sep 12 17:49:04 BRT 2017 after Jenkins restart"

          Here are some more log lines:

           

          > git checkout -f a4ab3c46a97093925f401a391b238821f1099417
          First time build. Skipping changelog.
          java.nio.channels.ClosedByInterruptException
           at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
           at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
           at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
           at java.nio.channels.Channels.writeFully(Channels.java:101)
           at java.nio.channels.Channels.access$000(Channels.java:61)
           at java.nio.channels.Channels$1.write(Channels.java:174)
           at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
           at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
           at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
           at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
           at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
           at java.io.BufferedWriter.flush(BufferedWriter.java:254)
           at hudson.util.AtomicFileWriter.flush(AtomicFileWriter.java:97)
           at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:75)
          Caused: com.thoughtworks.xstream.io.StreamException: : null
           at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:77)
           at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:322)
           at com.thoughtworks.xstream.io.WriterWrapper.endNode(WriterWrapper.java:37)
           at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:48)
           at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83)
           at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
           at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
           at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
           at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
           at hudson.XmlFile.write(XmlFile.java:171)
          Caused: java.io.IOException
           at hudson.XmlFile.write(XmlFile.java:174)
           at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.storeNode(SimpleXStreamFlowNodeStorage.java:93)
           at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.storeNode(CpsFlowExecution.java:1481)
           at org.jenkinsci.plugins.workflow.cps.FlowHead.newStartNode(FlowHead.java:109)
           at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:487)
           at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269)
           at hudson.model.ResourceController.execute(ResourceController.java:97)
           at hudson.model.Executor.run(Executor.java:405)
          Finished: FAILURE
          Resuming build at Wed Aug 30 18:32:00 BRT 2017 after Jenkins restart
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Aborted by user-at-company
          Resuming build at Tue Sep 12 17:49:04 BRT 2017 after Jenkins restart
          Resuming build at Wed Sep 13 18:05:24 BRT 2017 after Jenkins restart
          Resuming build at Mon Sep 18 11:07:36 BRT 2017 after Jenkins restart           		New: I have a Multibranch Pipeline job that failed on August 30th 2017 due to a restart on the master or the slave (they're in separate servers).

          I've tried to abort it multiple times (and the log shows "Aborted by <user-at-company>" every time I try to abort), but it doesn't work.

          I've tried restarting both master and slave, but when I do that, I get a message like this on the log:

          "Resuming build at Tue Sep 12 17:49:04 BRT 2017 after Jenkins restart"

          Here are some more log lines:

           

          > git checkout -f a4ab3c46a97093925f401a391b238821f1099417
           First time build. Skipping changelog.
           java.nio.channels.ClosedByInterruptException
           at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
           at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:216)
           at java.nio.channels.Channels.writeFullyImpl(Channels.java:78)
           at java.nio.channels.Channels.writeFully(Channels.java:101)
           at java.nio.channels.Channels.access$000(Channels.java:61)
           at java.nio.channels.Channels$1.write(Channels.java:174)
           at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
           at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
           at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295)
           at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141)
           at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
           at java.io.BufferedWriter.flush(BufferedWriter.java:254)
           at hudson.util.AtomicFileWriter.flush(AtomicFileWriter.java:97)
           at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:75)
           Caused: com.thoughtworks.xstream.io.StreamException: : null
           at com.thoughtworks.xstream.core.util.QuickWriter.flush(QuickWriter.java:77)
           at com.thoughtworks.xstream.io.xml.PrettyPrintWriter.endNode(PrettyPrintWriter.java:322)
           at com.thoughtworks.xstream.io.WriterWrapper.endNode(WriterWrapper.java:37)
           at com.thoughtworks.xstream.io.path.PathTrackingWriter.endNode(PathTrackingWriter.java:48)
           at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83)
           at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
           at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
           at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
           at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
           at hudson.XmlFile.write(XmlFile.java:171)
           Caused: java.io.IOException
           at hudson.XmlFile.write(XmlFile.java:174)
           at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.storeNode(SimpleXStreamFlowNodeStorage.java:93)
           at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution$TimingFlowNodeStorage.storeNode(CpsFlowExecution.java:1481)
           at org.jenkinsci.plugins.workflow.cps.FlowHead.newStartNode(FlowHead.java:109)
           at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:487)
           at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:269)
           at hudson.model.ResourceController.execute(ResourceController.java:97)
           at hudson.model.Executor.run(Executor.java:405)
           Finished: FAILURE
           Resuming build at Wed Aug 30 18:32:00 BRT 2017 after Jenkins restart
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Aborted by user-at-company
           Resuming build at Tue Sep 12 17:49:04 BRT 2017 after Jenkins restart
           Resuming build at Wed Sep 13 18:05:24 BRT 2017 after Jenkins restart
           Resuming build at Mon Sep 18 11:07:36 BRT 2017 after Jenkins restart
          Andrew Bayer made changes -
          Component/s New: workflow-cps-plugin [ 21713 ]
          Component/s New: workflow-job-plugin [ 21716 ]
          Component/s Original: pipeline [ 21692 ]

          Andrew Bayer added a comment -

          fwiw, you can fix this by stopping Jenkins, going into $JENKINS_HOME/org.jenkinsci.plugins.workflow.flow.FlowExecutionList.xml, and removing the entry for the build in question. If that doesn't work, stop Jenkins, and delete the build's directory on disk.

          Andrew Bayer added a comment - fwiw, you can fix this by stopping Jenkins, going into $JENKINS_HOME/org.jenkinsci.plugins.workflow.flow.FlowExecutionList.xml , and removing the entry for the build in question. If that doesn't work, stop Jenkins, and delete the build's directory on disk.

          Adam Zovits added a comment -

          I just had the same issue. Jenkins was restarted and a job got stuck. Aborting the build via the red X button did not do anything besides logging the message that I had aborted the build, BUILDURL/term did not do anything, but BUILDURL/kill did successfully stop the build.

           

          In previous cases sometimes even the kill command did not work, only by executing "Jenkins.instance.getItemByFullName("JOBNAME").getBuildByNumber(BUILDNUMBER).finish(hudson.model.Result.ABORTED, new java.io.IOException("Aborting build"));" in the Jenkins script console could I stop the build.

          Adam Zovits added a comment - I just had the same issue. Jenkins was restarted and a job got stuck. Aborting the build via the red X button did not do anything besides logging the message that I had aborted the build, BUILDURL/term did not do anything, but BUILDURL/kill did successfully stop the build.   In previous cases sometimes even the kill command did not work, only by executing "Jenkins.instance.getItemByFullName("JOBNAME").getBuildByNumber(BUILDNUMBER).finish(hudson.model.Result.ABORTED, new java.io.IOException("Aborting build"));" in the Jenkins script console could I stop the build.

          Meng Xin Zhu added a comment -

          +1, met it again in latest lts 2.121.2.

          BUILDURL/kill did the tricky.

          Meng Xin Zhu added a comment - +1, met it again in latest lts 2.121.2. BUILDURL/kill did the tricky.

          ivy cassidy added a comment -

          Just ran into as well.

          Jenkins ver. 2.135++
          workflow-cps 2.54

          had to buldurl/kill as well.

          ivy cassidy added a comment - Just ran into as well. Jenkins ver. 2.135++ workflow-cps 2.54 had to buldurl/kill as well.

          Eslam ElHusseiny added a comment -

          I have the same issue 

          Jenkins ver. 2.140

          Pipeline: Groovy/workflow-cps v2.54

          Eslam ElHusseiny added a comment - I have the same issue  Jenkins ver. 2.140 Pipeline: Groovy/workflow-cps v2.54

          Eyal David added a comment -

          same issue Jenkins master : Jenkins ver. 2.89.4
          Pipeline: Groovy/workflow-cps v2.53

          Eyal David added a comment - same issue Jenkins master : Jenkins ver. 2.89.4 Pipeline: Groovy/workflow-cps v2.53

            dnusbaum Devin Nusbaum
            elifarley Elifarley
            Votes:
            7 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated: