Loading...

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: workflow-durable-task-step-plugin
Labels:
- pipeline-hangs
- robustness
Environment:

Hide
Jenkins 2.7.2
Pipeline: Supporting APIs 2.8

Other plugins that may be relevant:
Pipeline: API 2.4
Pipeline: Basic Steps 2.2
Pipeline: Build Step 2.3
Pipeline: Groovy 2.18
Pipeline: Input Step 2.1
Pipeline: Job 2.7
Pipeline: Milestone Step 1.0
Pipeline: Multibranch 2.9
Pipeline: Nodes and Processes 2.5
Pipeline: REST API Plugin 2.1
Pipeline: SCM Step 2.2
Pipeline: Shared Groovy Libraries 2.3
Pipeline: Stage Step 2.2
Pipeline: Stage View Plugin 2.1
Pipeline: Step API 2.4
Self-Organizing Swarm Plug-in Modules 2.2 (Note: the client is using 2.0)

Show
Jenkins 2.7.2 Pipeline: Supporting APIs 2.8 Other plugins that may be relevant: Pipeline: API 2.4 Pipeline: Basic Steps 2.2 Pipeline: Build Step 2.3 Pipeline: Groovy 2.18 Pipeline: Input Step 2.1 Pipeline: Job 2.7 Pipeline: Milestone Step 1.0 Pipeline: Multibranch 2.9 Pipeline: Nodes and Processes 2.5 Pipeline: REST API Plugin 2.1 Pipeline: SCM Step 2.2 Pipeline: Shared Groovy Libraries 2.3 Pipeline: Stage Step 2.2 Pipeline: Stage View Plugin 2.1 Pipeline: Step API 2.4 Self-Organizing Swarm Plug-in Modules 2.2 (Note: the client is using 2.0)

Similar Issues:

Show
Epic Link:
Pipeline Durability

When running a Pipeline job (see end of this description) while the agent (swarm client v2.0) loses network connectivity (in this case connectivity is lost for a long while), the Pipeline hangs, showing the following in the console log:

Could not connect to <AGENT NAME> to send interrupt signal to process

After clicking the abort button (/stop), I get the same message printed in the log again and then the "Click here to terminate message":

Could not connect to <AGENT NAME> to send interrupt signal to process
Click here to forcibly terminate running steps

After forcing termination, I see the following in the console log:

Terminating sh
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Publish Results)
[Pipeline] step
09:51:54.766 Required context class hudson.Launcher is missing
09:51:54.766 Perhaps you forgot to surround the code with a step that provides this, such as: node
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag#actions for class org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage$Tag
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
	at hudson.XmlFile.write(XmlFile.java:178)
	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.saveActions(SimpleXStreamFlowNodeStorage.java:109)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.saveActions(CpsFlowExecution.java:887)
	at org.jenkinsci.plugins.workflow.graph.FlowNode.save(FlowNode.java:315)
	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.persist(FlowNode.java:302)
	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:278)
	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:269)
	at java.util.AbstractList.add(AbstractList.java:108)
	at hudson.model.Actionable.addAction(Actionable.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:273)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.throwException(ContinuationGroup.java:180)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:54)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45)
	at sun.reflect.GeneratedMethodAccessor1034.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at performance.Script1.runScript(Script1.groovy:74)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:50)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:58)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:324)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:78)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:236)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:224)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:791)
	Suppressed: java.lang.NullPointerException: while trying to invoke the method hudson.Launcher.getListener() of a null object loaded from field org.jenkinsci.plugins.docker.workflow.client.DockerClient.launcher of an object loaded from local variable 'this'
		at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:251)
		at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:247)
		at org.jenkinsci.plugins.docker.workflow.client.DockerClient.launch(DockerClient.java:244)
		at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:139)
		at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:104)
		at org.jenkinsci.plugins.docker.workflow.WithContainerStep.access$400(WithContainerStep.java:71)
		at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Callback.finished(WithContainerStep.java:281)
		at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onFailure(BodyExecutionCallback.java:124)
		at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:278)
		at com.cloudbees.groovy.cps.impl.ThrowBlock$1.receive(ThrowBlock.java:68)
		... 19 more
Caused by: java.lang.RuntimeException: Failed to serialize org.jenkinsci.plugins.workflow.actions.ErrorAction#error for class org.jenkinsci.plugins.workflow.actions.ErrorAction
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:88)
	at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:64)
	at com.thoughtworks.xstream.converters.collections.ArrayConverter.marshal(ArrayConverter.java:45)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1026)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1015)
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:988)
	at hudson.XmlFile.write(XmlFile.java:178)
	at org.jenkinsci.plugins.workflow.support.storage.SimpleXStreamFlowNodeStorage.saveActions(SimpleXStreamFlowNodeStorage.java:109)
	at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.saveActions(CpsFlowExecution.java:887)
	at org.jenkinsci.plugins.workflow.graph.FlowNode.save(FlowNode.java:315)
	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.persist(FlowNode.java:302)
	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:278)
	at org.jenkinsci.plugins.workflow.graph.FlowNode$1.add(FlowNode.java:269)
	at java.util.AbstractList.add(AbstractList.java:108)
	at hudson.model.Actionable.addAction(Actionable.java:129)
	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:273)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.throwException(ContinuationGroup.java:180)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:54)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.dispatch(CollectionLiteralBlock.java:55)
	at com.cloudbees.groovy.cps.impl.CollectionLiteralBlock$ContinuationImpl.item(CollectionLiteralBlock.java:45)
	at sun.reflect.GeneratedMethodAccessor1034.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor1031.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	... 26 more
Caused by: java.lang.RuntimeException: Failed to serialize java.lang.Throwable#cause for class java.lang.IllegalArgumentException
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:256)
	at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:224)
	at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:138)
	at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:209)
	at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:150)
	at com.thoughtworks.xstream.converters.extended.ThrowableConverter.marshal(ThrowableConverter.java:62)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
	... 83 more
Caused by: java.lang.UnsupportedOperationException: Refusing to marshal org.codehaus.groovy.runtime.InvokerInvocationException for security reasons
	at hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStream2.java:442)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
	at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:265)
	at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:252)
	... 93 more
Finished: ABORTED

Pipeline is defined like this:
Main Script:

def pipelineScript
node('my_node') {
    git changelog: false, credentialsId: '...', poll: false, url: 'ssh://...'
    pipelineScript = load '.....groovy'
}

pipelineScript.runScript()

Script1:

def runScript() {
    node("my_other_node") {

        timestamps {

            stage('S1') {
                seleniumHost = sh script: 'hostname --all-fqdns', returnStdout: true
                seleniumHost = seleniumHost.stripIndent().trim()

                env.PATH = '/usr/lib/jvm/jdk1.8.0_60//bin:$PATH:/usr/local/bin/'
            }

            stage('Checkout Git') {
                checkout(
                        [
                                $class           : 'GitSCM', branches: [[name: "${GERRIT_BRANCH}"]],
                                userRemoteConfigs: [[credentialsId: '...', url: "ssh://...", refspec: "${REFSPEC}"]],
                        ])
            }
            stage('S3') {
                sh """Some script""".stripIndent().trim()

                sh """Some other script""".stripIndent().trim()
            }

            docker.image('...').inside('') {

                try {
                    stage('S4') {
                        sh """
                        ...
                        npm install
                  """.stripIndent().trim()
                    }

                    stage('Run Tests") {
                        timeout(20) {
                            sh """grunt run_tests""".stripIndent().trim()
                        }
                    }
                } finally {
                    stage('Publish Results') {
                        archiveArtifacts artifacts: "**/${resultsFileName}"
                        junit "**/${resultsFileName}"
                    }

                }
            }

        }
    }
}

return this

Thanks!

duplicates

JENKINS-34488 Failing 'assert' hangs the pipeline

Resolved

is duplicated by

JENKINS-32228 Event when timeout is reach should be customizable

Resolved

relates to

JENKINS-37730 DurableTaskStep.Execution hanging after process is dead

Resolved

JENKINS-40613 DurableTaskStep.Execution.getWorkspace() should be subject to timeout

Resolved

links to

PR 23

Details

Description

Attachments

Issue Links

Activity

People

Dates