[JENKINS-42322] Docker rm/stop/... commands killed by the timeout, failing builds

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: docker-workflow-plugin
Labels:
- regression
Environment:
Jenkins 2.46
Debian GNU/Linux 7.8 / Docker 1.13.1
Docker worflow plugin 1.10

Similar Issues:
Powered by SuggestiMate

Show

Hi,

I've recently upgraded the docker workflow plugin from 1.8 to 1.10, in 1.8 my pipeline worked perfectly well, It uses 2 external containers and 1 where actions are done into.

In 1.10 I have the following error on the container launched with .inside { } method :

$ docker stop --time=1 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
$ docker rm -f 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
ERROR: Timeout after 10 seconds

And the job, fail with

java.io.IOException: Failed to rm container '6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02'.
	at org.jenkinsci.plugins.docker.workflow.client.DockerClient.rm(DockerClient.java:158)
	at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:145)
	at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:107)
	at org.jenkinsci.plugins.docker.workflow.WithContainerStep.access$400(WithContainerStep.java:74)
	at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Callback.finished(WithContainerStep.java:302)
	at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114)
	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362)
	at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

This seems to be related with the
https://github.com/jenkinsci/docker-workflow-plugin/pull/65
I tried to downgrade manualy to 1.8 and it works well again (as it does not specify the option --time=1 anymore).

Is there a way to disable this option ?

Thanks,

- - Sort By Name
  - Sort By Date
  - Ascending
  - Descending
  - Thumbnails
  - List
  - Download All

image-2017-06-15-10-44-46-058.png
21 kB
2017-06-15 14:44

is duplicated by

JENKINS-44767 Error when fingerprint docker: Cannot retrieve .Id

Resolved

JENKINS-42667 Job fails due to timeout when destroying docker container

Closed

is related to

JENKINS-37719 Build cannot be interrupted if `docker stop` hangs

Resolved

JENKINS-57136 Allow users to customize docker timeouts

Open

relates to

JENKINS-46969 Docker container closes prematurely

Resolved

links to

PR 107

(1 links to)

Kevin REMY created issue - 2017-02-25 19:29

Kevin REMY made changes - 2017-02-25 20:03

Description

Original: Hi,

I've recently upgraded the docker workflow plugin from 1.8 to 1.10, in 1.8 my pipeline worked perfectly well, It uses 2 external containers and 1 where actions are done into.

In 1.10 I have the following error on the container launched with .inside { } method :

$ docker stop --time=1 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
$ docker rm -f 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
ERROR: Timeout after 10 seconds

And the job, fail with
java.io.IOException: Failed to rm container '6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02'.
at org.jenkinsci.plugins.docker.workflow.client.DockerClient.rm(DockerClient.java:158)
at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:145)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:107)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep.access$400(WithContainerStep.java:74)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Callback.finished(WithContainerStep.java:302)
at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114)
at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362)
at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

This seems to be related with the
https://github.com/jenkinsci/docker-workflow-plugin/pull/65
I tried to downgrade manualy to 1.8 and it works well again (as it does not specify the option --time=1 anymore).

Is there a way to disable this option ?

Thanks,

New: Hi,

I've recently upgraded the docker workflow plugin from 1.8 to 1.10, in 1.8 my pipeline worked perfectly well, It uses 2 external containers and 1 where actions are done into.

In 1.10 I have the following error on the container launched with .inside { } method :

$ docker stop --time=1 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
$ docker rm -f 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
ERROR: Timeout after 10 seconds

And the job, fail with
{{java.io.IOException: Failed to rm container '6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02'.
at org.jenkinsci.plugins.docker.workflow.client.DockerClient.rm(DockerClient.java:158)
at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:145)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:107)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep.access$400(WithContainerStep.java:74)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Callback.finished(WithContainerStep.java:302)
at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114)
at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362)
at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)}}

This seems to be related with the
https://github.com/jenkinsci/docker-workflow-plugin/pull/65
I tried to downgrade manualy to 1.8 and it works well again (as it does not specify the option --time=1 anymore).

Is there a way to disable this option ?

Thanks,

Kevin REMY made changes - 2017-02-25 20:04

Description

Original: Hi,

I've recently upgraded the docker workflow plugin from 1.8 to 1.10, in 1.8 my pipeline worked perfectly well, It uses 2 external containers and 1 where actions are done into.

In 1.10 I have the following error on the container launched with .inside { } method :

$ docker stop --time=1 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
$ docker rm -f 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
ERROR: Timeout after 10 seconds

And the job, fail with
{{java.io.IOException: Failed to rm container '6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02'.
at org.jenkinsci.plugins.docker.workflow.client.DockerClient.rm(DockerClient.java:158)
at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:145)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:107)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep.access$400(WithContainerStep.java:74)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Callback.finished(WithContainerStep.java:302)
at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114)
at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362)
at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)}}

This seems to be related with the
https://github.com/jenkinsci/docker-workflow-plugin/pull/65
I tried to downgrade manualy to 1.8 and it works well again (as it does not specify the option --time=1 anymore).

Is there a way to disable this option ?

Thanks,

New: Hi,

I've recently upgraded the docker workflow plugin from 1.8 to 1.10, in 1.8 my pipeline worked perfectly well, It uses 2 external containers and 1 where actions are done into.

In 1.10 I have the following error on the container launched with .inside { } method :

$ docker stop --time=1 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
$ docker rm -f 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
ERROR: Timeout after 10 seconds

And the job, fail with

{code:java}
java.io.IOException: Failed to rm container '6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02'.
at org.jenkinsci.plugins.docker.workflow.client.DockerClient.rm(DockerClient.java:158)
at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:145)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:107)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep.access$400(WithContainerStep.java:74)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Callback.finished(WithContainerStep.java:302)
at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114)
at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362)
at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

{code}

This seems to be related with the
https://github.com/jenkinsci/docker-workflow-plugin/pull/65
I tried to downgrade manualy to 1.8 and it works well again (as it does not specify the option --time=1 anymore).

Is there a way to disable this option ?

Thanks,

Kevin REMY made changes - 2017-02-25 20:06

Description

Original: Hi,

I've recently upgraded the docker workflow plugin from 1.8 to 1.10, in 1.8 my pipeline worked perfectly well, It uses 2 external containers and 1 where actions are done into.

In 1.10 I have the following error on the container launched with .inside { } method :

$ docker stop --time=1 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
$ docker rm -f 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
ERROR: Timeout after 10 seconds

And the job, fail with

{code:java}
java.io.IOException: Failed to rm container '6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02'.
at org.jenkinsci.plugins.docker.workflow.client.DockerClient.rm(DockerClient.java:158)
at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:145)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:107)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep.access$400(WithContainerStep.java:74)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Callback.finished(WithContainerStep.java:302)
at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114)
at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362)
at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

{code}

This seems to be related with the
https://github.com/jenkinsci/docker-workflow-plugin/pull/65
I tried to downgrade manualy to 1.8 and it works well again (as it does not specify the option --time=1 anymore).

Is there a way to disable this option ?

Thanks,

New: Hi,

I've recently upgraded the docker workflow plugin from 1.8 to 1.10, in 1.8 my pipeline worked perfectly well, It uses 2 external containers and 1 where actions are done into.

In 1.10 I have the following error on the container launched with .inside { } method :

{code:bash}
$ docker stop --time=1 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
$ docker rm -f 6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02
ERROR: Timeout after 10 seconds
{code}

And the job, fail with

{code:java}
java.io.IOException: Failed to rm container '6b4b512400884e660dc4cd4eda6e9b3d7c358317f08a1c46399b5253ec7e1b02'.
at org.jenkinsci.plugins.docker.workflow.client.DockerClient.rm(DockerClient.java:158)
at org.jenkinsci.plugins.docker.workflow.client.DockerClient.stop(DockerClient.java:145)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep.destroy(WithContainerStep.java:107)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep.access$400(WithContainerStep.java:74)
at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Callback.finished(WithContainerStep.java:302)
at org.jenkinsci.plugins.workflow.steps.BodyExecutionCallback$TailCall.onSuccess(BodyExecutionCallback.java:114)
at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$SuccessAdapter.receive(CpsBodyExecution.java:362)
at com.cloudbees.groovy.cps.Outcome.resumeFrom(Outcome.java:73)
at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:146)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:165)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

{code}

This seems to be related with the
https://github.com/jenkinsci/docker-workflow-plugin/pull/65
I tried to downgrade manualy to 1.8 and it works well again (as it does not specify the option --time=1 anymore).

Is there a way to disable this option ?

Thanks,

Spoon added a comment - 2017-03-01 17:35

I see exactly the same timeout. Most often with the "docker rm" step, but also with the "docker stop" and even with Docker starting up!! This effectively makes using docker on a loaded system impractical as you don't know if you will hit the problem.

Please can you provide a way to remove/control the timeout.

Spoon added a comment - 2017-03-01 17:35 I see exactly the same timeout. Most often with the "docker rm" step, but also with the "docker stop" and even with Docker starting up!! This effectively makes using docker on a loaded system impractical as you don't know if you will hit the problem. Please can you provide a way to remove/control the timeout.

Spoon added a comment - 2017-03-01 17:56 - edited

This looks to be the result of the fix for https://issues.jenkins-ci.org/browse/JENKINS-42322.

While I understand the need to have some timeout, 10 seconds is short.

~~JENKINS-42322~~ mentions:

(All Docker commands in the plugin that are expected to take a long time, because they might contact a registry, are run from sh steps in Groovy code so they are durable and cleanly interruptible. Docker commands which we do not need to wait for, like docker exec to run processes in the container, are just launched and the process handle discarded. DockerClient with its blocking join call is only used for commands which under normal conditions should be close to instantaneous: docker run -d after checking for local availability of the image, docker inspect, docker stop, etc. But clearly this assumption is not completely reliable.

The suggested workaround to use sh is far from ideal when you are used to the lovely integration offered by the docker-workflow-plugin.

Could something like this be possible?

docker.image('image').inside("-u fred").timeout(60) {
...
}

Spoon added a comment - 2017-03-01 17:56 - edited This looks to be the result of the fix for https://issues.jenkins-ci.org/browse/JENKINS-42322 . While I understand the need to have some timeout, 10 seconds is short. JENKINS-42322 mentions: (All Docker commands in the plugin that are expected to take a long time, because they might contact a registry, are run from sh steps in Groovy code so they are durable and cleanly interruptible. Docker commands which we do not need to wait for, like docker exec to run processes in the container, are just launched and the process handle discarded. DockerClient with its blocking join call is only used for commands which under normal conditions should be close to instantaneous: docker run -d after checking for local availability of the image, docker inspect, docker stop, etc. But clearly this assumption is not completely reliable. The suggested workaround to use sh is far from ideal when you are used to the lovely integration offered by the docker-workflow-plugin. Could something like this be possible? docker.image('image').inside("-u fred").timeout(60) { ... }

Spoon made changes - 2017-03-01 17:57

Link

New: This issue is related to ~~JENKINS-37719~~ [ ~~JENKINS-37719~~ ]

Nazim Sitmanbetov added a comment - 2017-03-10 18:01

Suffering from the same issue. The worst thing about it is that it's hard to reproduce. What I observe is whenever build machine under high load, most probably jenkins docker plugin timeouts during `docker rm -f ...` step.

Still don't understand why `docker rm -f ...` times out. Under the hood it should be just SIGKILL command for corresponding process, but I'm not sure.

Was anyone able to make some workaround, or even have idea how to fix this?

Nazim Sitmanbetov added a comment - 2017-03-10 18:01 Suffering from the same issue. The worst thing about it is that it's hard to reproduce. What I observe is whenever build machine under high load, most probably jenkins docker plugin timeouts during `docker rm -f ...` step. Still don't understand why `docker rm -f ...` times out. Under the hood it should be just SIGKILL command for corresponding process, but I'm not sure. Was anyone able to make some workaround, or even have idea how to fix this?

Filip Pytloun added a comment - 2017-03-13 09:37 - edited

Unfortunately, we are also affected by this issue and the only workaround is downgrade.

Timeout for stop probably wouldn't be an issue but the problem is that it also affects removal of container - which may take a long time if container layer is big (eg. when there was installed a lot of packages during build) and/or storage is not fast enough.

Filip Pytloun added a comment - 2017-03-13 09:37 - edited Unfortunately, we are also affected by this issue and the only workaround is downgrade. Timeout for stop probably wouldn't be an issue but the problem is that it also affects removal of container - which may take a long time if container layer is big (eg. when there was installed a lot of packages during build) and/or storage is not fast enough.

Hugh Saunders added a comment - 2017-03-17 15:17

Also seeing this, downgrading is complicated by other plugins requiring at least v1.9 of this plugin. Trying a longer timeout. https://github.com/jenkinsci/docker-workflow-plugin/compare/master...hughsaunders:remove_timeout

Hugh Saunders added a comment - 2017-03-17 15:17 Also seeing this, downgrading is complicated by other plugins requiring at least v1.9 of this plugin. Trying a longer timeout. https://github.com/jenkinsci/docker-workflow-plugin/compare/master...hughsaunders:remove_timeout

Assignee:: Jesse Glick

Reporter:: Kevin REMY

Votes:: 34 Vote for this issue

Watchers:: 51 Start watching this issue

Created:: 2017-02-25 19:29

Updated:: 2022-03-14 16:29

Resolved:: 2017-06-16 13:28

Jenkins

Details

Description

Attachments

Attachments

Issue Links

Activity

Collapse comment: Spoon added a comment - 2017-03-01 17:35

Expand comment: Spoon added a comment - 2017-03-01 17:35

Collapse comment: Spoon added a comment - 2017-03-01 17:56, Edited by Spoon - 2017-03-01 18:01

Expand comment: Spoon added a comment - 2017-03-01 17:56, Edited by Spoon - 2017-03-01 18:01

Collapse comment: Nazim Sitmanbetov added a comment - 2017-03-10 18:01

Expand comment: Nazim Sitmanbetov added a comment - 2017-03-10 18:01

Collapse comment: Filip Pytloun added a comment - 2017-03-13 09:37, Edited by Filip Pytloun - 2017-03-13 09:37

Expand comment: Filip Pytloun added a comment - 2017-03-13 09:37, Edited by Filip Pytloun - 2017-03-13 09:37

Collapse comment: Hugh Saunders added a comment - 2017-03-17 15:17

Expand comment: Hugh Saunders added a comment - 2017-03-17 15:17

People

Dates