Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-49446

Regression with 1.15 and WithContainerStep

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Major Major
    • docker-workflow-plugin
    • None
    • Docker Pipeline 1.15 + Jenkins core 2.89.3

      I noticed that after upgrade to 1.15, steps such as {{docker.image().inside }} have begun to fail with:

      java.io.IOException: Failed to run top '7924b7207cfe14b8abba497c6051504cf0de0c02b40190b3688b78d680f3ee81'. Error: Error response from daemon: Container 7924b7207cfe14b8abba497c6051504cf0de0c02b40190b3688b78d680f3ee81 is not running
	at org.jenkinsci.plugins.docker.workflow.client.DockerClient.listProcess(DockerClient.java:140)
	at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:185)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:229)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:153)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:19)
	at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:135)
	at org.jenkinsci.plugins.docker.workflow.Docker.node(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:66)
	at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(jar:file:/var/jenkins_home/plugins/docker-workflow/WEB-INF/lib/docker-workflow.jar!/org/jenkinsci/plugins/docker/workflow/Docker.groovy:123)
	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.DockerPipelineScript.runImage(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/DockerPipelineScript.groovy:57)
	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.configureRegistry(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy:67)
	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.run(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/impl/AbstractDockerPipelineScript.groovy:53)
	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(jar:file:/var/jenkins_home/plugins/pipeline-model-extensions/WEB-INF/lib/pipeline-model-extensions.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/agent/CheckoutScript.groovy:63)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:331)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:82)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:243)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:231)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

      I have witnessed this with several Pipelines using docker.image.inside in Scripted Pipeline on ci.jenkins.io

      The only remediation I could find was to downgrade to 1.14

          [JENKINS-49446] Regression with 1.15 and WithContainerStep

          Nicolas De Loof added a comment -

          Removing support for entrypoint was a breaking change, restoring is another from some point of view. Making --entrypoint="" the default behaviour would be wrong as well.

          Nothing we can do here, docker.inside is just abusing container lifecycle. Should be deprecated imho, but being highly used by declarative we miss an alternative

           

          Nicolas De Loof added a comment - Removing support for entrypoint was a breaking change, restoring is another from some point of view. Making --entrypoint="" the default behaviour would be wrong as well. Nothing we can do here, docker.inside is just abusing container lifecycle. Should be deprecated imho, but being highly used by declarative we miss an alternative  

          R. Tyler Croy added a comment -

          ndeloof, I understand that defaulting to -entrypoint="" is problematic, but why wouldn't it be reasonable to add the behavior to try -entrypoint="" when that docker top check fails. That would be, IMHO, ideal for end users to avoid breakages and consistent usage of docker.image.inside

          R. Tyler Croy added a comment - ndeloof , I understand that defaulting to - entrypoint="" is problematic, but why wouldn't it be reasonable to add the behavior to try -entrypoint="" when that docker top check fails. That would be, IMHO, ideal for end users to avoid breakages and consistent usage of docker.image.inside

          Nicolas De Loof added a comment -

          I dislike the idea we magically disable entrypoint. If image doesn't support passing a command, should be explicit by end-user to disable entrypoint, of doing this in the background will result in some unexpected side effects and resurection for some older issues.

          but that's just my opinion, if others consider this a sane approach, feel free to follow up this way.

          Nicolas De Loof added a comment - I dislike the idea we magically disable entrypoint. If image doesn't support passing a command, should be explicit by end-user to disable entrypoint, of doing this in the background will result in some unexpected side effects and resurection for some older issues. but that's just my opinion, if others consider this a sane approach, feel free to follow up this way.

          R. Tyler Croy added a comment -

          ndeloof, I think that's a reasonable concern. Would you be willing perhaps to write for jenkins.io about entrypoint and changes for Docker Pipeline 1.15?

          My highest concern here is that a point release might introduce surprising, and seemingly breaking, behavior for the nearly 100k installations of this plugin.

          R. Tyler Croy added a comment - ndeloof , I think that's a reasonable concern. Would you be willing perhaps to write for jenkins.io about entrypoint and changes for Docker Pipeline 1.15? My highest concern here is that a point release might introduce surprising, and seemingly breaking, behavior for the nearly 100k installations of this plugin.

          Andrew Bayer added a comment -

          fwiw, "abusing container lifecycle" is, frankly, irrelevant here. What matters is that we keep from breaking users any more than absolutely necessary. I strongly agreed with going back to cmd rather than entrypoint, but there is a valid point that we may have waited too long to do so. Not to say I have the right answer, but we do need to think about this.

          Andrew Bayer added a comment - fwiw, "abusing container lifecycle" is, frankly, irrelevant here. What matters is that we keep from breaking users any more than absolutely necessary. I strongly agreed with going back to cmd rather than entrypoint, but there is a valid point that we may have waited too long to do so. Not to say I have the right answer, but we do need to think about this.

          marc young added a comment - - edited

          I'd also like to point out (aside from breaking everything at 3 of my contracts): you've based everything on "requirements" that are actually suggestions.

          The error I'm now facing in dozens of builds:

          04:11:59 ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements).

          Youve stated as required by . However those docs clearly say:

          All official images should provide a consistent interface.

          Should != require. Your use-case and assumptions are not a standard to hold an entire community to

          Please remember: I'm using other peoples official containers to run in jenkins. I cannot (and will not) fork every single one of these to meet your guidelines.

          marc young added a comment - - edited I'd also like to point out (aside from breaking everything at 3 of my contracts): you've based everything on "requirements" that are actually suggestions. The error I'm now facing in dozens of builds: 04:11:59 ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https: //github.com/docker-library/official-images#consistency for entrypoint consistency requirements). Youve stated as required by . However those docs clearly say: All official images should provide a consistent interface . Should != require. Your use-case and assumptions are not a standard to hold an entire community to Please remember: I'm using other peoples official containers to run in jenkins. I cannot (and will not) fork every single one of these to meet your guidelines.

          Nicolas De Loof added a comment -

          @myoung34 you're perfectly right, this is not a requirement but a convention on official image (it's required to get official image approved), and many custom image don't follow this.

          Our issue here is we are abusing the container lifecycle : an arbitrary docker image is not designed to run arbitrary command like we use them for.

          One option is to override entrypoint to run a custom command. But then we disable the initial entrypoint designed by image author, which in many cases is required for the image to make any sense. Typically, Selenium images do use it to run a X11 server. This option has been adopted in the past, introducing https://issues.jenkins-ci.org/browse/JENKINS-41316 regression.

          The other is to assume newcomers mostly will try docker pipeline using official images. So we offer a solution which works out-of-the box, and can't report issue with target image if detected not to match our requirements. Those who are already used with docker-pipeline then get a documented direction on how to fix their image or update their pipeline.

          I welcome any suggestion for a 3rd option. As one can't "docker exec" in a stopped container, we have to find some way for the container to run a "pause" command, and this is definitively not part of docker spec to support this.

          Nicolas De Loof added a comment - @myoung34 you're perfectly right, this is not a requirement but a convention on official image (it's required to get official image approved), and many custom image don't follow this. Our issue here is we are abusing the container lifecycle : an arbitrary docker image is  not designed to run arbitrary command like we use them for. One option is to override entrypoint to run a custom command. But then we disable the initial entrypoint designed by image author, which in many cases is required for the image to make any sense. Typically, Selenium images do use it to run a X11 server. This option has been adopted in the past, introducing https://issues.jenkins-ci.org/browse/JENKINS-41316  regression. The other is to assume newcomers mostly will try docker pipeline using official images. So we offer a solution which works out-of-the box, and can't report issue with target image if detected not to match our requirements. Those who are already used with docker-pipeline then get a documented direction on how to fix their image or update their pipeline. I welcome any suggestion for a 3rd option. As one can't " docker exec " in a stopped container, we  have to find some way for the container to run a "pause" command, and this is definitively not part of docker spec to support this.

          Nicolas De Loof added a comment -

          abayer 

          > "abusing container lifecycle" is, frankly, irrelevant here. What matters is that we keep from breaking users any more than absolutely necessary

          Sorry to say it has bee absolutely necessary. We need entrypoint support for many major use-cases, disabling them was a breaking change. It just has been restored, and introduced diagnostic code to assist end-user in fixing his pipeline/image.

          And we do abuse container lifecycle forcing the container to run a command it has not been designed for.

           

          Nicolas De Loof added a comment - abayer   > "abusing container lifecycle" is, frankly, irrelevant here. What matters is that we keep from breaking users any more than absolutely necessary Sorry to say it has bee absolutely necessary. We need entrypoint support for many major use-cases, disabling them was a breaking change. It just has been restored, and introduced diagnostic code to assist end-user in fixing his pipeline/image. And we  do abuse container lifecycle forcing the container to run a command it has not been designed for.  

          Mark Russell added a comment -

          This issue is stopping up from upgrading this plugins.  Is anyone looking at it?  Is there any chance for a solution on this?

          Mark Russell added a comment - This issue is stopping up from upgrading this plugins.  Is anyone looking at it?  Is there any chance for a solution on this?

          Nicolas De Loof added a comment -

          docker-workflow plugin comes with constraints, which unfortunately never have been clearly documented.

          Those are pretty comparable to https://github.com/knative/docs/blob/master/build/builder-contract.md

          But there's no way we support arbitrary docker images and entrypoint script. Adapt your docker images so they fit into our model

          Nicolas De Loof added a comment - docker-workflow plugin comes with constraints, which unfortunately never have been clearly documented. Those are pretty comparable to https://github.com/knative/docs/blob/master/build/builder-contract.md But there's no way we support arbitrary docker images and entrypoint script. Adapt your docker images so they fit into our model

            ndeloof Nicolas De Loof
            rtyler R. Tyler Croy
            Votes:
            13 Vote for this issue
            Watchers:
            19 Start watching this issue

              Created:
              Updated:
              Resolved: