Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-60473

Docker pipeline: 'absolute path' error when running linux container under windows host

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • docker-workflow-plugin
    • None
    • Jenkins 2.164.1
      Docker Pipeline 1.21
      Pipeline 2.6
      Windows 10

      I have a jenkins master on linux, a slave agent on windows 10. On the slave there is docker desktop installed. I can run commands under this docker from windows command line, for example:

      docker run --rm -v c:/Users/modtest:/home/pkruk node:alpine sh -c "ls -la"
      total 64
      drwxr-xr-x 1 root root 4096 Dec 12 17:09 .
      drwxr-xr-x 1 root root 4096 Dec 12 17:09 ..
      -rwxr-xr-x 1 root root 0 Dec 12 17:09 .dockerenv
      [...] 

      But when I try to execute following pipeline:

      pipeline {
          agent {
              docker { image 'node:alpine' }
          }
          stages {
              stage('Build') {
                  environment {
                      HOME = '.'
                  }
                  steps {
                      sh 'rm -f *.vsix'
                  }
                  post {
                      always {
                          archiveArtifacts artifacts: '*.vsix', fingerprint: true
                      }
                  }
              }
          }
      } 

      I get following error:

      Started by user pkruk
      Obtained Jenkinsfile from git [...]
      Running in Durability level: PERFORMANCE_OPTIMIZED
      [Pipeline] Start of Pipeline
      [Pipeline] node
      Running on piwon in C:\Jenkins\workspace\priv.pkruk
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (Declarative: Checkout SCM)
      [Pipeline] checkout
      [...]
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] withEnv
      [Pipeline] {
       > git rev-list --no-walk c05145e1abded6851debb54aa8e76bb1151e3fad # timeout=10
      [Pipeline] isUnix
      [Pipeline] batC:\Jenkins\workspace\priv.pkruk>docker inspect -f . node:alpine 
      .
      [Pipeline] withDockerContainer
      piwon does not seem to be running inside a container
      [Pipeline] // withDockerContainer
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      java.io.IOException: Failed to run image 'node:alpine'. Error: docker: Error response from daemon: the working directory 'C:/Jenkins/workspace/priv.pkruk/' is invalid, it needs to be an absolute path.
      See 'docker run --help'.
      	at org.jenkinsci.plugins.docker.workflow.client.WindowsDockerClient.run(WindowsDockerClient.java:57)
      	at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:199)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:286)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:179)
      	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
      	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
      	at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(Docker.groovy:126)
      	at org.jenkinsci.plugins.docker.workflow.Docker.node(Docker.groovy:66)
      	at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(Docker.groovy:114)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.DockerPipelineScript.runImage(DockerPipelineScript.groovy:54)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.configureRegistry(AbstractDockerPipelineScript.groovy:73)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.run(AbstractDockerPipelineScript.groovy:52)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(CheckoutScript.groovy:61)
      	at ___cps.transform___(Native Method)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
      	at sun.reflect.GeneratedMethodAccessor325.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:405)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:317)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:281)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE 

      I found a couple of SO posts with the same error but they offer no real solution:

      https://stackoverflow.com/questions/58447566/jenkins-pipeline-docker-agent-docker-run-error-failed-to-run-image-java-io

      https://stackoverflow.com/questions/56136415/jenkins-windows-docker-fails-to-run-image-working-directory-is-invalid-it-need

       

          [JENKINS-60473] Docker pipeline: 'absolute path' error when running linux container under windows host

          Robert Hecker added a comment - - edited

          Hello Together,

          i have the same problem when i try to execute the example from https://jenkins.io/doc/book/pipeline/docker/ on a Windows 10 Jenkins Master.

          The Pipline i use is the one from the example: 

          Jenkinsfile (Declarative Pipeline)pipeline {
              agent {
                  docker { image 'node:7-alpine' }
              }
              stages {
                  stage('Test') {
                      steps {
                          sh 'node --version'
                      }
                  }
              }
          }
          

          And the Error Message i get is following:

          C:\Program Files (x86)\Jenkins\workspace\docker-jenkins-test_master>docker inspect -f . python:3.7-alpine 
          .
          [Pipeline] withDockerContainer
          Jenkins does not seem to be running inside a container
          $ docker run -d -t -w "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/" -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/:C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/" -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master@tmp/:C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master@tmp/" -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** python:3.7-alpine cmd.exe
          [Pipeline] // withDockerContainer
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          java.io.IOException: Failed to run image 'python:3.7-alpine'. Error: docker: Error response from daemon: the working directory 'C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/' is invalid, it needs to be an absolute path.
          See 'docker run --help'.
          	at org.jenkinsci.plugins.docker.workflow.client.WindowsDockerClient.run(WindowsDockerClient.java:57)
          	at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:199)
          	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:286)
          	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:179)
          	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
          	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48)
          	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
          	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
          	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
          	at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(Docker.groovy:126)
          	at org.jenkinsci.plugins.docker.workflow.Docker.node(Docker.groovy:66)
          	at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(Docker.groovy:114)
          	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.DockerPipelineScript.runImage(DockerPipelineScript.groovy:54)
          	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.configureRegistry(AbstractDockerPipelineScript.groovy:73)
          	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.run(AbstractDockerPipelineScript.groovy:52)
          	at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(CheckoutScript.groovy:61)
          	at ___cps.transform___(Native Method)
          	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:84)
          	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
          	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
          	at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          	at java.lang.reflect.Method.invoke(Unknown Source)
          	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
          	at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
          	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
          	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
          	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
          	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
          	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
          	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
          	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
          	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
          	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:186)
          	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:370)
          	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:93)
          	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:282)
          	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:270)
          	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:66)
          	at java.util.concurrent.FutureTask.run(Unknown Source)
          	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
          	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
          	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
          	at java.util.concurrent.FutureTask.run(Unknown Source)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          	at java.lang.Thread.run(Unknown Source)
          Finished: FAILURE

          When i look to the Error Message, it seems there are multiple problems inside:

          1. The Docker Container is a Linux Container, so the option "-w" should be a linux path pointing inside the container to a existing folder. (e.g.  /usr/src/app)
          2. The volume mount of the container is wrong. Jenkins tries to mount a host Windows path to a container Linux path, using for both paths the windows path style, which is not working. (Is Situation: -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/:C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/") (Should Situation example: -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/:/docker-jenkins-test_master/")

          Robert Hecker added a comment - - edited Hello Together, i have the same problem when i try to execute the example from  https://jenkins.io/doc/book/pipeline/docker/  on a Windows 10 Jenkins Master. The Pipline i use is the one from the example:  Jenkinsfile (Declarative Pipeline)pipeline { agent { docker { image 'node:7-alpine' } } stages { stage( 'Test' ) { steps { sh 'node --version' } } } } And the Error Message i get is following: C:\Program Files (x86)\Jenkins\workspace\docker-jenkins-test_master>docker inspect -f . python:3.7-alpine . [Pipeline] withDockerContainer Jenkins does not seem to be running inside a container $ docker run -d -t -w "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/" -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/:C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/" -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master@tmp/:C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master@tmp/" -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** python:3.7-alpine cmd.exe [Pipeline] // withDockerContainer [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline java.io.IOException: Failed to run image 'python:3.7-alpine' . Error: docker: Error response from daemon: the working directory 'C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/' is invalid, it needs to be an absolute path. See 'docker run --help' . at org.jenkinsci.plugins.docker.workflow.client.WindowsDockerClient.run(WindowsDockerClient.java:57) at org.jenkinsci.plugins.docker.workflow.WithContainerStep$Execution.start(WithContainerStep.java:199) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:286) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:179) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:48) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20) at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(Docker.groovy:126) at org.jenkinsci.plugins.docker.workflow.Docker.node(Docker.groovy:66) at org.jenkinsci.plugins.docker.workflow.Docker$Image.inside(Docker.groovy:114) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.DockerPipelineScript.runImage(DockerPipelineScript.groovy:54) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.configureRegistry(AbstractDockerPipelineScript.groovy:73) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.AbstractDockerPipelineScript.run(AbstractDockerPipelineScript.groovy:52) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.CheckoutScript.checkoutAndRun(CheckoutScript.groovy:61) at ___cps.transform___(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:84) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83) at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:186) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:370) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:93) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:282) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:270) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:66) at java.util.concurrent.FutureTask.run(Unknown Source) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang. Thread .run(Unknown Source) Finished: FAILURE When i look to the Error Message, it seems there are multiple problems inside: The Docker Container is a Linux Container, so the option "-w" should be a linux path pointing inside the container to a existing folder. (e.g.   /usr/src/app) The volume mount of the container is wrong. Jenkins tries to mount a host Windows path to a container Linux path, using for both paths the windows path style, which is not working. (Is Situation: -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/:C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/") (Should Situation example: -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/:/docker-jenkins-test_master/")

          Eric Schneider added a comment - - edited

          I've been trying to track this issue. It's a bit of a surprise because there was a big push recently to get Windows supported in version 1.21 of the plugin. The effort culminated in this pull request:

          https://github.com/jenkinsci/docker-workflow-plugin/pull/184

          I was racking my brain trying to figure out how this was so thoroughly tested, but this path issue exists. The best I can tell, anyone who successfully tested these changes was running Windows containers on a Windows host. In that scenario the paths would be valid for both the host and the container. The error reported here happens when you run Linux containers on a Windows host. An elegant fix for this would be challenging because the environment would need to be checked to see what type of containers Docker is currently configured to run.

          Eric Schneider added a comment - - edited I've been trying to track this issue. It's a bit of a surprise because there was a big push recently to get Windows supported in version 1.21 of the plugin. The effort culminated in this pull request: https://github.com/jenkinsci/docker-workflow-plugin/pull/184 I was racking my brain trying to figure out how this was so thoroughly tested, but this path issue exists. The best I can tell, anyone who successfully tested these changes was running Windows containers on a Windows host. In that scenario the paths would be valid for both the host and the container. The error reported here happens when you run Linux containers on a Windows host. An elegant fix for this would be challenging because the environment would need to be checked to see what type of containers Docker is currently configured to run.

          I've started to make code changes in my environment to investigate how deep this issue is. I updated the WindowsDockerClient class to change the volume and workDir parameters to have Linux-friendly paths. This clears the issue reported here, but hits a new wall.

          I'm testing using withDockerContainer. This function runs cmd.exe to intentially keep the container running indefinitely, but this doesn't exist in Linux containers. Changing it to run 'cat' clears this error and the docker container starts.

          Next, it reports that 'cat' isn't running in the container. This is likely due to a different implementation of listProcess in WindowsDockerClient. I'm ignoring this because it seems to just be a warning.

          Finally, when my pipeline script attempts to run sh within the container, it freezes indefinitely. I need to manually kill the docker container. I haven't had time to troubleshoot this issue yet.

          Eric Schneider added a comment - I've started to make code changes in my environment to investigate how deep this issue is. I updated the WindowsDockerClient class to change the volume and workDir parameters to have Linux-friendly paths. This clears the issue reported here, but hits a new wall. I'm testing using withDockerContainer. This function runs cmd.exe to intentially keep the container running indefinitely, but this doesn't exist in Linux containers. Changing it to run 'cat' clears this error and the docker container starts. Next, it reports that 'cat' isn't running in the container. This is likely due to a different implementation of listProcess in WindowsDockerClient. I'm ignoring this because it seems to just be a warning. Finally, when my pipeline script attempts to run sh within the container, it freezes indefinitely. I need to manually kill the docker container. I haven't had time to troubleshoot this issue yet.

          Robert Hecker added a comment -

          I'm not sure if i get you correct, but in my understanding the problem is happening only on Windows host (master or slaves), when they try to start a Linux container with a Jenkins Pipeline as you wrote in the first comment.

          Are you exectuing the Test on a Windows host, with a Linux Docker Container? I don't get the point what you mean with 'cat', because if the commandline is correct, and Jenkins will start the Linux Container on the host, my expectation would be that it will run with the exactly same behaviour as if i would run it on my commandline on the host directly. 

          For sure, now the Test Steps can not more run inside a cmd or bat command, because the jenkins should execute now the command as shell. But if we use the Jenkinsfile like this:

          pipeline {
              agent {
                  docker { image 'node:7-alpine' }
              }
              stages {
                  stage('Test') {
                      steps {
                          sh 'node --version'
                      }
                  }
              }
          }
          

          then it is already written, that a Linux container is started, and inside the Linux node we call the command sh 'node --version'.

          In my understanding the challenge is to transfer the single 'sh' calls from a windows host where we have to use cmd to inside the docker container, and this via a opened cmd, where the docker container is already started.

          I'm currently not able to help here in coding, but it would be great to have a solution available which works, but what i can offer currently is the help to perform tests, if this would help here. Maybe also more would be possible, but i'm not familiar with the whole development environment for jenkins currently.

          Robert Hecker added a comment - I'm not sure if i get you correct, but in my understanding the problem is happening only on Windows host (master or slaves), when they try to start a Linux container with a Jenkins Pipeline as you wrote in the first comment. Are you exectuing the Test on a Windows host, with a Linux Docker Container? I don't get the point what you mean with 'cat', because if the commandline is correct, and Jenkins will start the Linux Container on the host, my expectation would be that it will run with the exactly same behaviour as if i would run it on my commandline on the host directly.  For sure, now the Test Steps can not more run inside a cmd or bat command, because the jenkins should execute now the command as shell. But if we use the Jenkinsfile like this: pipeline { agent { docker { image 'node:7-alpine' } } stages { stage( 'Test' ) { steps { sh 'node --version' } } } } then it is already written, that a Linux container is started, and inside the Linux node we call the command sh 'node --version'. In my understanding the challenge is to transfer the single 'sh' calls from a windows host where we have to use cmd to inside the docker container, and this via a opened cmd, where the docker container is already started. I'm currently not able to help here in coding, but it would be great to have a solution available which works, but what i can offer currently is the help to perform tests, if this would help here. Maybe also more would be possible, but i'm not familiar with the whole development environment for jenkins currently.

          bobby013

          I'm referencing a "behind the scenes" aspect about how the docker functionality works. When you enter into a pipeline region where you're running commands inside the docker container, Jenkins will start the docker container and it stays running until you leave that code region. So the order of operations for your sample that runs "node --version" is this:

          • Start the docker container
          • Run "node --version" inside the container
          • Stop the docker container

          The method that is used to start the container is a command is run that would never quit. For Linux it runs 'cat' and for Windows it runs 'cmd.exe'. Here is the relevant code from the docker workflow plugin:

          String command = launcher.isUnix() ? "cat" : "cmd.exe";
          container = dockerClient.run(env, step.image, step.args, ws, volumes, volumesFromContainers, envReduced, dockerClient.whoAmI(), /* expected to hang until killed */ command);
          

          The issue with this is it makes the decision based on the host environment and not the container environment. This decision is baked into the plugin and isn't based on input from the pipeline script. If you're on Windows and trying to run a Linux container, it always chooses to run cmd.exe.

          Eric Schneider added a comment - bobby013 I'm referencing a "behind the scenes" aspect about how the docker functionality works. When you enter into a pipeline region where you're running commands inside the docker container, Jenkins will start the docker container and it stays running until you leave that code region. So the order of operations for your sample that runs "node --version" is this: Start the docker container Run "node --version" inside the container Stop the docker container The method that is used to start the container is a command is run that would never quit. For Linux it runs 'cat' and for Windows it runs 'cmd.exe'. Here is the relevant code from the docker workflow plugin: String command = launcher.isUnix() ? "cat" : "cmd.exe" ; container = dockerClient.run(env, step.image, step.args, ws, volumes, volumesFromContainers, envReduced, dockerClient.whoAmI(), /* expected to hang until killed */ command); The issue with this is it makes the decision based on the host environment and not the container environment. This decision is baked into the plugin and isn't based on input from the pipeline script. If you're on Windows and trying to run a Linux container, it always chooses to run cmd.exe.

          I looked further into the indefinite freezing that I mentioned in a previous comment. My changes to the volume and working directory paths do result in the docker container starting successfully. The freeze is happening during the 'sh' command since the withDockerContainer block. The 'sh' command generates a really complex inline script to track the progress of the command and capture its output. The script's command also gets written to a file and is referenced. There are multiple absolute path references which use Windows's format. These fail on the Linux side and, because no output is written to the proper location, the 'sh' command waits indefinitely for it to finish. (It will eventually time out, but would otherwise wait forever.) If I capture the command in the Jenkins logs and rework all of the paths to be Linux-friendly, the command runs in docker successfully and Jenkins immediately moves to the next pipeline command/step.

          I tried to leverage the 'ws' command to force a Linux-friendly path inside the withDockerContainer block, but this causes new problems. Namely it causes the host-side of the 'sh' command to work in the wrong place - causing the same end result. You MIGHT be able to mostly work around this problem by using a Linux-friend path (like "/Jenkins") for the workspace root for the Windows node... but some of the commands may still use backslashes for path separators and cause failures. It's also not a user-friendly solution.

          At minimum, it seems like the Decorator class used by withDockerContainer needs to comb through the launcher's commands list ("cmds") and replace all of the Windows paths with Linux-friendly paths. A regular expression might be able to do it, but I haven't tried yet. It's tricky with the amount of text being added to the command list. Maybe someone with more knowledge of Jenkins has a more elegant solution.

          Eric Schneider added a comment - I looked further into the indefinite freezing that I mentioned in a previous comment. My changes to the volume and working directory paths do result in the docker container starting successfully. The freeze is happening during the 'sh' command since the withDockerContainer block. The 'sh' command generates a really complex inline script to track the progress of the command and capture its output. The script's command also gets written to a file and is referenced. There are multiple absolute path references which use Windows's format. These fail on the Linux side and, because no output is written to the proper location, the 'sh' command waits indefinitely for it to finish. (It will eventually time out, but would otherwise wait forever.) If I capture the command in the Jenkins logs and rework all of the paths to be Linux-friendly, the command runs in docker successfully and Jenkins immediately moves to the next pipeline command/step. I tried to leverage the 'ws' command to force a Linux-friendly path inside the withDockerContainer block, but this causes new problems. Namely it causes the host-side of the 'sh' command to work in the wrong place - causing the same end result. You MIGHT be able to mostly work around this problem by using a Linux-friend path (like "/Jenkins") for the workspace root for the Windows node... but some of the commands may still use backslashes for path separators and cause failures. It's also not a user-friendly solution. At minimum, it seems like the Decorator class used by withDockerContainer needs to comb through the launcher's commands list ("cmds") and replace all of the Windows paths with Linux-friendly paths. A regular expression might be able to do it, but I haven't tried yet. It's tricky with the amount of text being added to the command list. Maybe someone with more knowledge of Jenkins has a more elegant solution.

          I created a fork which contains changes that resolve this issue for my use case. I'd be curious if anyone else could test the changes.

          https://github.com/mrsonicblue/docker-workflow-plugin

          Eric Schneider added a comment - I created a fork which contains changes that resolve this issue for my use case. I'd be curious if anyone else could test the changes. https://github.com/mrsonicblue/docker-workflow-plugin

          Ralph Connors added a comment - - edited

          Hi Eric,

          I have run into the same issue as the original poster. I would like to test your fork - how do I integrate the plugin on Github into Jenkins?

          Ralph Connors added a comment - - edited Hi Eric, I have run into the same issue as the original poster. I would like to test your fork - how do I integrate the plugin on Github into Jenkins?

          rjconnors

          I'm going to email you a link to a built version of the code. I don't know if I should post a direct link here. But if you're willing to give it a try:

          You can upload the plugin by navigating "Manage Jenkins" > "Manage Plugins" > "Advanced" tab > "Upload Plugin". Make sure you only do it once, because the Jenkins interface will let you revert back one version, but not more than one. You can revert back to the standard plugin if my build makes things worse.

          Eric Schneider added a comment - rjconnors I'm going to email you a link to a built version of the code. I don't know if I should post a direct link here. But if you're willing to give it a try: You can upload the plugin by navigating "Manage Jenkins" > "Manage Plugins" > "Advanced" tab > "Upload Plugin". Make sure you only do it once, because the Jenkins interface will let you revert back one version, but not more than one. You can revert back to the standard plugin if my build makes things worse.

          Robert Hecker added a comment -

          Hi Eric,

          when you send me a link to your build version of your plugin, i can test it also on my environment.

          Please let me know, if i can help.

          Robert Hecker added a comment - Hi Eric, when you send me a link to your build version of your plugin, i can test it also on my environment. Please let me know, if i can help.

          Robert Hecker added a comment - - edited

          Hi Eric,

          Thank you very much for your effort, and the upload of the modified source code of the docker plugin.

          1.) I managed to get the plugin on my pc compiled.

          2.) I used for a test following pipeline as example: (From: https://jenkins.io/doc/book/pipeline/docker/)

          pipeline {
              agent {
                  docker { image 'node:7-alpine' }
              }
              stages {
                  stage('Test') {
                      steps {
                          sh 'node --version'
                      }
                  }
              }
          }
          

          3)Test execution is resulting in a failure (Exit Code -2):

          C:\Program Files (x86)\Jenkins\workspace\docker-jenkins-test_master>docker inspect -f . node:7-alpine 
          .
          [Pipeline] withDockerContainer
          Jenkins does not seem to be running inside a container
          $ docker run -d -t -w "/c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/" -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/:/c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/" -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master@tmp/:/c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master@tmp/" -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** node:7-alpine cat
          $ docker top ee78f93ed896282757e96f87cc6c73210b63e14f4a6bdc94f229db4468721db1
          [Pipeline] {
          [Pipeline] stage
          [Pipeline] { (Test)
          [Pipeline] sh
          process apparently never started in C:\Program Files (x86)\Jenkins\workspace\docker-jenkins-test_master@tmp\durable-275a00eb
          (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          $ docker stop --time=1 ee78f93ed896282757e96f87cc6c73210b63e14f4a6bdc94f229db4468721db1
          $ docker rm -f ee78f93ed896282757e96f87cc6c73210b63e14f4a6bdc94f229db4468721db1
          [Pipeline] // withDockerContainer
          [Pipeline] }
          [Pipeline] // withEnv
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          ERROR: script returned exit code -2
          Finished: FAILURE
          

          4) Is the docker option "-v" correct? (-v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/:/c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/") The bold marked path looks like an windows path, but does this path work inside a linux container ? 

          Robert Hecker added a comment - - edited Hi Eric, Thank you very much for your effort, and the upload of the modified source code of the docker plugin. 1.) I managed to get the plugin on my pc compiled. 2.) I used for a test following pipeline as example: (From:  https://jenkins.io/doc/book/pipeline/docker/ ) pipeline { agent { docker { image 'node:7-alpine' } } stages { stage( 'Test' ) { steps { sh 'node --version' } } } } 3)Test execution is resulting in a failure (Exit Code -2): C:\Program Files (x86)\Jenkins\workspace\docker-jenkins-test_master>docker inspect -f . node:7-alpine . [Pipeline] withDockerContainer Jenkins does not seem to be running inside a container $ docker run -d -t -w "/c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/" -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/:/c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/" -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master@tmp/:/c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master@tmp/" -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** node:7-alpine cat $ docker top ee78f93ed896282757e96f87cc6c73210b63e14f4a6bdc94f229db4468721db1 [Pipeline] { [Pipeline] stage [Pipeline] { (Test) [Pipeline] sh process apparently never started in C:\Program Files (x86)\Jenkins\workspace\docker-jenkins-test_master@tmp\durable-275a00eb (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS= true might make the problem clearer) [Pipeline] } [Pipeline] // stage [Pipeline] } $ docker stop --time=1 ee78f93ed896282757e96f87cc6c73210b63e14f4a6bdc94f229db4468721db1 $ docker rm -f ee78f93ed896282757e96f87cc6c73210b63e14f4a6bdc94f229db4468721db1 [Pipeline] // withDockerContainer [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline ERROR: script returned exit code -2 Finished: FAILURE 4) Is the docker option "-v" correct? (-v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/: /c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/ ") The bold marked path looks like an windows path, but does this path work inside a linux container ? 

          Eric Schneider added a comment - - edited

          bobby013

          Thank you for the detailed notes. When I run your pipeline example, it does work for me. I need to figure out what's different in your environment.

          The path that you bolded is correct. If you look at the "docker run" command in the log, you can see that the workspace path from the Windows host is being mounted via a volume to that path in the Linux container. The path is translated into something Linux-friendly. For example: C:\this\path becomes /c/this/path. Even though your path is is Windows-y, it doesn't really matter. The path can be arbitrary.

          What's unfortunate is that the sh command doesn't output the command it runs into the job log. It's difficult to capture. I'm including directions here in case you want to try. Navigate to "Managed Jenkins" > "System Log" > "Add new log recorder". Give it any name and click "OK". Under "Loggers" add "hudson" and "org.jenkinsci.plugins". Ensure that the log level is "ALL" for both. Save the recorder. This should open the log. Run your test job and right when it gets to the 'sh' step, refresh the log. Search the output for "docker" and see if you can find the "docker exec" line in the log. It can be hard to capture, since the log only shows a small number of lines. Here's an example from my log:

          docker exec --env BUILD_DISPLAY_NAME=#4 --env BUILD_ID=4 --env BUILD_NUMBER=4 --env BUILD_TAG=jenkins-Parenthesis Test (wee)-4 --env BUILD_URL=http://example.com/job/Parenthesis%20Test%20(wee)/4/ --env CLASSPATH= --env EXECUTOR_NUMBER=2 --env HUDSON_HOME=/var/lib/jenkins --env HUDSON_SERVER_COOKIE=a78bfd62731ccf0a --env HUDSON_URL=http://example.com/ --env JENKINS_NODE_COOKIE=06d630b9-7764-43ee-85e0-eb9a26bcc2d2 --env JENKINS_SERVER_COOKIE=please-do-not-kill-me --env JENKINS_URL=http://example.com/ --env JOB_BASE_NAME=Parenthesis Test (wee) --env JOB_DISPLAY_URL=http://example.com/job/Parenthesis%20Test%20(wee)/display/redirect --env JOB_NAME=Parenthesis Test (wee) --env JOB_URL=http://example.com/job/Parenthesis%20Test%20(wee)/ --env NODE_LABELS=localhost master --env NODE_NAME=master --env RUN_CHANGES_DISPLAY_URL=http://example.com/job/Parenthesis%20Test%20(wee)/4/display/redirect?page=changes --env RUN_DISPLAY_URL=http://example.com/job/Parenthesis%20Test%20(wee)/4/display/redirect --env STAGE_NAME=Test --env WORKSPACE=/var/lib/jenkins/workspace/Parenthesis Test (wee) --env library.migration-library.version=master 63c3793d8f3730cafcc37fb3e495d25ec8e520d12cf9a599342697fc64616393 nohup sh -c ({ while [ -d '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9' -a \! -f '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt' ]; do touch '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-log.txt'; sleep 3; done } & jsc=durable-27a2324005ef9d26531c4508bfcb22d4; JENKINS_SERVER_COOKIE=$jsc 'sh' -xe '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/script.sh' > '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-log.txt' 2>&1; echo $? > '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt.tmp'; mv '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt.tmp' '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt'; wait) >&- 2>&- &
          

          This would be the command where the job gets stuck and needs to be troubleshooted. You can try running this manually on your node to see if it returns a meaningful message.

          Eric Schneider added a comment - - edited bobby013 Thank you for the detailed notes. When I run your pipeline example, it does work for me. I need to figure out what's different in your environment. The path that you bolded is correct. If you look at the "docker run" command in the log, you can see that the workspace path from the Windows host is being mounted via a volume to that path in the Linux container. The path is translated into something Linux-friendly. For example: C:\this\path becomes /c/this/path. Even though your path is is Windows-y, it doesn't really matter. The path can be arbitrary. What's unfortunate is that the sh command doesn't output the command it runs into the job log. It's difficult to capture. I'm including directions here in case you want to try. Navigate to "Managed Jenkins" > "System Log" > "Add new log recorder". Give it any name and click "OK". Under "Loggers" add "hudson" and "org.jenkinsci.plugins". Ensure that the log level is "ALL" for both. Save the recorder. This should open the log. Run your test job and right when it gets to the 'sh' step, refresh the log. Search the output for "docker" and see if you can find the "docker exec" line in the log. It can be hard to capture, since the log only shows a small number of lines. Here's an example from my log: docker exec --env BUILD_DISPLAY_NAME=#4 --env BUILD_ID=4 --env BUILD_NUMBER=4 --env BUILD_TAG=jenkins-Parenthesis Test (wee)-4 --env BUILD_URL=http: //example.com/job/Parenthesis%20Test%20(wee)/4/ --env CLASSPATH= --env EXECUTOR_NUMBER=2 --env HUDSON_HOME=/ var /lib/jenkins --env HUDSON_SERVER_COOKIE=a78bfd62731ccf0a --env HUDSON_URL=http://example.com/ --env JENKINS_NODE_COOKIE=06d630b9-7764-43ee-85e0-eb9a26bcc2d2 --env JENKINS_SERVER_COOKIE=please- do -not-kill-me --env JENKINS_URL=http://example.com/ --env JOB_BASE_NAME=Parenthesis Test (wee) --env JOB_DISPLAY_URL=http://example.com/job/Parenthesis%20Test%20(wee)/display/redirect --env JOB_NAME=Parenthesis Test (wee) --env JOB_URL=http://example.com/job/Parenthesis%20Test%20(wee)/ --env NODE_LABELS=localhost master --env NODE_NAME=master --env RUN_CHANGES_DISPLAY_URL=http://example.com/job/Parenthesis%20Test%20(wee)/4/display/redirect?page=changes --env RUN_DISPLAY_URL=http://example.com/job/Parenthesis%20Test%20(wee)/4/display/redirect --env STAGE_NAME=Test --env WORKSPACE=/ var /lib/jenkins/workspace/Parenthesis Test (wee) --env library.migration-library.version=master 63c3793d8f3730cafcc37fb3e495d25ec8e520d12cf9a599342697fc64616393 nohup sh -c ({ while [ -d '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9' -a \! -f '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt' ]; do touch '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-log.txt' ; sleep 3; done } & jsc=durable-27a2324005ef9d26531c4508bfcb22d4; JENKINS_SERVER_COOKIE=$jsc 'sh' -xe '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/script.sh' > '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-log.txt' 2>&1; echo $? > '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt.tmp' ; mv '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt.tmp' '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt' ; wait) >&- 2>&- & This would be the command where the job gets stuck and needs to be troubleshooted. You can try running this manually on your node to see if it returns a meaningful message.

          Robert Hecker added a comment - - edited

          Hi Eric,

          unfortunatly i do not find a "docker exec" string in my log, and i do not find the correct positions in the log, to interpret what is going on. So i will attach my complete log to this comment, in the hope, that you are able to get the needed informations. When i read your answer, it comes to my mind, that my setup is maybe different from yours, so here some additional informations about my setup:

          I'm running a jenkins installation on windows10, and directly on this master i try to execute this docker pipeline example (From: https://jenkins.io/doc/book/pipeline/docker/) Maybe this is an big difference to your setup, because i assume now you have a jenkins master installed in linux, and execute your job an a windows slave. If my assumption is correct, i hope this gives you a hint, where the problems could be. Here the log file: Jenkins_Log.zip

           

          When i start the docker command fom the jenkins console output log manualy and type in the docker top command it locks like following:

          D:\>docker run -d -t -w "/c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/" -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/:/c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/" -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master@tmp/:/c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master@tmp/" -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** node:7-alpine cat
          e735b5e12baebfe12af2e56bea9fd031f088503d835a343b21ecb876b5caf18b
          
          D:\>docker ps
          CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
          e735b5e12bae        node:7-alpine       "cat"                    11 seconds ago      Up 10 seconds                                crazy_johnson
          
          D:\>docker top e735b5e12baebfe12af2e56bea9fd031f088503d835a343b21ecb876b5caf18b
          PID                 USER                TIME                COMMAND
          5093                root                0:00                cat
          
          D:\>
          

          So it looks like the container started, and also the internal process with the PID 5093 is running. I'm wondering, why in the jenkins console log is mentioned "process apparently never started in ..."

          [Pipeline] {
          [Pipeline] stage
          [Pipeline] { (Test)
          [Pipeline] sh
          process apparently never started in C:\Program Files (x86)\Jenkins\workspace\docker-jenkins-test_master@tmp\durable-7243594c
          (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          

          So it could be that Jenkins realy never started the "docker exec" command on my Windows Machine.

          Robert Hecker added a comment - - edited Hi Eric, unfortunatly i do not find a " docker exec " string in my log, and i do not find the correct positions in the log, to interpret what is going on. So i will attach my complete log to this comment, in the hope, that you are able to get the needed informations. When i read your answer, it comes to my mind, that my setup is maybe different from yours, so here some additional informations about my setup: I'm running a jenkins installation on windows10, and directly on this master i try to execute this docker pipeline example (From:  https://jenkins.io/doc/book/pipeline/docker/ ) Maybe this is an big difference to your setup, because i assume now you have a jenkins master installed in linux, and execute your job an a windows slave. If my assumption is correct, i hope this gives you a hint, where the problems could be. Here the log file:  Jenkins_Log.zip   When i start the docker command fom the jenkins console output log manualy and type in the docker top command it locks like following: D:\>docker run -d -t -w "/c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/" -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/:/c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master/" -v "C:/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master@tmp/:/c/Program Files (x86)/Jenkins/workspace/docker-jenkins-test_master@tmp/" -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** node:7-alpine cat e735b5e12baebfe12af2e56bea9fd031f088503d835a343b21ecb876b5caf18b D:\>docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e735b5e12bae node:7-alpine "cat" 11 seconds ago Up 10 seconds crazy_johnson D:\>docker top e735b5e12baebfe12af2e56bea9fd031f088503d835a343b21ecb876b5caf18b PID USER TIME COMMAND 5093 root 0:00 cat D:\> So it looks like the container started, and also the internal process with the PID 5093 is running. I'm wondering, why in the jenkins console log is mentioned " process apparently never started in ... " [Pipeline] { [Pipeline] stage [Pipeline] { (Test) [Pipeline] sh process apparently never started in C:\Program Files (x86)\Jenkins\workspace\docker-jenkins-test_master@tmp\durable-7243594c (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS= true might make the problem clearer) [Pipeline] } [Pipeline] // stage [Pipeline] } So it could be that Jenkins realy never started the "docker exec" command on my Windows Machine.

          Robert Hecker added a comment -

          Hi mrsonicblue,

          When i try to run the command manualy on my windows machine, where i manualy started the container like above, i get following error code:

          D:\>docker exec --env BUILD_DISPLAY_NAME=#4 --env BUILD_ID=4 --env BUILD_NUMBER=4 --env BUILD_TAG=jenkins-Parenthesis Test (wee)-4 --env BUILD_URL=http://example.com/job/Parenthesis%20Test%20(wee)/4/ --env CLASSPATH= --env EXECUTOR_NUMBER=2 --env HUDSON_HOME=/var/lib/jenkins --env HUDSON_SERVER_COOKIE=a78bfd62731ccf0a --env HUDSON_URL=http://example.com/ --env JENKINS_NODE_COOKIE=06d630b9-7764-43ee-85e0-eb9a26bcc2d2 --env JENKINS_SERVER_COOKIE=please-do-not-kill-me --env JENKINS_URL=http://example.com/ --env JOB_BASE_NAME=Parenthesis Test (wee) --env JOB_DISPLAY_URL=http://example.com/job/Parenthesis%20Test%20(wee)/display/redirect --env JOB_NAME=Parenthesis Test (wee) --env JOB_URL=http://example.com/job/Parenthesis%20Test%20(wee)/ --env NODE_LABELS=localhost master --env NODE_NAME=master --env RUN_CHANGES_DISPLAY_URL=http://example.com/job/Parenthesis%20Test%20(wee)/4/display/redirect?page=changes --env RUN_DISPLAY_URL=http://example.com/job/Parenthesis%20Test%20(wee)/4/display/redirect --env STAGE_NAME=Test --env WORKSPACE=/var/lib/jenkins/workspace/Parenthesis Test (wee) --env library.migration-library.version=master 63c3793d8f3730cafcc37fb3e495d25ec8e520d12cf9a599342697fc64616393 nohup sh -c ({ while [ -d '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9' -a \! -f '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt' ]; do touch '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-log.txt'; sleep 3; done } & jsc=durable-27a2324005ef9d26531c4508bfcb22d4; JENKINS_SERVER_COOKIE=$jsc 'sh' -xe '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/script.sh' > '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-log.txt' 2>&1; echo $? > '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt.tmp'; mv '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt.tmp' '/var/lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt'; wait) >&- 2>&- &
          ">&" kann syntaktisch an dieser Stelle nicht verarbeitet werden.
          

          To get the expected output i tried it with this "docker exec" command:

          D:\>docker exec -it e735b5e12baebfe12af2e56bea9fd031f088503d835a343b21ecb876b5caf18b node --version
          v7.10.1
          

          This delivered me the expected version information as output. Hope this information help you for your investigation.  !

          Robert Hecker added a comment - Hi mrsonicblue , When i try to run the command manualy on my windows machine, where i manualy started the container like above, i get following error code: D:\>docker exec --env BUILD_DISPLAY_NAME=#4 --env BUILD_ID=4 --env BUILD_NUMBER=4 --env BUILD_TAG=jenkins-Parenthesis Test (wee)-4 --env BUILD_URL=http: //example.com/job/Parenthesis%20Test%20(wee)/4/ --env CLASSPATH= --env EXECUTOR_NUMBER=2 --env HUDSON_HOME=/ var /lib/jenkins --env HUDSON_SERVER_COOKIE=a78bfd62731ccf0a --env HUDSON_URL=http://example.com/ --env JENKINS_NODE_COOKIE=06d630b9-7764-43ee-85e0-eb9a26bcc2d2 --env JENKINS_SERVER_COOKIE=please- do -not-kill-me --env JENKINS_URL=http://example.com/ --env JOB_BASE_NAME=Parenthesis Test (wee) --env JOB_DISPLAY_URL=http://example.com/job/Parenthesis%20Test%20(wee)/display/redirect --env JOB_NAME=Parenthesis Test (wee) --env JOB_URL=http://example.com/job/Parenthesis%20Test%20(wee)/ --env NODE_LABELS=localhost master --env NODE_NAME=master --env RUN_CHANGES_DISPLAY_URL=http://example.com/job/Parenthesis%20Test%20(wee)/4/display/redirect?page=changes --env RUN_DISPLAY_URL=http://example.com/job/Parenthesis%20Test%20(wee)/4/display/redirect --env STAGE_NAME=Test --env WORKSPACE=/ var /lib/jenkins/workspace/Parenthesis Test (wee) --env library.migration-library.version=master 63c3793d8f3730cafcc37fb3e495d25ec8e520d12cf9a599342697fc64616393 nohup sh -c ({ while [ -d '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9' -a \! -f '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt' ]; do touch '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-log.txt' ; sleep 3; done } & jsc=durable-27a2324005ef9d26531c4508bfcb22d4; JENKINS_SERVER_COOKIE=$jsc 'sh' -xe '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/script.sh' > '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-log.txt' 2>&1; echo $? > '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt.tmp' ; mv '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt.tmp' '/ var /lib/jenkins/workspace/Parenthesis Test (wee)@tmp/durable-11d2bcc9/jenkins-result.txt' ; wait) >&- 2>&- & ">&" kann syntaktisch an dieser Stelle nicht verarbeitet werden. To get the expected output i tried it with this " docker exec " command: D:\>docker exec -it e735b5e12baebfe12af2e56bea9fd031f088503d835a343b21ecb876b5caf18b node --version v7.10.1 This delivered me the expected version information as output. Hope this information help you for your investigation.   !

          Robert Hecker added a comment -

          Hello Together,

          in the meantime i found more details why it is not working on a Jenkins Master which is installed on Windows. I got now extended messages on the execution log, which points more clearer to the Problem. The interesting part of the log is:

          [Pipeline] withDockerContainer
          Jenkins does not seem to be running inside a container
          $ docker run -d -t -w /d/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test/ -v D:/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test/:/d/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test/ -v D:/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test@tmp/:/d/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test@tmp/ -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** node:7-alpine cat
          $ docker top 0c8e23cea55f3e827eb013a74fed9b28e0585a5a243bef9ec897441dfc320a60
          [Pipeline] {
          [Pipeline] stage
          [Pipeline] { (Test)
          [Pipeline] sh
          sh: can't create /d/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test@tmp/durable-f0f56139/jenkins-log.txt: nonexistent directory
          sh: can't create /d/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test@tmp/durable-f0f56139/jenkins-result.txt.tmp: nonexistent directory
          mv: can't rename '/d/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test@tmp/durable-f0f56139/jenkins-result.txt.tmp': No such file or directory
          process apparently never started in D:\Sandbox_GIT\Jenkins_Plugin\docker-workflow-plugin-master\work\workspace\test@tmp\durable-f0f56139
          [Pipeline] }
          

          For me it seems that some folders are not existing, and due this he can not create the file. 

          Following 2 Folders exist on my Windows Master:

          • D:\Sandbox_GIT\Jenkins_Plugin\docker-workflow-plugin-master\work\workspace\test
          • D:\Sandbox_GIT\Jenkins_Plugin\docker-workflow-plugin-master\work\workspace\test@tmp

          What needs to be changed in the Plugin, to get this working?

          Robert Hecker added a comment - Hello Together, in the meantime i found more details why it is not working on a Jenkins Master which is installed on Windows. I got now extended messages on the execution log, which points more clearer to the Problem. The interesting part of the log is: [Pipeline] withDockerContainer Jenkins does not seem to be running inside a container $ docker run -d -t -w /d/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test/ -v D:/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test/:/d/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test/ -v D:/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test@tmp/:/d/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test@tmp/ -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** node:7-alpine cat $ docker top 0c8e23cea55f3e827eb013a74fed9b28e0585a5a243bef9ec897441dfc320a60 [Pipeline] { [Pipeline] stage [Pipeline] { (Test) [Pipeline] sh sh: can't create /d/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test@tmp/durable-f0f56139/jenkins-log.txt: nonexistent directory sh: can't create /d/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test@tmp/durable-f0f56139/jenkins-result.txt.tmp: nonexistent directory mv: can 't rename ' /d/Sandbox_GIT/Jenkins_Plugin/docker-workflow-plugin-master/work/workspace/test@tmp/durable-f0f56139/jenkins-result.txt.tmp': No such file or directory process apparently never started in D:\Sandbox_GIT\Jenkins_Plugin\docker-workflow-plugin-master\work\workspace\test@tmp\durable-f0f56139 [Pipeline] } For me it seems that some folders are not existing, and due this he can not create the file.  Following 2 Folders exist on my Windows Master: D:\Sandbox_GIT\Jenkins_Plugin\docker-workflow-plugin-master\work\workspace\test D:\Sandbox_GIT\Jenkins_Plugin\docker-workflow-plugin-master\work\workspace\test@tmp What needs to be changed in the Plugin, to get this working?

          Robert Hecker added a comment -

          i saw that the subdirectories on the Windows host where created ("\durable-f0f56139") and a .sh script is inside there with the command from the pipeline *"node --version" is inside.

          So the next question would be where, and with which parameters does the plugin call the docker exec command? 

          Can this found in a jenkins log ? When yes, how to configure the log to get this information ?

          Any help would be great! Thanks a lot in advance!

          Robert Hecker added a comment - i saw that the subdirectories on the Windows host where created ( "\durable-f0f56139" ) and a .sh script is inside there with the command from the pipeline *"node --version" is inside. So the next question would be where, and with which parameters does the plugin call the docker exec command?  Can this found in a jenkins log ? When yes, how to configure the log to get this information ? Any help would be great! Thanks a lot in advance!

          James Pan added a comment -

          Hi All,

          I am facing the same issue now and wondering if the problem has been solved? When I use Docker Desktop on windows, I usually use the following argument to map a path on my window host to a linux docker

          -v /d/jenkins_data:/var/jenkins_home

          I am wondering if this is the issue course the error because I have searched on the internet and lots of post described docker doesn't support path format in windows format.

          James Pan added a comment - Hi All, I am facing the same issue now and wondering if the problem has been solved? When I use Docker Desktop on windows, I usually use the following argument to map a path on my window host to a linux docker -v /d/jenkins_data:/ var /jenkins_home I am wondering if this is the issue course the error because I have searched on the internet and lots of post described docker doesn't support path format in windows format.

          James Pan added a comment -

          I thing I have found the root course. When I use a docker image with the same type of the host, it's fine, however, when I use a docker image which is different from the host, this plugin will have exceptions. For example, if the host is a windows and wanted to use a linux image as the slave, then working directory will not work because D:\ or C:\ is not a absolution path in the linux system, that must be changed to /d or /c.

          However, since the host is a windows, the WindowsDockerClient will be used and cmd.exe will be run in the linux image, and that will lead to another exception. The conclusion is to use an image which align with the host.

          I haven't tested on a Mac but I think Mac<>Linux will be fine but Mac<>Windows will still have the same issue.

          James Pan added a comment - I thing I have found the root course. When I use a docker image with the same type of the host, it's fine, however, when I use a docker image which is different from the host, this plugin will have exceptions. For example, if the host is a windows and wanted to use a linux image as the slave, then working directory will not work because D:\ or C:\ is not a absolution path in the linux system, that must be changed to /d or /c. However, since the host is a windows, the WindowsDockerClient will be used and cmd.exe will be run in the linux image, and that will lead to another exception. The conclusion is to use an image which align with the host. I haven't tested on a Mac but I think Mac< >Linux will be fine but Mac< >Windows will still have the same issue.

          Alex added a comment -

          Thanks for all the hard work here but I am still at a loss to understand how to resolve the issue with Jenkins or Docker not figuring out which path to use.  I have the exact error as started in this post.  I am running Jenkins on WIndows and the goal is to run a linux container on a windows host (running a linux vm).   What I noticed when modifying the config.xml to a path like /c/users/mydir/ the output produced is $ docker run -d -t -w C:/c/users/mydir.  All I was trying to do is figure out what configuration I can change to convert it to a path that linux likes and I cannot figure out how to tell Jenkins to create the right path.  I know what I did is not the solution (again trying to repro the problem).

          Is there a single configuration item somewhere that tells jenkins to do this

          $ docker run -d -t -w /c/users/

          instead of this?

          $ docker run -d -t -w C:/users/

           

          Alex added a comment - Thanks for all the hard work here but I am still at a loss to understand how to resolve the issue with Jenkins or Docker not figuring out which path to use.  I have the exact error as started in this post.  I am running Jenkins on WIndows and the goal is to run a linux container on a windows host (running a linux vm).   What I noticed when modifying the config.xml to a path like /c/users/mydir/ the output produced is $ docker run -d -t -w C: /c/users/mydir.  All I was trying to do is figure out what configuration I can change to convert it to a path that linux likes and I cannot figure out how to tell Jenkins to create the right path.  I know what I did is not the solution (again trying to repro the problem). Is there a single configuration item somewhere that tells jenkins to do this $ docker run -d -t -w /c/users/ instead of this? $ docker run -d -t -w C:/users/  

          shim added a comment -

          The updates Eric Schneider made in https://github.com/mrsonicblue/docker-workflow-plugin fixes the issue.  If using an older version of Docker you'll get a "The container started but didn't run the expected command" or "process apparently never started" due to how BusyBox returns output from the "top" command.  BusyBox 'ps' command prints the cat command as "{cat}", so it looks like it never spawned.    I created a PR (https://github.com/mrsonicblue/docker-workflow-plugin/pull/1) that changes the code to also look for {cat} and [cat].   This should make it work in the general case

           

           

          shim added a comment - The updates Eric Schneider made in  https://github.com/mrsonicblue/docker-workflow-plugin  fixes the issue.  If using an older version of Docker you'll get a "The container started but didn't run the expected command" or "process apparently never started" due to how BusyBox returns output from the "top" command.  BusyBox 'ps' command prints the cat command as "{cat}", so it looks like it never spawned.    I created a PR ( https://github.com/mrsonicblue/docker-workflow-plugin/pull/1)  that changes the code to also look for {cat} and [cat] .   This should make it work in the general case    

          Mate Rigo added a comment -

          Thanks shim for your response.

          The only thing I do not understand: why weren't any of the pull requests approved?

          https://github.com/jenkinsci/docker-workflow-plugin/pull/197

          https://github.com/jenkinsci/docker-workflow-plugin/pull/234

          https://github.com/jenkinsci/docker-workflow-plugin/pull/235

          I mean this is a serious issue, for which there seems to be a solution.

           

          This was my intention to have a single slave machine which is running win10 and every build job would be performed in a docker container regardless if it was a windows native or linux native build.

          In the end this is the ultimate promise of containerization to once create the containers and execute them regardless of the host.

          Cheers!

          Mate Rigo added a comment - Thanks shim for your response. The only thing I do not understand: why weren't any of the pull requests approved? https://github.com/jenkinsci/docker-workflow-plugin/pull/197 https://github.com/jenkinsci/docker-workflow-plugin/pull/234 https://github.com/jenkinsci/docker-workflow-plugin/pull/235 I mean this is a serious issue, for which there seems to be a solution.   This was my intention to have a single slave machine which is running win10 and every build job would be performed in a docker container regardless if it was a windows native or linux native build. In the end this is the ultimate promise of containerization to once create the containers and execute them regardless of the host. Cheers!

          Hi, this issue is a serios blocker for us. When should we expect it to be published?

          Thank you!

          Bulent Karaahmed added a comment - Hi, this issue is a serios blocker for us. When should we expect it to be published? Thank you!

          Lachlan added a comment -

          I have found the solution to this issue, and will be putting in a pull request shortly.

          Lachlan added a comment - I have found the solution to this issue, and will be putting in a pull request shortly.

            Unassigned Unassigned
            pkruk Piotr Krukowiecki
            Votes:
            15 Vote for this issue
            Watchers:
            20 Start watching this issue

              Created:
              Updated: