Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63449

Process apparently never started when using two Docker agents

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • Ubuntu Xenial, Jenkins 2.239 with all plugins up-to-date

      The following Jenkins works fine:

       

      pipeline {
          agent none
          stages {
              stage('Test') {
                  agent {
                      docker {
                          image 'node:14-alpine'
                          alwaysPull true
                      }
                  }
                  steps {
                      sh 'pwd'
                      sh 'node --version'
                  }
              }
          }
      }
      

       

      The following got some issues and it's stuck on "docker pull node:14-alpine" (from Blue Ocean view):

       

       

      pipeline {
          agent {
              dockerfile {
                  filename 'docker/jenkins-agent/Dockerfile'
                  additionalBuildArgs "--build-arg UID=113"
              }
          }
      
      stages {
          stage('Test') {
                agent {
                    docker {
                        image 'node:14-alpine'
                        alwaysPull true
                    }
                }
                steps {
                    sh 'node --version'
                }
            }
          }
      }

       

      When tried without "alwaysPull true", it got stuck on "docker inspect -f . node:14-alpine — Shell Script".

      My Dockerfile:

       

      FROM ubuntu:focal AS ubuntu
      ENV DEBIAN_FRONTEND noninteractive
      # Creates a user.
      ARG UID=1001
      RUN useradd -rm -d /home/jenkins -s /bin/bash -g root -u $UID jenkins
      USER jenkins
      WORKDIR /home/jenkins
      

       

      Jenkins output:

      ...
      [Pipeline] withEnv
      [Pipeline] {
      [Pipeline] isUnix
      [Pipeline] sh (
      (hang)

      Error after timeout (docker pull node:14-alpine — Shell Script):

      process apparently never started in /var/lib/jenkins/workspace/Deploy@2@tmp/durable-75354819 (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) script returned exit code -2
      

      My understanding is when specifying "agent" inside "stage" it should override/replace the outer "agent", but it seems Jenkins got some issues with this configuration.

          [JENKINS-63449] Process apparently never started when using two Docker agents

          I got the same error after an upgrade of Jenkins and a bunch of plugins (including durable). I'm not using Docker. But it happens when in a stage with an explicit node{ sh ... } to hand over to a slave. But I have only got the error on one particular slave, it seams to work for a different slave. I don't know what the difference is.

          This is the error:
          process apparently never started in C:\Jenkins\workspace\APPBUILDER-MONGOOSE-PIPELINE-DEVELOP@tmp\durable-b7aab64d
          (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer)

          I have now downgraded to version 1.34, and it seams to help. But I not 100% sure yet, if this error is possibly intermittent in nature.

          Thomas Gronwall added a comment - I got the same error after an upgrade of Jenkins and a bunch of plugins (including durable). I'm not using Docker. But it happens when in a stage with an explicit node{ sh ... } to hand over to a slave. But I have only got the error on one particular slave, it seams to work for a different slave. I don't know what the difference is. This is the error: process apparently never started in C:\Jenkins\workspace\APPBUILDER-MONGOOSE-PIPELINE-DEVELOP@tmp\durable-b7aab64d (running Jenkins temporarily with -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true might make the problem clearer) I have now downgraded to version 1.34, and it seams to help. But I not 100% sure yet, if this error is possibly intermittent in nature.

          I have a similar issue and what introduced it in my case was the UID.

          In your file:

          additionalBuildArgs "--build-arg UID=113"
          

          Will give something like

          docker run -t -d -u 111:117 -u 113
          

          The duplication of the -u argument seems to cause the hang.

          Craig Lawton-Devine added a comment - I have a similar issue and what introduced it in my case was the UID. In your file: additionalBuildArgs "--build-arg UID=113" Will give something like docker run -t -d -u 111:117 -u 113 The duplication of the -u argument seems to cause the hang.

          Rob McGee added a comment -

          This doesn't work for us either on the latest Jenkins build and plugins, which we've only just migrated to from another CI system.

          This is a critical use case for us in testing our client-server applications and works fine when using non-dockerized agents.

          Can we get some attention on this?

          Rob McGee added a comment - This doesn't work for us either on the latest Jenkins build and plugins, which we've only just migrated to from another CI system. This is a critical use case for us in testing our client-server applications and works fine when using non-dockerized agents. Can we get some attention on this?

          Arkadiusz added a comment - - edited

          Any news on this?

          After setting org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true in https://<JENKINS>/manage/script I get:

          OCI runtime exec failed: exec failed: unable to start container process: chdir to cwd ("/home/jenkins/workspace/<PROJECT>_PR-5@2") set in config.json failed: no such file or directory: unknown

          This tells me that Jenkins tries to run sh script defined in the stage in the running docker container started in the previous stage, not in the new requested docker container.

          It can be reproduced like that:

          $ docker run -it --rm -d ubuntu
          a9c2ca6ccff4a2c016117b60c378776f626dfbe278f6366a927d41f9e89694c2
          $ docker exec -w /abcde a9c2ca6ccff4a2c016117b60c378776f626dfbe278f6366a927d41f9e89694c2 date
          OCI runtime exec failed: exec failed: unable to start container process: chdir to cwd ("/abcde") set in config.json failed: no such file or directory: unknown
          

          I think Jenkins tries to run exec in an existing container because the new requested image isn't even pulled.
           

          Arkadiusz added a comment - - edited Any news on this? After setting org.jenkinsci.plugins.durabletask.BourneShellScript.LAUNCH_DIAGNOSTICS=true in https://<JENKINS>/manage/script I get: OCI runtime exec failed: exec failed: unable to start container process: chdir to cwd ("/home/jenkins/workspace/<PROJECT>_PR-5@2") set in config.json failed: no such file or directory: unknown This tells me that Jenkins tries to run sh script defined in the stage in the running docker container started in the previous stage, not in the new requested docker container. It can be reproduced like that: $ docker run -it --rm -d ubuntu a9c2ca6ccff4a2c016117b60c378776f626dfbe278f6366a927d41f9e89694c2 $ docker exec -w /abcde a9c2ca6ccff4a2c016117b60c378776f626dfbe278f6366a927d41f9e89694c2 date OCI runtime exec failed: exec failed: unable to start container process: chdir to cwd ("/abcde") set in config.json failed: no such file or directory: unknown I think Jenkins tries to run exec in an existing container because the new requested image isn't even pulled.  

            Unassigned Unassigned
            kenorb kenorb
            Votes:
            9 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated: