Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54389

Containers with ENTRYPOINT is not started correctly

      I try to add some service startup to Dockerfile ENTRYPOINT.

      But Jenkins starts container with `cat` CMD and immediately check `cat` in `docker ps` output. As result, there are race condition between `docker ps` and ENTRYPOINT script.

      I created minimalistic example for reproducing this issue on stock Jenkins: https://github.com/bozaro/jenkins-docker-issue

      This example contains:

      1. Dockerfile declaring ENTRYPOINT:
      FROM alpine:latest
      COPY entry.sh /opt/bin/
      ENTRYPOINT ["/opt/bin/entry.sh"]
      CMD ["/bin/sh"] 
      1. ENTRYPOINT script with 3 seconds delay and creating marker file:
      #!/bin/sh -ex
      # Starting very important services...
      sleep 3
      # Service is ready now
      echo > /tmp/ready
      exec "$@"
      1. Jenkinsfile running container with marker file check:
      pipeline {
          agent {
              dockerfile {
                  filename 'Dockerfile'
              }
          }
          stages {
              stage('Check file') {
                  steps {
                      sh "cat /tmp/ready"
                  }
              }
          }
      }
      

      Expected result for Jenkins test:

      1. At least a 3-second delay on pipeline withDockerContainer step
      2. ENTRYPOINT script output in build output;
      3. Green build

      Actual result for Jenkins test:

      1. False positive error message:

      ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements).

      1. Failed "Check file" step because ENTRYPOINT is not completed yet and /tmp/ready file is not exists;
      2. There are no ENTRYPOINT script output in build.

      Theoretically this issue can be solved if Jenkins would send some random string marker to docker container and wait this marker in container output.

          [JENKINS-54389] Containers with ENTRYPOINT is not started correctly

          Andrew Nicols added a comment -

          This could also be solved if Jenkins were to respect a healthcheck and the plugin changed to check the health of the container.

          This could also be specified on the CLI:

          docker run -u 501:501 -t --health-cmd '[ -f /tmp/ready ]' --health-start-period 1s --health-interval 100s myimage sh -c 'touch /tmp/ready && cat'
          

          Either way we do need a way to allow slower entrypoints to do their job.

          Andrew Nicols added a comment - This could also be solved if Jenkins were to respect a healthcheck and the plugin changed to check the health of the container. This could also be specified on the CLI: docker run -u 501:501 -t --health-cmd '[ -f /tmp/ready ]' --health-start-period 1s --health-interval 100s myimage sh -c 'touch /tmp/ready && cat' Either way we do need a way to allow slower entrypoints to do their job.

          Similarly, I'm hitting this with builds of Hugo.  The container has a single binary in it: hugo.

           

          I'm watching both the race condition called out above occurring as well as the incorrect behavior of attempting to override the valid ENTRYPOINT defined in the file.

          Brian Redbeard added a comment - Similarly, I'm hitting this with builds of Hugo .  The container has a single binary in it: hugo.   I'm watching both the race condition called out above occurring as well as the incorrect behavior of attempting to override the valid ENTRYPOINT defined in the file.

          Günter Grodotzki added a comment - - edited

          hitting this issue with dind. Either a simple (configurable) delay would already help the majority of cases, or better, polling for the health-check.

           

          FWIW, I was able to solve this locally with a dirty hack:

           

          /usr/local/bin/docker:
          
          #!/usr/bin/env bash
          
          set -eo pipefail
          
          if [[ "${1}" == "top" ]]; then
           sleep 5
          fi
          
          set -- /usr/bin/docker "${@}"
          
          exec "${@}"
          
          

           

           

          Günter Grodotzki added a comment - - edited hitting this issue with dind. Either a simple (configurable) delay would already help the majority of cases, or better, polling for the health-check.   FWIW, I was able to solve this locally with a dirty hack:   /usr/local/bin/docker: #!/usr/bin/env bash set -eo pipefail if [[ "${1}" == "top" ]]; then sleep 5 fi set -- /usr/bin/docker "${@}" exec "${@}"    

          Axel Heider added a comment -

          We are affected by this issue also, this race condition creates nasty random build failures everywhere. But after months of investigating it's good that the root cause is clear now at least. Any improvement here is highly appreciated.

          Axel Heider added a comment - We are affected by this issue also, this race condition creates nasty random build failures everywhere. But after months of investigating it's good that the root cause is clear now at least. Any improvement here is highly appreciated.

          Carl Winbäck added a comment -

          Any updates on this issue?

          Carl Winbäck added a comment - Any updates on this issue?

          Matt added a comment -

          This issue is hitting my team as well and is causing significant issues. 

          Matt added a comment - This issue is hitting my team as well and is causing significant issues. 

          Ryan Leonard added a comment -

          This is definitely a blocker, right?

          WORKAROUND: I thought it was a total blocker, but then I tried passing "–entrypoint=''" and it works for my case!

          docker.image('nvuillam/npm-groovy-lint').inside("--entrypoint=''") {
              sh 'npm-groovy-lint --output html'        
          }

           

          Ryan Leonard added a comment - This is definitely a blocker, right? WORKAROUND: I thought it was a total blocker, but then I tried passing "–entrypoint=''" and it works for my case! docker.image( 'nvuillam/npm-groovy-lint' ).inside( "--entrypoint=''" ) { sh 'npm-groovy-lint --output html'         }  

          Axel Heider added a comment -

          Hmm, I have tried this ages ago, but that did not work. So if it works now, that is nice.

          Axel Heider added a comment - Hmm, I have tried this ages ago, but that did not work. So if it works now, that is nice.

          Alex added a comment - - edited

          I stepped into the same problem, I assumed the entrypoint get's executed and stepped into the same issue.
          The problem is that `docker top` gets executed immediately after `docker run`
            https://github.com/jenkinsci/docker-workflow-plugin/blob/master/src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java#L201
          may be a polling loop that tries again for `DockerClient.CLIENT_TIMEOUT` would solve the problem.

          Alternative would be a `HEALTHCHECK` for longer delays.

          Alex added a comment - - edited I stepped into the same problem, I assumed the entrypoint get's executed and stepped into the same issue. The problem is that `docker top` gets executed immediately after `docker run`   https://github.com/jenkinsci/docker-workflow-plugin/blob/master/src/main/java/org/jenkinsci/plugins/docker/workflow/WithContainerStep.java#L201 may be a polling loop that tries again for `DockerClient.CLIENT_TIMEOUT` would solve the problem. Alternative would be a `HEALTHCHECK` for longer delays.

            Unassigned Unassigned
            bozaro Artem V. Navrotskiy
            Votes:
            18 Vote for this issue
            Watchers:
            24 Start watching this issue

              Created:
              Updated: