Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54389

Containers with ENTRYPOINT is not started correctly

      I try to add some service startup to Dockerfile ENTRYPOINT.

      But Jenkins starts container with `cat` CMD and immediately check `cat` in `docker ps` output. As result, there are race condition between `docker ps` and ENTRYPOINT script.

      I created minimalistic example for reproducing this issue on stock Jenkins: https://github.com/bozaro/jenkins-docker-issue

      This example contains:

      1. Dockerfile declaring ENTRYPOINT:
      FROM alpine:latest
      COPY entry.sh /opt/bin/
      ENTRYPOINT ["/opt/bin/entry.sh"]
      CMD ["/bin/sh"] 
      1. ENTRYPOINT script with 3 seconds delay and creating marker file:
      #!/bin/sh -ex
      # Starting very important services...
      sleep 3
      # Service is ready now
      echo > /tmp/ready
      exec "$@"
      1. Jenkinsfile running container with marker file check:
      pipeline {
          agent {
              dockerfile {
                  filename 'Dockerfile'
              }
          }
          stages {
              stage('Check file') {
                  steps {
                      sh "cat /tmp/ready"
                  }
              }
          }
      }
      

      Expected result for Jenkins test:

      1. At least a 3-second delay on pipeline withDockerContainer step
      2. ENTRYPOINT script output in build output;
      3. Green build

      Actual result for Jenkins test:

      1. False positive error message:

      ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements).

      1. Failed "Check file" step because ENTRYPOINT is not completed yet and /tmp/ready file is not exists;
      2. There are no ENTRYPOINT script output in build.

      Theoretically this issue can be solved if Jenkins would send some random string marker to docker container and wait this marker in container output.

          [JENKINS-54389] Containers with ENTRYPOINT is not started correctly

          Artem V. Navrotskiy created issue -
          Artem V. Navrotskiy made changes -
          Description Original: I try to add some service startup to Dockerfile ENTRYPOINT.

          But Jenkins starts container with `cat` CMD and immediately check `cat` in `docker ps` output. As result, there are race condition between `docker ps` and ENTRYPOINT script.

          I created minimalistic example for reproducing this issue on stock Jenkins: [https://github.com/bozaro/jenkins-docker-issue]

          This example contains:
           # Dockerfile declaring ENTRYPOINT:

          {code}
          FROM alpine:latest
          COPY entry.sh /opt/bin/
          ENTRYPOINT ["/opt/bin/entry.sh"]
          CMD ["/bin/sh"] {code}
           # ENTRYPOINT script with 3 seconds delay and creating marker file:

          {code}
          #!/bin/sh -ex
          # Starting very important services...
          sleep 3
          # Service is ready now
          echo > /tmp/ready
          exec "$@"{code}
           # Jenkinsfile running container with marker file check:

          {code:java}
          pipeline {
              agent {
                  dockerfile {
                      filename 'Dockerfile'
                  }
              }
              stages {
                  stage('Check file') {
                      steps {
                          sh "cat /tmp/ready"
                      }
                  }
              }
          }
          {code}
          Expected result for Jenkins test:
           # At least a 3-second delay on pipeline withDockerContainer step
           # Green build

          Actual result for Jenkins test:
           # False positive error message:

          ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see [https://github.com/docker-library/official-images#consistency] for entrypoint consistency requirements).
           # Failed "Check file" step because ENTRYPOINT is not completed yet and /tmp/ready file is not exists.

          Theoretically this issue can be solved if Jenkins would send some random string marker to docker container and wait this marker in container output.
          New: I try to add some service startup to Dockerfile ENTRYPOINT.

          But Jenkins starts container with `cat` CMD and immediately check `cat` in `docker ps` output. As result, there are race condition between `docker ps` and ENTRYPOINT script.

          I created minimalistic example for reproducing this issue on stock Jenkins: [https://github.com/bozaro/jenkins-docker-issue]

          This example contains:
           # Dockerfile declaring ENTRYPOINT:

          {code:java}
          FROM alpine:latest
          COPY entry.sh /opt/bin/
          ENTRYPOINT ["/opt/bin/entry.sh"]
          CMD ["/bin/sh"] {code}
           # ENTRYPOINT script with 3 seconds delay and creating marker file:

          {code:java}
          #!/bin/sh -ex
          # Starting very important services...
          sleep 3
          # Service is ready now
          echo > /tmp/ready
          exec "$@"{code}
           # Jenkinsfile running container with marker file check:

          {code:java}
          pipeline {
              agent {
                  dockerfile {
                      filename 'Dockerfile'
                  }
              }
              stages {
                  stage('Check file') {
                      steps {
                          sh "cat /tmp/ready"
                      }
                  }
              }
          }
          {code}
          Expected result for Jenkins test:
           # At least a 3-second delay on pipeline withDockerContainer step
           # ENTRYPOINT script output in build output;
           # Green build

          Actual result for Jenkins test:
           # False positive error message:

          ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see [https://github.com/docker-library/official-images#consistency] for entrypoint consistency requirements).
           # Failed "Check file" step because ENTRYPOINT is not completed yet and /tmp/ready file is not exists;
           # There are no ENTRYPOINT script output in build.

          Theoretically this issue can be solved if Jenkins would send some random string marker to docker container and wait this marker in container output.
          Andrew Nicols made changes -
          Priority Original: Minor [ 4 ] New: Blocker [ 1 ]

          Andrew Nicols added a comment -

          This could also be solved if Jenkins were to respect a healthcheck and the plugin changed to check the health of the container.

          This could also be specified on the CLI:

          docker run -u 501:501 -t --health-cmd '[ -f /tmp/ready ]' --health-start-period 1s --health-interval 100s myimage sh -c 'touch /tmp/ready && cat'
          

          Either way we do need a way to allow slower entrypoints to do their job.

          Andrew Nicols added a comment - This could also be solved if Jenkins were to respect a healthcheck and the plugin changed to check the health of the container. This could also be specified on the CLI: docker run -u 501:501 -t --health-cmd '[ -f /tmp/ready ]' --health-start-period 1s --health-interval 100s myimage sh -c 'touch /tmp/ready && cat' Either way we do need a way to allow slower entrypoints to do their job.
          Andrew Nicols made changes -
          Link New: This issue duplicates JENKINS-39748 [ JENKINS-39748 ]

          Similarly, I'm hitting this with builds of Hugo.  The container has a single binary in it: hugo.

           

          I'm watching both the race condition called out above occurring as well as the incorrect behavior of attempting to override the valid ENTRYPOINT defined in the file.

          Brian Redbeard added a comment - Similarly, I'm hitting this with builds of Hugo .  The container has a single binary in it: hugo.   I'm watching both the race condition called out above occurring as well as the incorrect behavior of attempting to override the valid ENTRYPOINT defined in the file.
          Rodrigo Carvalho Silva made changes -
          Component/s New: docker-plugin [ 18724 ]
          Rodrigo Carvalho Silva made changes -
          Component/s New: docker [ 20834 ]
          Component/s Original: docker-plugin [ 18724 ]
          Rodrigo Carvalho Silva made changes -
          Component/s Original: docker [ 20834 ]
          Rodrigo Carvalho Silva made changes -
          Comment [ I tested and this also happens using the declarative docker sintax directly, so I believe this isn't directly related do docker-workflow plugin ]

            Unassigned Unassigned
            bozaro Artem V. Navrotskiy
            Votes:
            18 Vote for this issue
            Watchers:
            24 Start watching this issue

              Created:
              Updated: