Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-54389

Containers with ENTRYPOINT is not started correctly

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I try to add some service startup to Dockerfile ENTRYPOINT.

      But Jenkins starts container with `cat` CMD and immediately check `cat` in `docker ps` output. As result, there are race condition between `docker ps` and ENTRYPOINT script.

      I created minimalistic example for reproducing this issue on stock Jenkins: https://github.com/bozaro/jenkins-docker-issue

      This example contains:

      1. Dockerfile declaring ENTRYPOINT:
      FROM alpine:latest
      COPY entry.sh /opt/bin/
      ENTRYPOINT ["/opt/bin/entry.sh"]
      CMD ["/bin/sh"] 
      1. ENTRYPOINT script with 3 seconds delay and creating marker file:
      #!/bin/sh -ex
      # Starting very important services...
      sleep 3
      # Service is ready now
      echo > /tmp/ready
      exec "$@"
      1. Jenkinsfile running container with marker file check:
      pipeline {
          agent {
              dockerfile {
                  filename 'Dockerfile'
              }
          }
          stages {
              stage('Check file') {
                  steps {
                      sh "cat /tmp/ready"
                  }
              }
          }
      }
      

      Expected result for Jenkins test:

      1. At least a 3-second delay on pipeline withDockerContainer step
      2. ENTRYPOINT script output in build output;
      3. Green build

      Actual result for Jenkins test:

      1. False positive error message:

      ERROR: The container started but didn't run the expected command. Please double check your ENTRYPOINT does execute the command passed as docker run argument, as required by official docker images (see https://github.com/docker-library/official-images#consistency for entrypoint consistency requirements).

      1. Failed "Check file" step because ENTRYPOINT is not completed yet and /tmp/ready file is not exists;
      2. There are no ENTRYPOINT script output in build.

      Theoretically this issue can be solved if Jenkins would send some random string marker to docker container and wait this marker in container output.

        Attachments

          Issue Links

            Activity

            Hide
            andrewnicols Andrew Nicols added a comment -

            This could also be solved if Jenkins were to respect a healthcheck and the plugin changed to check the health of the container.

            This could also be specified on the CLI:

            docker run -u 501:501 -t --health-cmd '[ -f /tmp/ready ]' --health-start-period 1s --health-interval 100s myimage sh -c 'touch /tmp/ready && cat'
            

            Either way we do need a way to allow slower entrypoints to do their job.

            Show
            andrewnicols Andrew Nicols added a comment - This could also be solved if Jenkins were to respect a healthcheck and the plugin changed to check the health of the container. This could also be specified on the CLI: docker run -u 501:501 -t --health-cmd '[ -f /tmp/ready ]' --health-start-period 1s --health-interval 100s myimage sh -c 'touch /tmp/ready && cat' Either way we do need a way to allow slower entrypoints to do their job.
            Hide
            brianredbeard Brian Redbeard added a comment -

            Similarly, I'm hitting this with builds of Hugo.  The container has a single binary in it: hugo.

             

            I'm watching both the race condition called out above occurring as well as the incorrect behavior of attempting to override the valid ENTRYPOINT defined in the file.

            Show
            brianredbeard Brian Redbeard added a comment - Similarly, I'm hitting this with builds of Hugo .  The container has a single binary in it: hugo.   I'm watching both the race condition called out above occurring as well as the incorrect behavior of attempting to override the valid ENTRYPOINT defined in the file.
            Hide
            lifeofguenter Gunter Grodotzki added a comment - - edited

            hitting this issue with dind. Either a simple (configurable) delay would already help the majority of cases, or better, polling for the health-check.

             

            FWIW, I was able to solve this locally with a dirty hack:

             

            /usr/local/bin/docker:
            
            #!/usr/bin/env bash
            
            set -eo pipefail
            
            if [[ "${1}" == "top" ]]; then
             sleep 5
            fi
            
            set -- /usr/bin/docker "${@}"
            
            exec "${@}"
            
            

             

             

            Show
            lifeofguenter Gunter Grodotzki added a comment - - edited hitting this issue with dind. Either a simple (configurable) delay would already help the majority of cases, or better, polling for the health-check.   FWIW, I was able to solve this locally with a dirty hack:   /usr/local/bin/docker: #!/usr/bin/env bash set -eo pipefail if [[ "${1}" == "top" ]]; then sleep 5 fi set -- /usr/bin/docker "${@}" exec "${@}"    
            Hide
            axelh Axel Heider added a comment -

            We are affected by this issue also, this race condition creates nasty random build failures everywhere. But after months of investigating it's good that the root cause is clear now at least. Any improvement here is highly appreciated.

            Show
            axelh Axel Heider added a comment - We are affected by this issue also, this race condition creates nasty random build failures everywhere. But after months of investigating it's good that the root cause is clear now at least. Any improvement here is highly appreciated.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              bozaro Artem V. Navrotskiy
              Votes:
              12 Vote for this issue
              Watchers:
              15 Start watching this issue

                Dates

                Created:
                Updated: