Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-67592

Jenkins starts pipeline before the agent pod is ready

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • kubernetes-plugin
    • None
    • Jenkins 2.289.2
      Kubernetes plugin 1.30.11
      Openshift 3.11 and Openshift 4.6

      Sometimes, randomly, when we try to run a job with a pod template with two containers (jnlp container and an additional one), Jenkins starts running the pipeline before the pod is ready. The result is that when the pipeline reaches the container step it fails. 

      18:14:20 [Pipeline] container

      18:14:20 [Pipeline] {

      18:14:20 [Pipeline] sh

      18:14:20 java.net.ProtocolException: Expected HTTP 101 response but was '500 Internal Server Error'

      18:14:20 at okhttp3.internal.ws.RealWebSocket.checkResponse(RealWebSocket.java:229)

      18:14:20 at okhttp3.internal.ws.RealWebSocket$2.onResponse(RealWebSocket.java:196)

      18:14:20 at okhttp3.RealCall$AsyncCall.execute(RealCall.java:203)

      18:14:20 at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)

      18:14:20 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

      18:14:20 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

      18:14:20 at java.base/java.lang.Thread.run(Thread.java:834)

       

      This issue happens when the second container uses a big image (around 1Gb). The jnlp container starts but the second container is not started yet, but nevertheless Jenkins starts running the pipeline. 

          [JENKINS-67592] Jenkins starts pipeline before the agent pod is ready

          Did you use `container(CONTAINER_NAME)` with the correct CONTAINER_NAME? or is a default container specified in your podTemplate?

          Florin Hillebrand added a comment - Did you use `container(CONTAINER_NAME)` with the correct CONTAINER_NAME? or is a default container specified in your podTemplate?

          Pablo Gomez added a comment -

          Yes I did, as I mentioned it is random issue. 

          Pablo Gomez added a comment - Yes I did, as I mentioned it is random issue. 

          I see, as a workaround you could let the jnlp container sleep for a few seconds. Have you tried setting readiness/liveness probes for the container in question (but I doubt this will change something)?
          I would expect some logic in kubernetes-plugin or jnlp slave that would wait for the pod to be fully available. Is your jnlp agent up-to-date?

          Florin Hillebrand added a comment - I see, as a workaround you could let the jnlp container sleep for a few seconds. Have you tried setting readiness/liveness probes for the container in question (but I doubt this will change something)? I would expect some logic in kubernetes-plugin or jnlp slave that would wait for the pod to be fully available. Is your jnlp agent up-to-date?

          Pablo Gomez added a comment -

          We are using a custom jnlp image. But I think that logic should be in the Kubernetes plugin and not in image. The image should be Kubernetes-agnostic. 

          I thought about readiness probe but that won't change anything. Jenkins starts running the pipeline even if the pod is not in Running state yet. 

          Pablo Gomez added a comment - We are using a custom jnlp image. But I think that logic should be in the Kubernetes plugin and not in image. The image should be Kubernetes-agnostic.  I thought about readiness probe but that won't change anything. Jenkins starts running the pipeline even if the pod is not in Running state yet. 

          It is surprising because the plugin waits for readiness of all containers within the agent pod before letting the build use it. Have you tried with a recent version of the plugin to check whether it still happens?

          Vincent Latombe added a comment - It is surprising because the plugin waits for readiness of all containers within the agent pod before letting the build use it. Have you tried with a recent version of the plugin to check whether it still happens?

            Unassigned Unassigned
            pgomez Pablo Gomez
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: