Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-47144

Kubernetes pod slaves that never start successfully never get cleaned up

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • kubernetes-plugin
    • None
    • Jenkins 2.73.1, kubernetes-plugin 1.0

      If I define a pod template with an invalid command and the container never becomes ready in the pod, then I see the following issues:

      1. The job never times out and provisioning doesn't seem to timeout. It spawns pods that continue to fail up to the instance cap.
      2. When I cancel the job it's getting stuck and throwing exceptions because the agent is offline and continuously getting terminated exceptions.
      3. Eventually forcing the job to cancel works, but the agent is removed from jenkins, but the pod is still left around.
      4. The left over pod never gets deleted, even with container cleanup timeout specified.

      I see errors like this in the logs:
      https://gist.github.com/chancez/27c6afdaaff3e91aa82dfe03055273dd

      I'm also seeings logs like `Failed to delete pod for agent jenkins/test-tmp-drvtq: not found` occassionally right after a build finishes, and the pod exists but isn't deleted.

      https://gist.github.com/chancez/4d65118c11af054860f22df76364fa31 Is an example of a regular pipeline Jenkinsfile which i created to reproduce this issue.

            Unassigned Unassigned
            chancez Chance Zibolski
            Votes:
            7 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated: