Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-73779

PeriodicGarbageCollection Delete Non-Orphan Pods

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • kubernetes-plugin
    • Jenkins Version 2.440.3
      Kubernetes Plugin 4287.v73451380b_576

      We have two Kubernetes Clouds defined: OnPremK8S and GKE.
      Both have garbage collection enabled.

      All of our jobs are running on the OnPremK8S cloud, which is properly configured and has the necessary permissions.

      The GKE cloud, however, is missing a critical permission and cannot list Pods.
      None of our jobs are configured to use the GKE cloud.

      We have observed that some jobs, including their nested jobs, are suddenly aborted.
      Upon reviewing the Jenkins service logs, we found that the garbage collection module is deleting these jobs, incorrectly marking them as orphaned.

      This always occurs immediately after an exception is thrown while attempting to list Pods in the GKE cloud (due to the missing permissions).

      I've attached the crucial part of the log.
      Unfortunately I cannot provide the full log so please let me know if there is any critical information missing.

          [JENKINS-73779] PeriodicGarbageCollection Delete Non-Orphan Pods

          Mark Waite added a comment -

          This looks like a request for diagnostic help or for help with a configuration issue. We use the Jenkins issue tracker for bugs and enhancement requests, not for diagnostic help or configuration issues.

          You'll find a larger audience that may be able to help you on https://community.jenkins.io and on the Jenkins user mailing list. Those are locations for diagnostic help and for questions.

          Mark Waite added a comment - This looks like a request for diagnostic help or for help with a configuration issue. We use the Jenkins issue tracker for bugs and enhancement requests, not for diagnostic help or configuration issues. You'll find a larger audience that may be able to help you on https://community.jenkins.io and on the Jenkins user mailing list. Those are locations for diagnostic help and for questions.

          Daniel added a comment - - edited

          Thanks for the quick response, I resolved it by disabling the garbage collector for that misconfigured cloud, so no help needed. 

          So you consider it “working as expected” when one misconfigured cloud marks ALL Kubernetes agents as orphaned and deletes them regardless to the cloud agent they are running on? 

          Daniel added a comment - - edited Thanks for the quick response, I resolved it by disabling the garbage collector for that misconfigured cloud, so no help needed.  So you consider it “working as expected” when one misconfigured cloud marks ALL Kubernetes agents as orphaned and deletes them regardless to the cloud agent they are running on? 

          Mark Waite added a comment -

          Thanks for the correction. I've set it back to "Open".

          Mark Waite added a comment - Thanks for the correction. I've set it back to "Open".

            Unassigned Unassigned
            danielshaal92 Daniel
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: