That is intended so it doesn't keep spawning pods, and allows to inspect the errors
There is another jira for "Failed to delete pod" errors

Also seeing this on master.

So this really breaks some use cases for us because we're not giving people full access to the namespace if we can avoid it (i mean, they indirectly have access via Jenkins, but that's about it), so the moment they trigger a few bad builds, then their instanceCap fills up and their jobs can no longer run until someone manually deletes the pods in the cluster.

Our podTemplates aren't getting cleaned up either, so unless we change the label on the the pod, often the next provision still re-uses the broken podTemplate.

I'm looking at the other JIRA issues, and they seem similiar, and I noticed there was a PR merged recently that looks related but it doesn't seem to be helping any when using a custom build from the current master.

I'm also curious to why the agent immediately goes offline in Jenkins if it's container isn't the one failing within the pod. It seems to just get connected an immediately terminated over and over. It seems like this is also related to why it can't cancel/kill the running (failing) job.

We are also affected by this issue, and it will be great figure out some way to have some retries limit and delete created pods after that.

ok, so there are some things that can be done in the plugin and some that can not because they happen in jenkins core. Let's work on a proposal

1. The job never times out and provisioning doesn't seem to timeout. It spawns pods that continue to fail up to the instance cap.

jobs stay running waiting for an agent to come up and seems the right thing to do. We could make the provisioner look at the last spawned pod for an agent template and not launch new ones if last one was error

2. When I cancel the job it's getting stuck and throwing exceptions because the agent is offline and continuously getting terminated exceptions.

if you talk about ClosedChannelException I don't think there's anything that can be done here

3. Eventually forcing the job to cancel works, but the agent is removed from jenkins, but the pod is still left around.

pods in error state are left for inspection. kubernetes-plugin right now is only called when provisioning so may need a service that periodically cleans up

4. The left over pod never gets deleted, even with container cleanup timeout specified.

if you talk about kubernetes cleanup I guess it won't delete it until the pod is deleted

Possible duplicate of https://issues.jenkins-ci.org/browse/JENKINS-54540

yes i confirm. i had same issue. i am cleaning them now manually with k -n jenkins delete pod -ljenkins=slave