[JENKINS-71967] kubernetes-plugin creates 2 pods per run

Type: Bug
Resolution: Unresolved
Priority: Major
Component/s: kubernetes-plugin
Labels:
None
Environment:
Jenkins 2.414.1
kubernetes plugin 4029.v5712230ccb_f8

Similar Issues:
Powered by SuggestiMate

Show

After latest upgrade of jenkins itself (latest lts from previous version of lts) and kubernetes plugin I noticed that it now creates 2 pods per job run.

It started happening just recently but I cannot tell what component upgrade exactly has caused this.

I create a pod using a single `podTemplate` function.

This is how jenkins logs look in the UI:

00:00:07.660  Created Pod: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76
00:00:21.526  Still waiting to schedule task
00:00:21.528  Waiting for next available executor on ‘jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76’
00:00:38.214  Agent jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76 is provisioned from template jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1
00:00:38.237  ---
00:00:38.237  apiVersion: "v1"
00:00:38.237  kind: "Pod"
00:00:38.237  metadata:
00:00:38.237  <redacted>
00:00:38.238  
00:00:38.347  Created Pod: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw
00:00:38.679  Running on jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76 in /home/jenkins/agent/workspace/apps--certly-client_v2

See that the prefix is the same and only last 5 characters differ.

And this is how stdout of jenkins server looks

2023-09-05 03:43:52.304+0000 [id=416]    INFO    hudson.slaves.NodeProvisioner#update: jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76 provisioning successfully completed. We have now 2 computer(s)
2023-09-05 03:43:52.383+0000 [id=418]    INFO    o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76
2023-09-05 03:43:57.188+0000 [id=974]    INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #7 from /10.51.1.43:58702
2023-09-05 03:44:22.869+0000 [id=418]    INFO    o.c.j.p.k.KubernetesLauncher#launch: Pod is running: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76
2023-09-05 03:44:23.007+0000 [id=47]    INFO    hudson.slaves.NodeProvisioner#update: jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw provisioning successfully completed. We have now 3 computer(s)
2023-09-05 03:44:23.070+0000 [id=418]    INFO    o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw
2023-09-05 03:44:27.955+0000 [id=1029]    INFO    h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #8 from /10.51.1.46:42336
...

2023-09-05 03:47:02.719+0000 [id=976]    INFO    j.s.DefaultJnlpSlaveReceiver#channelClosed: Computer.threadPoolForRemoting [#23] for jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76 terminated: java.nio.channels.ClosedChannelException 
2023-09-05 03:50:42.663+0000 [id=1549]    INFO    o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw                                                                                          
2023-09-05 03:50:42.886+0000 [id=1549]    INFO    o.c.j.p.k.KubernetesSlave#deleteSlavePod: Terminated Kubernetes instance for agent adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw                                                               
2023-09-05 03:50:42.888+0000 [id=1549]    INFO    o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw                                                                                                              
2023-09-05 03:50:42.889+0000 [id=1549]    INFO    j.s.DefaultJnlpSlaveReceiver#channelClosed: Computer.threadPoolForRemoting [#33] for jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw terminated: java.nio.channels.ClosedChannelException

So one of the pods exits immediately right after the job has completed, and the other stays there til the job timeout.

Ivan Kurnosov created issue - 2023-09-05 03:51

Ivan Kurnosov made changes - 2023-09-05 03:52

Description

Original: After latest upgrade of jenkins itself (latest lts from previous version of lts) and kubernetes plugin I noticed that it now creates 2 pods per job run.

It started happening just recently but I cannot tell what component upgrade exactly has caused this.

I create a pod using a single `podTemplate` function.

This is how jenkins logs look in the UI:

{code:java}
00:00:07.660 Created Pod: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76
00:00:21.526 Still waiting to schedule task
00:00:21.528 Waiting for next available executor on ‘jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76’
00:00:38.214 Agent jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76 is provisioned from template jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1
00:00:38.237 ---
00:00:38.237 apiVersion: "v1"
00:00:38.237 kind: "Pod"
00:00:38.237 metadata:
00:00:38.237 <redacted>
00:00:38.238
00:00:38.347 Created Pod: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw
00:00:38.679 Running on jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76 in /home/jenkins/agent/workspace/apps--certly-client_v2
{code}

See that the prefix is the same and only last 5 characters differ.

And this is how stdout of jenkins server looks

{code:java}
2023-09-05 03:43:52.304+0000 [id=416] INFO hudson.slaves.NodeProvisioner#update: jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76 provisioning successfully completed. We have now 2 computer(s)
2023-09-05 03:43:52.383+0000 [id=418] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76
2023-09-05 03:43:57.188+0000 [id=974] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #7 from /10.51.1.43:58702
2023-09-05 03:44:22.869+0000 [id=418] INFO o.c.j.p.k.KubernetesLauncher#launch: Pod is running: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76
2023-09-05 03:44:23.007+0000 [id=47] INFO hudson.slaves.NodeProvisioner#update: jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw provisioning successfully completed. We have now 3 computer(s)
2023-09-05 03:44:23.070+0000 [id=418] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw
2023-09-05 03:44:27.955+0000 [id=1029] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #8 from /10.51.1.46:42336
...

2023-09-05 03:47:02.719+0000 [id=976] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Computer.threadPoolForRemoting [#23] for jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76 terminated: java.nio.channels.ClosedChannelException
2023-09-05 03:50:42.663+0000 [id=1549] INFO o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw
2023-09-05 03:50:42.886+0000 [id=1549] INFO o.c.j.p.k.KubernetesSlave#deleteSlavePod: Terminated Kubernetes instance for agent adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw
2023-09-05 03:50:42.888+0000 [id=1549] INFO o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw
2023-09-05 03:50:42.889+0000 [id=1549] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Computer.threadPoolForRemoting [#33] for jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw terminated: java.nio.channels.ClosedChannelException
{code}

So one of the pods exits immediately, and the other stays there til the job timeout.

New: After latest upgrade of jenkins itself (latest lts from previous version of lts) and kubernetes plugin I noticed that it now creates 2 pods per job run.

It started happening just recently but I cannot tell what component upgrade exactly has caused this.

I create a pod using a single `podTemplate` function.

This is how jenkins logs look in the UI:

{code:java}
00:00:07.660 Created Pod: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76
00:00:21.526 Still waiting to schedule task
00:00:21.528 Waiting for next available executor on ‘jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76’
00:00:38.214 Agent jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76 is provisioned from template jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1
00:00:38.237 ---
00:00:38.237 apiVersion: "v1"
00:00:38.237 kind: "Pod"
00:00:38.237 metadata:
00:00:38.237 <redacted>
00:00:38.238
00:00:38.347 Created Pod: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw
00:00:38.679 Running on jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76 in /home/jenkins/agent/workspace/apps--certly-client_v2
{code}

See that the prefix is the same and only last 5 characters differ.

And this is how stdout of jenkins server looks

{code:java}
2023-09-05 03:43:52.304+0000 [id=416] INFO hudson.slaves.NodeProvisioner#update: jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76 provisioning successfully completed. We have now 2 computer(s)
2023-09-05 03:43:52.383+0000 [id=418] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76
2023-09-05 03:43:57.188+0000 [id=974] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #7 from /10.51.1.43:58702
2023-09-05 03:44:22.869+0000 [id=418] INFO o.c.j.p.k.KubernetesLauncher#launch: Pod is running: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76
2023-09-05 03:44:23.007+0000 [id=47] INFO hudson.slaves.NodeProvisioner#update: jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw provisioning successfully completed. We have now 3 computer(s)
2023-09-05 03:44:23.070+0000 [id=418] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw
2023-09-05 03:44:27.955+0000 [id=1029] INFO h.TcpSlaveAgentListener$ConnectionHandler#run: Accepted JNLP4-connect connection #8 from /10.51.1.46:42336
...

2023-09-05 03:47:02.719+0000 [id=976] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Computer.threadPoolForRemoting [#23] for jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-7ws76 terminated: java.nio.channels.ClosedChannelException
2023-09-05 03:50:42.663+0000 [id=1549] INFO o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw
2023-09-05 03:50:42.886+0000 [id=1549] INFO o.c.j.p.k.KubernetesSlave#deleteSlavePod: Terminated Kubernetes instance for agent adm-prod-jenkins-agents/jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw
2023-09-05 03:50:42.888+0000 [id=1549] INFO o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw
2023-09-05 03:50:42.889+0000 [id=1549] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Computer.threadPoolForRemoting [#33] for jenkins-slave-d2e6bd35-409a-48e5-8b13-bf255719d5c2-zxfx1-dr2nw terminated: java.nio.channels.ClosedChannelException
{code}

So one of the pods exits immediately right after the job has completed, and the other stays there til the job timeout.

Oleh made changes - 2023-10-18 18:50

Priority

Original: Minor [ 4 ]

New: Major [ 3 ]

Jenkins

Details

Description

Attachments

Activity

People

Dates