-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins 2.401.2
Kubernetes Plugin 3971.v94b_4c914ca_75
Kubernetes v1.25.9
Configuration-as-code 1647.ve39ca_b_829b_42
Issue
I have podRetention set to "always" at the Cloud/Kubernetes level. I have 2 containers in my PodTemplate one for jnlp and another for docker-dind. Both have ttyEnabled: true.
When idleMinutes comes to bear, the log below correctly shows that jenkins wants to disconnect the agent and that the pod is retained according to the podRetention policy. However, the jnlp container is "completed" and the pod stays up but is unusable.
Expectation
I am expecting for the pod to indeed stay up but be operational by keeping the jnlp container running.
Logs
Here is the jenkins log:
2023-07-12 17:00:13.813+0000 [id=370] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes cicd-int/jenkins-agent-1kdjt
2023-07-12 17:00:15.024+0000 [id=370] INFO o.c.j.p.k.KubernetesLauncher#launch: Pod is running: kubernetes cicd-int/jenkins-agent-1kdjt
2023-07-12 17:08:06.465+0000 [id=26] INFO h.slaves.CloudRetentionStrategy#check: Disconnecting jenkins-agent-1kdjt
2023-07-12 17:08:06.466+0000 [id=26] INFO o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent jenkins-agent-1kdjt
2023-07-12 17:08:06.487+0000 [id=26] WARNING o.c.j.p.k.KubernetesSlave#_terminate: Agent pod jenkins-agent-1kdjt was not deleted due to retention policy Default.
2023-07-12 17:08:06.488+0000 [id=26] INFO o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer jenkins-agent-1kdjt
2023-07-12 17:08:06.507+0000 [id=18] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Jetty (winstone)-18 for jenkins-agent-1kdjt terminated: java.nio.channels.ClosedChannelException
Here is the agent's jnlp container log:
Jul 12, 2023 5:00:14 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: jenkins-agent-1kdjt
Jul 12, 2023 5:00:14 PM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 3131.vf2b_b_798b_ce99
Jul 12, 2023 5:00:14 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
Jul 12, 2023 5:00:14 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
Jul 12, 2023 5:00:15 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: WebSocket connection open
Jul 12, 2023 5:00:15 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Jul 12, 2023 5:08:06 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave$SlaveDisconnector call
INFO: Disabled agent engine reconnects.
Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Write side closed
Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Read side closed
Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Read side closed
[JENKINS-71631] PodRetention Always: Pod stays up but jnlp container is completed
Nicolas
created issue -
Nicolas
made changes -
| Description |
Original:
I am expecting for the pod to indeed be up and the jnlp container to stay running.
I have *podRetention* set to "always" at the Cloud/Kubernetes level. I have 2 containers in my PodTemplate one for jnlp and another for docker-dind. Both have {*}ttyEnabled{*}: true. When *idleMinutes* comes to bear, i have indication that jenkins wants to disconnect the agent and that the pod is retained accordingly to the podRetention policy. However, the jnlp container is "completed". Here is the jenkins log: {{{}2023-07-12 16:36:34.633+0000 [id=113] INFO o.c.j.p.k.pod.retention.Reaper#watchCloud: set up watcher on kubernetes{}}}{{{}2023-07-12 16:36:36.036+0000 [id=113] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes cicd-int/jenkins-agent-hnnfk{}}}{{{}2023-07-12 16:36:36.678+0000 [id=113] INFO o.c.j.p.k.KubernetesLauncher#launch: Pod is running: kubernetes cicd-int/jenkins-agent-hnnfk{}}}{{{}2023-07-12 16:40:08.417+0000 [id=220] WARNING c.a.j.p.anchore.AnchoreBuilder#perform: Starting Anchore Container Image Scanner step, project: dbg-ad-mgmt, job: 1{}}}{{{}2023-07-12 16:40:54.471+0000 [id=220] WARNING c.a.j.p.anchore.AnchoreBuilder#perform: Completed Anchore Container Image Scanner step, project: dbg-ad-mgmt, job: 1{}}}{{{}2023-07-12 16:45:06.445+0000 [id=50] INFO h.slaves.CloudRetentionStrategy#check: Disconnecting jenkins-agent-hnnfk{}}}{{{}2023-07-12 16:45:06.447+0000 [id=50] INFO o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent jenkins-agent-hnnfk{}}}{{{}2023-07-12 16:45:06.498+0000 [id=50] WARNING o.c.j.p.k.KubernetesSlave#_terminate: Agent pod jenkins-agent-hnnfk was not deleted due to retention policy Default.{}}}{{{}2023-07-12 16:45:06.498+0000 [id=50] INFO o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer jenkins-agent-hnnfk{}}}{{{}2023-07-12 16:45:06.544+0000 [id=16] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Jetty (winstone)-16 for jenkins-agent-hnnfk terminated: java.nio.channels.ClosedChannelException{}}} I am expecting for the pod to indeed be up and the jnlp container to stay running. |
New:
I am expecting for the pod to indeed be up and the jnlp container to stay running.
I have *podRetention* set to "always" at the Cloud/Kubernetes level. I have 2 containers in my PodTemplate one for jnlp and another for docker-dind. Both have {*}ttyEnabled{*}: true. When *idleMinutes* comes to bear, i have indication that jenkins wants to disconnect the agent and that the pod is retained accordingly to the podRetention policy. However, the jnlp container is "completed". Here is the *jenkins* log: {{2023-07-12 17:00:13.813+0000 [id=370] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes cicd-int/jenkins-agent-1kdjt}} {{2023-07-12 17:00:15.024+0000 [id=370] INFO o.c.j.p.k.KubernetesLauncher#launch: Pod is running: kubernetes cicd-int/jenkins-agent-1kdjt}} {{2023-07-12 17:08:06.465+0000 [id=26] INFO h.slaves.CloudRetentionStrategy#check: Disconnecting jenkins-agent-1kdjt}} {{2023-07-12 17:08:06.466+0000 [id=26] INFO o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent jenkins-agent-1kdjt}} {{2023-07-12 17:08:06.487+0000 [id=26] WARNING o.c.j.p.k.KubernetesSlave#_terminate: Agent pod jenkins-agent-1kdjt was not deleted due to retention policy Default.}} {{2023-07-12 17:08:06.488+0000 [id=26] INFO o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer jenkins-agent-1kdjt}} {{2023-07-12 17:08:06.507+0000 [id=18] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Jetty (winstone)-18 for jenkins-agent-1kdjt terminated: java.nio.channels.ClosedChannelException}} Here is the *agent's* *jnlp* container log: {{{}Jul 12, 2023 5:00:14 PM hudson.remoting.jnlp.Main createEngine{}}}{{{}INFO: Setting up agent: jenkins-agent-1kdjt{}}}{{{}Jul 12, 2023 5:00:14 PM hudson.remoting.Engine startEngine{}}}{{{}INFO: Using Remoting version: 3131.vf2b_b_798b_ce99{}}}{{{}Jul 12, 2023 5:00:14 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir{}}}{{{}INFO: Using /home/jenkins/agent/remoting as a remoting work directory{}}}{{{}Jul 12, 2023 5:00:14 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging{}}}{{{}INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting{}}}{{{}Jul 12, 2023 5:00:15 PM hudson.remoting.jnlp.Main$CuiListener status{}}}{{{}INFO: WebSocket connection open{}}}{{{}Jul 12, 2023 5:00:15 PM hudson.remoting.jnlp.Main$CuiListener status{}}}{{{}INFO: Connected{}}}{{{}Jul 12, 2023 5:08:06 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave$SlaveDisconnector call{}}}{{{}INFO: Disabled agent engine reconnects.{}}}{{{}Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status{}}}{{{}INFO: Write side closed{}}}{{{}Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status{}}}{{{}INFO: Read side closed{}}}{{{}Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status{}}}{{{}INFO: Terminated{}}}{{{}Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status{}}}{{{}INFO: Read side closed{}}} |
Nicolas
made changes -
| Description |
Original:
I am expecting for the pod to indeed be up and the jnlp container to stay running.
I have *podRetention* set to "always" at the Cloud/Kubernetes level. I have 2 containers in my PodTemplate one for jnlp and another for docker-dind. Both have {*}ttyEnabled{*}: true. When *idleMinutes* comes to bear, i have indication that jenkins wants to disconnect the agent and that the pod is retained accordingly to the podRetention policy. However, the jnlp container is "completed". Here is the *jenkins* log: {{2023-07-12 17:00:13.813+0000 [id=370] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes cicd-int/jenkins-agent-1kdjt}} {{2023-07-12 17:00:15.024+0000 [id=370] INFO o.c.j.p.k.KubernetesLauncher#launch: Pod is running: kubernetes cicd-int/jenkins-agent-1kdjt}} {{2023-07-12 17:08:06.465+0000 [id=26] INFO h.slaves.CloudRetentionStrategy#check: Disconnecting jenkins-agent-1kdjt}} {{2023-07-12 17:08:06.466+0000 [id=26] INFO o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent jenkins-agent-1kdjt}} {{2023-07-12 17:08:06.487+0000 [id=26] WARNING o.c.j.p.k.KubernetesSlave#_terminate: Agent pod jenkins-agent-1kdjt was not deleted due to retention policy Default.}} {{2023-07-12 17:08:06.488+0000 [id=26] INFO o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer jenkins-agent-1kdjt}} {{2023-07-12 17:08:06.507+0000 [id=18] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Jetty (winstone)-18 for jenkins-agent-1kdjt terminated: java.nio.channels.ClosedChannelException}} Here is the *agent's* *jnlp* container log: {{{}Jul 12, 2023 5:00:14 PM hudson.remoting.jnlp.Main createEngine{}}}{{{}INFO: Setting up agent: jenkins-agent-1kdjt{}}}{{{}Jul 12, 2023 5:00:14 PM hudson.remoting.Engine startEngine{}}}{{{}INFO: Using Remoting version: 3131.vf2b_b_798b_ce99{}}}{{{}Jul 12, 2023 5:00:14 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir{}}}{{{}INFO: Using /home/jenkins/agent/remoting as a remoting work directory{}}}{{{}Jul 12, 2023 5:00:14 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging{}}}{{{}INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting{}}}{{{}Jul 12, 2023 5:00:15 PM hudson.remoting.jnlp.Main$CuiListener status{}}}{{{}INFO: WebSocket connection open{}}}{{{}Jul 12, 2023 5:00:15 PM hudson.remoting.jnlp.Main$CuiListener status{}}}{{{}INFO: Connected{}}}{{{}Jul 12, 2023 5:08:06 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave$SlaveDisconnector call{}}}{{{}INFO: Disabled agent engine reconnects.{}}}{{{}Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status{}}}{{{}INFO: Write side closed{}}}{{{}Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status{}}}{{{}INFO: Read side closed{}}}{{{}Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status{}}}{{{}INFO: Terminated{}}}{{{}Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status{}}}{{{}INFO: Read side closed{}}} |
New:
*Issue*
I have *podRetention* set to "always" at the Cloud/Kubernetes level. I have 2 containers in my PodTemplate one for *jnlp* and another for docker-dind. Both have {*}ttyEnabled{*}: true. When *idleMinutes* comes to bear, the log below correctly shows that jenkins wants to disconnect the agent and that the pod is retained according to the podRetention policy. However, the jnlp container is "completed" and the pod stays up but is unusable. *Expectation* I am expecting for the pod to indeed stay up but be operational by keeping the jnlp container running. *Logs* Here is the *jenkins* log: 2023-07-12 17:00:13.813+0000 [id=370] INFO o.c.j.p.k.KubernetesLauncher#launch: Created Pod: kubernetes cicd-int/jenkins-agent-1kdjt 2023-07-12 17:00:15.024+0000 [id=370] INFO o.c.j.p.k.KubernetesLauncher#launch: Pod is running: kubernetes cicd-int/jenkins-agent-1kdjt 2023-07-12 17:08:06.465+0000 [id=26] INFO h.slaves.CloudRetentionStrategy#check: Disconnecting jenkins-agent-1kdjt 2023-07-12 17:08:06.466+0000 [id=26] INFO o.c.j.p.k.KubernetesSlave#_terminate: Terminating Kubernetes instance for agent jenkins-agent-1kdjt 2023-07-12 17:08:06.487+0000 [id=26] WARNING o.c.j.p.k.KubernetesSlave#_terminate: Agent pod jenkins-agent-1kdjt was not deleted due to retention policy Default. 2023-07-12 17:08:06.488+0000 [id=26] INFO o.c.j.p.k.KubernetesSlave#_terminate: Disconnected computer jenkins-agent-1kdjt 2023-07-12 17:08:06.507+0000 [id=18] INFO j.s.DefaultJnlpSlaveReceiver#channelClosed: Jetty (winstone)-18 for jenkins-agent-1kdjt terminated: java.nio.channels.ClosedChannelException Here is the *agent's* *jnlp* container log: Jul 12, 2023 5:00:14 PM hudson.remoting.jnlp.Main createEngine INFO: Setting up agent: jenkins-agent-1kdjt Jul 12, 2023 5:00:14 PM hudson.remoting.Engine startEngine INFO: Using Remoting version: 3131.vf2b_b_798b_ce99 Jul 12, 2023 5:00:14 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /home/jenkins/agent/remoting as a remoting work directory Jul 12, 2023 5:00:14 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting Jul 12, 2023 5:00:15 PM hudson.remoting.jnlp.Main$CuiListener status INFO: WebSocket connection open Jul 12, 2023 5:00:15 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Jul 12, 2023 5:08:06 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave$SlaveDisconnector call INFO: Disabled agent engine reconnects. Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Write side closed Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Read side closed Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Jul 12, 2023 5:08:06 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Read side closed |