JENKINS-40050 reported a similar failure though it has been years since anyone has reported a case of that issue.

The comment in JENKINS-40050 indicates that the issue seemed to be specific to a particular machine.

If the "jarCache" directory on the agent is removed and the agent restarted, does the problem persist?

If the agent is destroyed and recreated, does the problem persist?

Is the agent running a recent version of Jenkins remoting? The version node monitors plugin will show the remoting version in a table on the list of agents.

I'm not able to duplicate the issue in my environment. We aren't able to duplicate the issue on any of the Jenkins controllers that are run by the Jenkins project.

You've not provided enough information for others to try to duplicate the issue. Please provide the information listed in "How to report an issue" so that others will be more likely to attempt to duplicate the issue.

markewaite thanks for your reply. I'll try to answer your question:

If the "jarCache" directory on the agent is removed and the agent restarted, does the problem persist?

The setup for this Jenkins for both Master and agent running in kubernetes, and the agent will only created on demand when there is a job running. And since the agent suddenly terminated, the pods suddenly change it's state from running to be succeeded. In this case I'm not sure how to removed "jarCache" as I'm not able to access the pod.

Is the agent running a recent version of Jenkins remoting?

Yes there is information about the agent version, here is the full logs, the agent is using Remoting version: 3107.v665000b_51092

Jun 18, 2024 2:24:44 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Jun 18, 2024 2:24:44 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
Jun 18, 2024 2:24:44 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: jenkins-iac-agent.jenkins-iac.svc.cluster.local
Agent port: 50000
Identity: f5:a1:99:d7:bd:a3:78:61:29:10:30:17:be:7a:53:b7
Jun 18, 2024 2:24:44 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Jun 18, 2024 2:24:44 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-iac-agent.jenkins-iac.svc.cluster.local:50000
Jun 18, 2024 2:24:44 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Jun 18, 2024 2:24:44 PM org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader run
INFO: Waiting for ProtocolStack to start.
Jun 18, 2024 2:24:44 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: f5:a1:99:d7:bd:a3:78:61:29:10:30:17:be:7a:53:b7
Jun 18, 2024 2:24:44 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Jun 18, 2024 2:24:50 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave$SlaveDisconnector call
INFO: Disabled agent engine reconnects.
Jun 18, 2024 2:24:50 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Jun 18, 2024 2:24:50 PM hudson.remoting.Request$2 run
INFO: Failed to send back a reply to the request UserRequest:hudson.Launcher$RemoteLauncher$KillTask@1a39a65d: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@186a7a0b:JNLP4-connect connection to jenkins-iac-agent.jenkins-iac.svc.cluster.local/10.xx.71.2xx:50000": channel is already closed
Exception in thread "Thread-22" java.lang.NoClassDefFoundError: org/eclipse/jgit/internal/JGitText
at org.eclipse.jgit.internal.util.ShutdownHook.cleanup(ShutdownHook.java:85)at java.base/java.lang.Thread.run(Unknown Source)

Additional Information:
Currently we are using Jenkins 2.401.3 and have 2 different agents in our docker image:

• jenkins-slave-java-app 3107.v665000b_51092
• jenkins-inbound-agent 4.13.3-1-jdk11
  Both of the agents having similar issue when running the job

Kindly let me know if you need some other information, Thank You!