-
Bug
-
Resolution: Unresolved
-
Major
-
Jenkins: 2.263.1
jenkins/inbound-agent:4.6-1
kubernetes-plugin: 1.28.5
durable-task-plugin: 1.35
Jobs do not resume after a master restart. Issue present in Jenkins 2.263.1 with kubernetes-plugin 1.28.5.
job console log:
Running on jenkins-agent-test-experimental-qtmr6 in /home/jenkins/agent/workspace/test
[Pipeline] {
[Pipeline] stage
[Pipeline] { (long-running-job)
[Pipeline] sh
Resuming build at Thu Jan 14 18:44:48 UTC 2021 after Jenkins restart
Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
Ready to run at Thu Jan 14 18:46:49 UTC 2021
logs from agent: * remoting.log.1:
Jan 14, 2021 6:39:59 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://jenkins-svc-default-experimental:8080/] Jan 14, 2021 6:39:59 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Jan 14, 2021 6:39:59 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check Jan 14, 2021 6:39:59 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Agent discovery successful Agent address: jenkins-svc-default-experimental Agent port: 50000 Identity: af:93:9a:2b:e2:a2:16:cc:f4:07:df:8b:e8:66:24:80 Jan 14, 2021 6:39:59 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Jan 14, 2021 6:39:59 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins-svc-default-experimental:50000 Jan 14, 2021 6:40:00 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP4-connect Jan 14, 2021 6:40:02 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Remote identity confirmed: af:93:9a:2b:e2:a2:16:cc:f4:07:df:8b:e8:66:24:80 Jan 14, 2021 6:40:10 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Jan 14, 2021 6:41:40 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Terminated Jan 14, 2021 6:41:51 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Failed to connect to the master. Will try again: java.net.ConnectException Connection refused (Connection refused) Jan 14, 2021 6:42:02 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Failed to connect to the master. Will try again: java.net.ConnectException Connection refused (Connection refused) Jan 14, 2021 6:42:14 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Failed to connect to the master. Will try again: java.net.ConnectException Connection refused (Connection refused) Jan 14, 2021 6:42:25 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Failed to connect to the master. Will try again: java.net.ConnectException Connection refused (Connection refused) Jan 14, 2021 6:42:36 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Failed to connect to the master. Will try again: java.net.ConnectException Connection refused (Connection refused) Jan 14, 2021 6:42:46 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Failed to connect to the master. Will try again: java.net.ConnectException Connection refused (Connection refused) Jan 14, 2021 6:42:58 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503 Jan 14, 2021 6:43:08 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503 Jan 14, 2021 6:43:18 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503 Jan 14, 2021 6:43:28 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503 Jan 14, 2021 6:43:38 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503 Jan 14, 2021 6:43:48 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503 Jan 14, 2021 6:43:59 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503 Jan 14, 2021 6:44:09 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503 Jan 14, 2021 6:44:19 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503 Jan 14, 2021 6:44:29 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503 Jan 14, 2021 6:44:39 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503 Jan 14, 2021 6:44:50 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503 Jan 14, 2021 6:45:00 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503 Jan 14, 2021 6:45:11 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Performing onReconnect operation. Jan 14, 2021 6:45:11 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$FindEffectiveRestarters$1 onReconnect INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@39ad2ce
- remoting.log.0
Jan 14, 2021 6:45:30 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Locating server among [http://jenkins-svc-default-experimental:8080/] Jan 14, 2021 6:45:31 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping] Jan 14, 2021 6:45:31 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check Jan 14, 2021 6:45:31 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Agent discovery successful Agent address: jenkins-svc-default-experimental Agent port: 50000 Identity: af:93:9a:2b:e2:a2:16:cc:f4:07:df:8b:e8:66:24:80 Jan 14, 2021 6:45:31 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Handshaking Jan 14, 2021 6:45:31 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connecting to jenkins-svc-default-experimental:50000 Jan 14, 2021 6:45:31 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Trying protocol: JNLP4-connect Jan 14, 2021 6:45:34 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Remote identity confirmed: af:93:9a:2b:e2:a2:16:cc:f4:07:df:8b:e8:66:24:80 Jan 14, 2021 6:45:43 PM hudson.remoting.jnlp.Main$CuiListener status INFO: Connected Jan 14, 2021 6:46:56 PM org.jenkinsci.plugins.durabletask.FileMonitoringTask$Watcher run WARNING: giving up on watching /home/jenkins/agent/workspace/test@tmp/durable-2321c62b java.lang.IllegalStateException: Not expecting pod template to be null at this point at org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave.getTemplate(KubernetesSlave.java:92) at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.secretsOf(SecretsMasker.java:144) at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:122) at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:94) at org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:94) at org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:139) at org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:75) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.getListener(DefaultStepContext.java:127) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:87) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.exited(DurableTaskStep.java:623) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:566) at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:936) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:909) at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:860) at hudson.remoting.UserRequest.perform(UserRequest.java:211) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:375) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73) at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18) at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:22) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.lang.Thread.run(Thread.java:834) Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection to jenkins-svc-default-experimental/172.20.249.199:50000 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) at hudson.remoting.Channel.call(Channel.java:1001) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286) at org.jenkinsci.plugins.workflow.steps.durable_task.$Proxy9.exited(Unknown Source) at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$HandlerImpl.exited(DurableTaskStep.java:743) at org.jenkinsci.plugins.durabletask.FileMonitoringTask$Watcher.run(FileMonitoringTask.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ...
logs from Log Recorder for `org.csanchez.jenkins.plugins.kubernetes` on Jenkins master:
Jan 14, 2021 6:39:52 PM FINEST org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher [MODIFIED] jenkins-agent-test-experimental-qtmr6 Jan 14, 2021 6:39:52 PM FINE org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher All containers are running for pod jenkins-agent-test-experimental-qtmr6 Jan 14, 2021 6:39:52 PM FINE org.csanchez.jenkins.plugins.kubernetes.TaskListenerEventWatcher TaskListenerEventWatcher onClose: jenkins-agent-test-experimental-qtmr6 Jan 14, 2021 6:39:52 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch Pod is running: k8s-experimental default/jenkins-agent-test-experimental-qtmr6
- is blocked by
-
JENKINS-64788 k8s agents unusable after restart
- Fixed but Unreleased
- is duplicated by
-
JENKINS-64788 k8s agents unusable after restart
- Fixed but Unreleased
-
JENKINS-64905 Pipeline jobs fail to connect to agents after jenkins controller restart
- Closed
- relates to
-
JENKINS-63359 IllegalStateException after restart of Jenkins
- Closed
- links to
(3 links to)