Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64628

Stop using random PodTemplate.id (was: Kubernetes Plugin IllegalStateException: Not expecting pod template to be null at this point)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • kubernetes-plugin
    • Jenkins: 2.263.1
      jenkins/inbound-agent:4.6-1
      kubernetes-plugin: 1.28.5
      durable-task-plugin: 1.35

      Jobs do not resume after a master restart. Issue present in Jenkins 2.263.1 with kubernetes-plugin 1.28.5. 

       
      job console log:

      Running on jenkins-agent-test-experimental-qtmr6 in /home/jenkins/agent/workspace/test
      [Pipeline] {
      [Pipeline] stage
      [Pipeline] { (long-running-job)
      [Pipeline] sh
      Resuming build at Thu Jan 14 18:44:48 UTC 2021 after Jenkins restart
      Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
      Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
      Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
      Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
      Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
      Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
      Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
      Waiting to resume part of test #14: ‘jenkins-agent-test-experimental-qtmr6’ is offline
      Ready to run at Thu Jan 14 18:46:49 UTC 2021
      

       
      logs from agent: * remoting.log.1:

       

      Jan 14, 2021 6:39:59 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Locating server among [http://jenkins-svc-default-experimental:8080/]
      Jan 14, 2021 6:39:59 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
      Jan 14, 2021 6:39:59 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
      Jan 14, 2021 6:39:59 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Agent discovery successful
        Agent address: jenkins-svc-default-experimental
        Agent port:    50000
        Identity:      af:93:9a:2b:e2:a2:16:cc:f4:07:df:8b:e8:66:24:80
      Jan 14, 2021 6:39:59 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Handshaking
      Jan 14, 2021 6:39:59 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to jenkins-svc-default-experimental:50000
      Jan 14, 2021 6:40:00 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP4-connect
      Jan 14, 2021 6:40:02 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Remote identity confirmed: af:93:9a:2b:e2:a2:16:cc:f4:07:df:8b:e8:66:24:80
      Jan 14, 2021 6:40:10 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connected
      Jan 14, 2021 6:41:40 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Terminated
      Jan 14, 2021 6:41:51 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Failed to connect to the master. Will try again: java.net.ConnectException Connection refused (Connection refused)
      Jan 14, 2021 6:42:02 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Failed to connect to the master. Will try again: java.net.ConnectException Connection refused (Connection refused)
      Jan 14, 2021 6:42:14 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Failed to connect to the master. Will try again: java.net.ConnectException Connection refused (Connection refused)
      Jan 14, 2021 6:42:25 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Failed to connect to the master. Will try again: java.net.ConnectException Connection refused (Connection refused)
      Jan 14, 2021 6:42:36 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Failed to connect to the master. Will try again: java.net.ConnectException Connection refused (Connection refused)
      Jan 14, 2021 6:42:46 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Failed to connect to the master. Will try again: java.net.ConnectException Connection refused (Connection refused)
      Jan 14, 2021 6:42:58 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503
      Jan 14, 2021 6:43:08 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503
      Jan 14, 2021 6:43:18 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503
      Jan 14, 2021 6:43:28 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503
      Jan 14, 2021 6:43:38 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503
      Jan 14, 2021 6:43:48 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503
      Jan 14, 2021 6:43:59 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503
      Jan 14, 2021 6:44:09 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503
      Jan 14, 2021 6:44:19 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503
      Jan 14, 2021 6:44:29 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503
      Jan 14, 2021 6:44:39 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503
      Jan 14, 2021 6:44:50 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503
      Jan 14, 2021 6:45:00 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver waitForReady
      INFO: Master isn't ready to talk to us on http://jenkins-svc-default-experimental:8080/tcpSlaveAgentListener/. Will try again: response code=503
      Jan 14, 2021 6:45:11 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Performing onReconnect operation.
      Jan 14, 2021 6:45:11 PM jenkins.slaves.restarter.JnlpSlaveRestarterInstaller$FindEffectiveRestarters$1 onReconnect
      INFO: Restarting agent via jenkins.slaves.restarter.UnixSlaveRestarter@39ad2ce
      

       

      • remoting.log.0 

       

      Jan 14, 2021 6:45:30 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Locating server among [http://jenkins-svc-default-experimental:8080/]
      Jan 14, 2021 6:45:31 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
      Jan 14, 2021 6:45:31 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
      INFO: Remoting TCP connection tunneling is enabled. Skipping the TCP Agent Listener Port availability check
      Jan 14, 2021 6:45:31 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Agent discovery successful
        Agent address: jenkins-svc-default-experimental
        Agent port:    50000
        Identity:      af:93:9a:2b:e2:a2:16:cc:f4:07:df:8b:e8:66:24:80
      Jan 14, 2021 6:45:31 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Handshaking
      Jan 14, 2021 6:45:31 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connecting to jenkins-svc-default-experimental:50000
      Jan 14, 2021 6:45:31 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Trying protocol: JNLP4-connect
      Jan 14, 2021 6:45:34 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Remote identity confirmed: af:93:9a:2b:e2:a2:16:cc:f4:07:df:8b:e8:66:24:80
      Jan 14, 2021 6:45:43 PM hudson.remoting.jnlp.Main$CuiListener status
      INFO: Connected
      Jan 14, 2021 6:46:56 PM org.jenkinsci.plugins.durabletask.FileMonitoringTask$Watcher run
      WARNING: giving up on watching /home/jenkins/agent/workspace/test@tmp/durable-2321c62b
      java.lang.IllegalStateException: Not expecting pod template to be null at this point
      	at org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave.getTemplate(KubernetesSlave.java:92)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.secretsOf(SecretsMasker.java:144)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:122)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:94)
      	at org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:94)
      	at org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:139)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135)
      	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297)
      	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:75)
      	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.getListener(DefaultStepContext.java:127)
      	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:87)
      	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.exited(DurableTaskStep.java:623)
      	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:566)
      	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:936)
      	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:909)
      	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:860)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:375)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
      	at org.jenkinsci.remoting.CallableDecorator.call(CallableDecorator.java:18)
      	at hudson.remoting.CallableDecoratorList$1.call(CallableDecoratorList.java:22)
      	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
      	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.lang.Thread.run(Thread.java:834)
      	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection to jenkins-svc-default-experimental/172.20.249.199:50000
      		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
      		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
      		at hudson.remoting.Channel.call(Channel.java:1001)
      		at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:286)
      		at org.jenkinsci.plugins.workflow.steps.durable_task.$Proxy9.exited(Unknown Source)
      		at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$HandlerImpl.exited(DurableTaskStep.java:743)
      		at org.jenkinsci.plugins.durabletask.FileMonitoringTask$Watcher.run(FileMonitoringTask.java:531)
      		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
      		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      		at java.lang.Thread.run(Thread.java:748)
      ...
      

       

      logs from Log Recorder for `org.csanchez.jenkins.plugins.kubernetes` on Jenkins master:

      Jan 14, 2021 6:39:52 PM FINEST org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher
      [MODIFIED] jenkins-agent-test-experimental-qtmr6
      Jan 14, 2021 6:39:52 PM FINE org.csanchez.jenkins.plugins.kubernetes.AllContainersRunningPodWatcher
      All containers are running for pod jenkins-agent-test-experimental-qtmr6
      Jan 14, 2021 6:39:52 PM FINE org.csanchez.jenkins.plugins.kubernetes.TaskListenerEventWatcher
      TaskListenerEventWatcher onClose: jenkins-agent-test-experimental-qtmr6
      Jan 14, 2021 6:39:52 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
      Pod is running: k8s-experimental default/jenkins-agent-test-experimental-qtmr6
      

            vlatombe Vincent Latombe
            mdalton_ti Michael
            Votes:
            1 Vote for this issue
            Watchers:
            13 Start watching this issue

              Created:
              Updated: