Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63359

IllegalStateException after restart of Jenkins

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • kubernetes-plugin
    • None
    • Jenkins 2.235.3 (LTS)
      Kubernetes plugin 1.26.4

      If jenkins is restarted, existing pods end up rendered useless. Running jobs get stuck and when deleting the pod, it ends with the "Oops page" of Jenkins. The log of the jobs show the following stacktrace:

      java.lang.IllegalStateException: Not expecting pod template to be null at this point
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave.getTemplate(KubernetesSlave.java:91)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.secretsOf(SecretsMasker.java:144)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:122)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:94)
	at org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:94)
	at org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:138)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135)
	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297)
	at org.jenkinsci.plugins.workflow.cps.CpsBodySubContext.doGet(CpsBodySubContext.java:88)
	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:67)
	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.getListener(DefaultStepContext.java:116)
	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:79)
	at org.jenkinsci.plugins.workflow.cps.steps.ParallelStep$ResultHandler$Callback.onFailure(ParallelStep.java:133)
	at org.jenkinsci.plugins.workflow.cps.CpsBodyExecution$FailureAdapter.receive(CpsBodyExecution.java:361)
	at com.cloudbees.groovy.cps.impl.ThrowBlock$1.receive(ThrowBlock.java:68)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)

      The pod templates are defined in cloud configuration and have multiple labels.

      KubernetesSlave.java
          public PodTemplate getTemplate() {
        // Look up updated pod template after a restart
        if (template == null) {
            template = getKubernetesCloud().getTemplate(Label.get(Util.fixEmpty(getLabelString())));
            if (template == null) {
                throw new IllegalStateException("Not expecting pod template to be null at this point");
            }
        }
        return template;
    }

      Maybe a simple fix would be to store the name of the pod template like its already done with the cloud name.

          [JENKINS-63359] IllegalStateException after restart of Jenkins

          Dietmar Scheidl added a comment -

          PR: https://github.com/jenkinsci/kubernetes-plugin/pull/816

          Dietmar Scheidl added a comment - PR: https://github.com/jenkinsci/kubernetes-plugin/pull/816

          Julien Carsique added a comment -

          https://github.com/jenkinsci/kubernetes-plugin/pull/841 has been merged instead of #816 but the exception is still raised with Kubernetes plugin 1.27.3:

          java.lang.IllegalStateException: Not expecting pod template to be null at this point
	at org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave.getTemplate(KubernetesSlave.java:92)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.secretsOf(SecretsMasker.java:144)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:122)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:94)
	at org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:94)
	at org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:138)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135)
	at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297)
	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:67)
	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.getListener(DefaultStepContext.java:116)
	at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:79)

          Julien Carsique added a comment - https://github.com/jenkinsci/kubernetes-plugin/pull/841 has been merged instead of #816 but the exception is still raised with Kubernetes plugin 1.27.3: java.lang.IllegalStateException: Not expecting pod template to be null at this point at org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave.getTemplate(KubernetesSlave.java:92) at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.secretsOf(SecretsMasker.java:144) at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:122) at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:94) at org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:94) at org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:138) at org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:67) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.getListener(DefaultStepContext.java:116) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:79)

          Yves Schumann added a comment -

          Any news at this issue? We're spot the same and so our CI is not usable...

          Yves Schumann added a comment - Any news at this issue? We're spot the same and so our CI is not usable...

          Vincent Latombe added a comment -

          Fixed in 1.28.2

          Vincent Latombe added a comment - Fixed in 1.28.2

          Sithik Settu Mohamed added a comment -

          Issue still exists with 1.28.5 version. 

          Jenkins version: 2.266

          kubernetes plugin version: 1.28.5

          Description: Issue occurs when jenkins restarted.

          java.lang.IllegalStateException: Not expecting pod template to be null at this point
          at org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave.getTemplate(KubernetesSlave.java:92)
          at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.secretsOf(SecretsMasker.java:144)
          at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:122)
          at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:94)
          at org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:94)
          at org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:139)
          at org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135)
          at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297)
          at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:75)
          at org.jenkinsci.plugins.workflow.support.DefaultStepContext.getListener(DefaultStepContext.java:127)
          at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:79)
          at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:258)
          at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)
          at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

          Sithik Settu Mohamed added a comment - Issue still exists with 1.28.5 version.  Jenkins version: 2.266 kubernetes plugin version: 1.28.5 Description: Issue occurs when jenkins restarted. java.lang.IllegalStateException: Not expecting pod template to be null at this point at org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave.getTemplate(KubernetesSlave.java:92) at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.secretsOf(SecretsMasker.java:144) at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:122) at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker$Factory.get(SecretsMasker.java:94) at org.jenkinsci.plugins.workflow.steps.DynamicContext$Typed.get(DynamicContext.java:94) at org.jenkinsci.plugins.workflow.cps.ContextVariableSet.get(ContextVariableSet.java:139) at org.jenkinsci.plugins.workflow.cps.CpsThread.getContextVariable(CpsThread.java:135) at org.jenkinsci.plugins.workflow.cps.CpsStepContext.doGet(CpsStepContext.java:297) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:75) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.getListener(DefaultStepContext.java:127) at org.jenkinsci.plugins.workflow.support.DefaultStepContext.get(DefaultStepContext.java:79) at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:258) at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193) at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

          Sharath Surya added a comment -

          Hey scddev , Any update on this issue ?

          Sharath Surya added a comment - Hey scddev  , Any update on this issue ?

          Dietmar Scheidl added a comment -

          sharath_1280 my proposed solution was superseded by a change of vlatombe

          Dietmar Scheidl added a comment - sharath_1280 my proposed solution was superseded by a change of vlatombe

          Sharath Surya added a comment -

          Thanks for your reply scddev.

          Any idea on when would be the solution released back again ? (or) do you suggest us to downgrade the plugin (can you provide the version number) ?

          Sharath Surya added a comment - Thanks for your reply scddev . Any idea on when would be the solution released back again ? (or) do you suggest us to downgrade the plugin (can you provide the version number) ?

          Julio Morimoto added a comment -

          I don't understand why this is closed. The error has not been fixed.

          Jenkins 2.280
          Kubernetes plugin 1.29.0

          Julio Morimoto added a comment - I don't understand why this is closed. The error has not been fixed. Jenkins 2.280 Kubernetes plugin 1.29.0

          Vincent Latombe added a comment -

          sharath_1280 juliohm This is OSS software that is provided to you for free. If something doesn't work as you expect, you should provide valuable information (like environment and steps to reproduce) if you expect any help.
          Someone pointed out in JENKINS-64628 that remaining issue could be caused by usage of casc.

          Vincent Latombe added a comment - sharath_1280 juliohm This is OSS software that is provided to you for free. If something doesn't work as you expect, you should provide valuable information (like environment and steps to reproduce) if you expect any help. Someone pointed out in JENKINS-64628 that remaining issue could be caused by usage of casc.

            Unassigned Unassigned
            scddev Dietmar Scheidl
            Votes:
            7 Vote for this issue
            Watchers:
            11 Start watching this issue

              Created:
              Updated:
              Resolved: