Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65674

mesos stage agent fails when top-level agent is kubernetes (MesosSlave cannot be cast to class KubernetesSlave)

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      See the stack trace below:

      java.lang.ClassCastException: class org.jenkinsci.plugins.mesos.MesosSlave cannot be cast to class org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave (org.jenkinsci.plugins.mesos.MesosSlave is in unnamed module of loader hudson.ClassicPluginStrategy$AntClassLoader2 @342133d0; org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave is in unnamed module of loader hudson.ClassicPluginStrategy$AntClassLoader2 @4bef49f3)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:254)
      	at hudson.Launcher$ProcStarter.start(Launcher.java:508)
      	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:234)
      	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:103)
      	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:317)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:319)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:193)
      	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
      	at jdk.internal.reflect.GeneratedMethodAccessor824.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
      	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
      	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
      	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:135)
      	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
      	at WorkflowScript.run(WorkflowScript:279)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.delegateAndExecute(ModelInterpreter.groovy:137)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executeSingleStage(ModelInterpreter.groovy:661)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:395)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:393)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executeSingleStage(ModelInterpreter.groovy:660)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:288)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.toolsBlock(ModelInterpreter.groovy:544)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.toolsBlock(ModelInterpreter.groovy:543)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:276)
      	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withEnvBlock(ModelInterpreter.groovy:438)
      	at ___cps.transform___(Native Method)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
      	at jdk.internal.reflect.GeneratedMethodAccessor674.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:107)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
      	at jdk.internal.reflect.GeneratedMethodAccessor674.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:89)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
      	at jdk.internal.reflect.GeneratedMethodAccessor674.invoke(Unknown Source)
      	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:139)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
      	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
      	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
      	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
      	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      	at java.base/java.lang.Thread.run(Thread.java:834) 

       

      We are using the declarative pipelines and using a kubernetes agent at the top-level and then pinning one stage to a mesos agent, but the mesos one never starts running the steps.

       

      The jenkinsfiles looks like this (pseudo-code):

      pipeline {
        agent {
          kubernetes {
            defaultContainer 'main'
            yamlFile 'JenkinsKubernetesPod.yaml'
          }
        } 
        stages {
          stage('foo') {
          ...
          }
          stage('bar') {
          ...
          }
          stage('baz') {
            agent {
              label 'mesoslabel:dockerimage:latest'
            }
            steps {
              sh 'echo FOO'
            }
            post {
              failure {
                 gitHubNotify('FAILURE')
              }
            }
          }
        }
      }

       

      We do see the top level pod created and things run properly in stage foo and bar.

      When stage baz is called, a new jenkins agent running in mesos is spun up, the jnlp connection is established but the shell step is never run.

       

      The console logs show:

      [Pipeline] checkout
      19:04:21 The recommended git tool is: git
      19:04:23 using credential foo
      19:04:23 Warning: JENKINS-30600: special launcher org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1@6c1e4861; decorates RemoteLauncher[hudson.remoting.Channel@776a136a:JNLP4-connect connection from mesos] will be ignored (a typical symptom is the Git executable not being run inside a designated container)
      19:04:23 Cloning the remote Git repository
      Cloning repository edited
       > git init /tmp/jenkins/workspace/edited # timeout=10
      Fetching upstream changes from edited
       > git --version # timeout=10
       > git --version # 'git version 2.11.0'
      using GIT_ASKPASS to set credentials edited
       > git fetch --tags --progress -- edited +refs/heads/*:refs/remotes/origin/* # timeout=10
      19:04:27 Avoid second fetch
      19:04:27 Checking out Revision edited (main)
       > git config remote.origin.url edited # timeout=10
       > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
       > git config core.sparsecheckout # timeout=10
       > git checkout -f edited # timeout=10
      19:04:28 Commit message: "edited"
      [Pipeline] withEnv
      [Pipeline] {
      [Pipeline] withEnv
      [Pipeline] {
      [Pipeline] githubNotify
      [Pipeline] sh
      Post stage
      [Pipeline] githubNotify
      [Pipeline] script
      [Pipeline] {
      [Pipeline] wrap
      [Pipeline] {
      [Pipeline] }
      [Pipeline] // wrap
      [Pipeline] sh
      [Pipeline] }
      [Pipeline] // script
      Error when executing failure post condition: 

      It goes from '[Pipeline] sh' to 'Post stage' without returning any error.

       I'm not sure if the stack trace is a red herring but it comes up after 'Error when executing failure post condition: '

       

      I looked at the jenkins server jenkins.log file but there was no stack trace there either.

       

      It looks like its mixing the top level kubernetes agent with the stage level agent and that was not expected. There should not be any mention of kubernetes in the lower level agent declared step.

       

      I'm not sure if this is a kubernetes plugin issue or a main jenkinsfile/pipeline code issue.

       

      But setting the label to use a mesos cloud, I did not expect the code path to use the kubernetes plugin like

      org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator 

        Attachments

          Activity

          Hide
          sbeaulie Samuel Beaulieu added a comment - - edited

          I tried reversing the agents, where the top level pipeline agent is mesos and lower level stage agents are kubernetes and that works so there is some kind of bug that the top level kubernetes agent tries to take the lower level agents and cast them as KubernetesSlave

           

          https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L255

           

          ping Carlos Sanchez

          Show
          sbeaulie Samuel Beaulieu added a comment - - edited I tried reversing the agents, where the top level pipeline agent is mesos and lower level stage agents are kubernetes and that works so there is some kind of bug that the top level kubernetes agent tries to take the lower level agents and cast them as KubernetesSlave   https://github.com/jenkinsci/kubernetes-plugin/blob/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/pipeline/ContainerExecDecorator.java#L255   ping Carlos Sanchez
          Hide
          rroques Remi Roques added a comment -

          For your information, I am seeing the same behaviour when using an EC2 agent instead of a Mesos agent. 

          java.lang.ClassCastException: hudson.plugins.ec2.EC2OndemandSlave cannot be cast to org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave
          Show
          rroques Remi Roques added a comment - For your information, I am seeing the same behaviour when using an EC2 agent instead of a Mesos agent.  java.lang.ClassCastException: hudson.plugins.ec2.EC2OndemandSlave cannot be cast to org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            sbeaulie Samuel Beaulieu
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: