Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42136

shared-library abstraction causing RejectedExecutionException when running sh() commands

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • kubernetes-plugin
    • None

      Hello,

      When using this plug-in plus a rather abstracted shared-pipeline-library and tyring to run a sh() command on the kubernetes node/container, i get the following error

      java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@4a046f09 rejected from java.util.concurrent.ThreadPoolExecutor@2314aa63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8]
      	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
      	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
      	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
      	at okhttp3.Dispatcher.enqueue(Dispatcher.java:129)
      	at okhttp3.RealCall.enqueue(RealCall.java:78)
      	at okhttp3.ws.WebSocketCall.enqueue(WebSocketCall.java:109)
      	at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:210)
      	at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:54)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:119)
      	at hudson.Launcher$ProcStarter.start(Launcher.java:384)
      	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:157)
      	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:63)
      	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:172)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
      	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
      	at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1280)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1174)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024)
      	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:18)
      	at deployDevBuild.call(/var/jenkins_home/jobs/trynew/builds/52/libs/sugarcrm-pipeline-library/vars/deployDevBuild.groovy:13)
      	at helmNode.call(/var/jenkins_home/jobs/trynew/builds/52/libs/sugarcrm-pipeline-library/vars/helmNode.groovy:10)
      	at ___cps.transform___(Native Method)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor806.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:74)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      I have my shared library setups and attached here

      helmTemplate.groovy
      #!/usr/bin/groovy
      def call(Map parameters = [:], body) {
      
          def defaultLabel = "helm.${env.JOB_NAME}.${env.BUILD_NUMBER}".replace('-', '_').replace('/', '_')
          def defaultImage = "my.repo.com/kubernetes/helm-kubectl:latest"
          def label = parameters.get('label', defaultLabel)
          def helmImage = parameters.get('image', defaultImage)
          def inheritFrom = parameters.get('inheritFrom', 'base')
      
          podTemplate(label: label,  inheritFrom: "${inheritFrom}",
                  containers: [
                          [name: 'helm', image: "${helmImage}", command: 'cat', ttyEnabled: true]
                  ]) {
              body()
          }
      }
      
      helmNode.groovy
      #!/usr/bin/groovy
      def call(Map parameters = [:], body) {
      
          def defaultLabel = "helm.${env.JOB_NAME}.${env.BUILD_NUMBER}".replace('-', '_').replace('/', '_')
          def label = parameters.get('label', defaultLabel)
      
          helmTemplate(parameters) {
              node(label) {
                  container('helm') {
                      body()
                  }
              }
          }
      }
      

      and then I call these in my Jenkins file by doing this

      helmNode(parameters) {
              sh "helm version"
      }
      

      and it just bombs out with that error. It doesn't matter if the jenkins host in my kubernetes cluster or not.

      What is weird is that if i don't abstract it out and just put everything in my Jenkinsfile, it works fine, but that isn't an option for us as we want as much reusable code as possible.

          [JENKINS-42136] shared-library abstraction causing RejectedExecutionException when running sh() commands

          Julien Duchesne added a comment - - edited

          This has started happening systematically recently on our side. I have a job that calls a `sh` every two minutes for ~an hour and it fails after ~30 minutes every time. I have set the Max connections to Kubernetes API to 200 and I am running a single job so it shouldn't be the issue. I am running the latest plugin version (1.13.7). I am reverting to the version I was before (1.12.1), I'll let you know if that fixes my issue.

          Julien Duchesne added a comment - - edited This has started happening systematically recently on our side. I have a job that calls a `sh` every two minutes for ~an hour and it fails after ~30 minutes every time. I have set the Max connections to Kubernetes API to 200 and I am running a single job so it shouldn't be the issue. I am running the latest plugin version (1.13.7). I am reverting to the version I was before (1.12.1), I'll let you know if that fixes my issue.

          Julien Duchesne added a comment - - edited

          Update: I've pinpointed the bug in version 1.13.6.

          1.13.6: Systematic issues

          1.13.5: Was working fine for weeks (another jenkins instance than the one on 1.12.1 )

          Julien Duchesne added a comment - - edited Update: I've pinpointed the bug in version 1.13.6. 1.13.6: Systematic issues 1.13.5: Was working fine for weeks (another jenkins instance than the one on 1.12.1 )

          Steffen Seckler added a comment - - edited

          can confirm this issue, seeing this with kubernetes plugin 1.13.7.

          Increased the max number of connections to the kubernetes api (according to https://github.com/jenkinsci/kubernetes-plugin/blob/master/CHANGELOG.md#1136), will check whether this helps.

           

          Edit:

          increasing the max number of connections to the kubernetes api actually helped

          Steffen Seckler added a comment - - edited can confirm this issue, seeing this with kubernetes plugin 1.13.7. Increased the max number of connections to the kubernetes api (according to https://github.com/jenkinsci/kubernetes-plugin/blob/master/CHANGELOG.md#1136 ), will check whether this helps.   Edit: increasing the max number of connections to the kubernetes api actually helped

          Yo-An Lin added a comment -

          can confirm this issue. this happens even if you updated the kubernetes plugin to 1.13.7

          Jenkins 2.150.1

          Yo-An Lin added a comment - can confirm this issue. this happens even if you updated the kubernetes plugin to 1.13.7 Jenkins 2.150.1

          Downgrade to 1.13.5 to fix the issue

          Carlos Sanchez added a comment - Downgrade to 1.13.5 to fix the issue

          This should be fixed when JENKINS-55138 gets fixed

          Carlos Sanchez added a comment - This should be fixed when JENKINS-55138 gets fixed

          This should have been fixed in 1.13.8

          Carlos Sanchez added a comment - This should have been fixed in 1.13.8

          csanchez On Kubernetes plugin 1.13.8 and still getting this

          Error occurred while checking out Git: Task okhttp3.RealCall$AsyncCall@789b60f9 rejected from java.util.concurrent.ThreadPoolExecutor@67ba2f6e[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 14]

           

          Krzysztof Mulica added a comment - csanchez On Kubernetes plugin 1.13.8 and still getting this Error occurred while checking out Git: Task okhttp3.RealCall$AsyncCall@789b60f9 rejected from java.util.concurrent.ThreadPoolExecutor@67ba2f6e [Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 14]  

          Carlos Sanchez added a comment - see JENKINS-55392

          Thanks, missed the related to link 

          Krzysztof Mulica added a comment - Thanks, missed the related to link 

            csanchez Carlos Sanchez
            jwhitcraft Jon Whitcraft
            Votes:
            4 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved: