Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-55392

java.util.concurrent.RejectedExecutionException AGAIN

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Blocker Blocker
    • kubernetes-plugin
    • None
    • 1.15.4

      I saw closed issue https://issues.jenkins-ci.org/browse/JENKINS-42136, but the problem still occurs for us (even with the newest plugin version).

      Stacktrace:

      java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@4b0f8dab rejected from java.util.concurrent.ThreadPoolExecutor@71ed53eb[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 7]
      	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
      	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
      	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
      	at okhttp3.Dispatcher.enqueue(Dispatcher.java:130)
      	at okhttp3.RealCall.enqueue(RealCall.java:100)
      	at okhttp3.internal.ws.RealWebSocket.connect(RealWebSocket.java:183)
      	at okhttp3.OkHttpClient.newWebSocket(OkHttpClient.java:436)
      	at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:267)
      	at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:61)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:319)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:237)
      	at hudson.Launcher$ProcStarter.start(Launcher.java:455)
      	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:188)
      	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:99)
      	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:278)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:270)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:178)
      	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
      	at sun.reflect.GeneratedMethodAccessor416.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
      	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
      	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
      	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
      Caused: java.io.IOException: Connection was rejected, you should increase the Max connections to Kubernetes API
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.doLaunch(ContainerExecDecorator.java:329)
      	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:237)
      	at hudson.Launcher$ProcStarter.start(Launcher.java:455)
      	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:188)
      	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:99)
      	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:278)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:270)
      	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:178)
      	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
      	at sun.reflect.GeneratedMethodAccessor416.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
      	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
      	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
      	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
      	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
      	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
      	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
      	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
      	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
      	at WorkflowScript.run(WorkflowScript:100)
      	at ___cps.transform___(Native Method)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor304.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:103)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor304.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:60)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
      	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
      	at sun.reflect.GeneratedMethodAccessor304.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
      	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
      	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
      	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
      	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
      	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
      	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
      	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
      	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
      	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
      	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
      	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Finished: FAILURE
      

      We noticed that issue exists, since you added kubernetes clients cache. It seems, that you close the kubernetes client (so also dispatcher executor service), which is still available in the cache.

          [JENKINS-55392] java.util.concurrent.RejectedExecutionException AGAIN

          the two pods starting issue is tracked in JENKINS-56347

          Carlos Sanchez added a comment - the two pods starting issue is tracked in JENKINS-56347

          Not sure if anyone else is facing this problem  When ever we are running shell commands from inside a running container we see  this problem of termination of pipeline 

          Caused: java.io.IOException: Connection was rejected, you should increase the Max connections to Kubernetes API 

           Pattern we tried  so far.

          • Reset the Max API connection numbers from default 32 to a higher number ( did not fix the problem)
          • Restarted the Jenkins instance that did not fix either
          • We tried to focus on the stage where the pipeline failed and it was executing the shell command once we commented the shell commands the replayed the pipeline build was successful 
          • After this we commentated the shell command instructions this time it went through

          Not sure if was fluke problem or some other behavior with the plugin that we are not aware of   We are not confident to  upgrade the plugin from current 1.13.8 version that we are running in our production instance. 

          Is anyone still experiencing this issue what is version we need to upgrade to avoid this particular issue ?

           

          Thank you all for any feed backs.

          Ravi.

           

          Ravi Venkatasubbaiah added a comment - Not sure if anyone else is facing this problem  When ever we are running shell commands from inside a running container we see  this problem of termination of pipeline  Caused: java.io.IOException: Connection was rejected, you should increase the Max connections to Kubernetes API  Pattern we tried  so far. Reset the Max API connection numbers from default 32 to a higher number ( did not fix the problem) Restarted the Jenkins instance that did not fix either We tried to focus on the stage where the pipeline failed and it was executing the shell command once we commented the shell commands the replayed the pipeline build was successful  After this we commentated the shell command instructions this time it went through Not sure if was fluke problem or some other behavior with the plugin that we are not aware of   We are not confident to  upgrade the plugin from current 1.13.8 version that we are running in our production instance.  Is anyone still experiencing this issue what is version we need to upgrade to avoid this particular issue ?   Thank you all for any feed backs. Ravi.  

          any update on the problem?

          We still see similar issues:

          kubernetes plugin 1.14.9, jenkins 2.170

          [Pipeline] { (Delete old jobs)
          [Pipeline] sh
          java.io.InterruptedIOException: executor rejected
          	at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:185)
          	at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:186)...
          
          
          Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@192d2ae5 rejected from java.util.concurrent.ThreadPoolExecutor@698d8fe9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 228] ...
          
          hudson.remoting.ProxyException: io.fabric8.kubernetes.client.KubernetesClientException: No response
          	at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.onFailure(ExecWebSocketListener.java:230)
          	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571)
          	at okhttp3.internal.ws.RealWebSocket$2.onFailure(RealWebSocket.java:221)
          	at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:188)
          	at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:186)
          
          
          

           

           

          Amadeu Carvalho added a comment - any update on the problem? We still see similar issues: kubernetes plugin 1.14.9, jenkins 2.170 [Pipeline] { (Delete old jobs) [Pipeline] sh java.io.InterruptedIOException: executor rejected at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:185) at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:186)... Caused by: java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@192d2ae5 rejected from java.util.concurrent.ThreadPoolExecutor@698d8fe9[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 228] ... hudson.remoting.ProxyException: io.fabric8.kubernetes.client.KubernetesClientException: No response at io.fabric8.kubernetes.client.dsl.internal.ExecWebSocketListener.onFailure(ExecWebSocketListener.java:230) at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) at okhttp3.internal.ws.RealWebSocket$2.onFailure(RealWebSocket.java:221) at okhttp3.RealCall$AsyncCall.executeOn(RealCall.java:188) at okhttp3.Dispatcher.promoteAndExecute(Dispatcher.java:186)    

          hello Sanket,

          I'm having the same problem

          Can you solve it?

          Amadeu Carvalho added a comment - hello Sanket, I'm having the same problem Can you solve it?

          Jesse Glick added a comment -

          kubernetes plugin 1.14.9

          Please do not use JIRA to discuss issues present in obsolete software versions.

          Jesse Glick added a comment - kubernetes plugin 1.14.9 Please do not use JIRA to discuss issues present in obsolete software versions.

          Karol Gil added a comment -

          We are running into the same issue with version 1.17.2 with no `containerLog` steps used - every time is happens it's on `sh` step. Increasing max connections, even to absurd numbers (100k or more) does not help at all. If I can provide any logs or information that could be useful in finally solving that please let me know.

          Karol Gil added a comment - We are running into the same issue with version 1.17.2 with no `containerLog` steps used - every time is happens it's on `sh` step. Increasing max connections, even to absurd numbers (100k or more) does not help at all. If I can provide any logs or information that could be useful in finally solving that please let me know.

          Jesse Glick added a comment -

          karolgil better to file a fresh bug report with as much detail as you can and Link it to this one.

          Jesse Glick added a comment - karolgil better to file a fresh bug report with as much detail as you can and Link it to this one.

          Karol Gil added a comment -

          I think I solved it in my case, maybe it'll help others*.*
          TL;DR; If your API tokens have limited lifetime then decrease cache expiry time.

          As documented in README of this plugin EKS (it's our case) clusters use aws-iam-authenticator and tokens grantend to kubernetes clients have limited lifetime (15 minutes). The default expiry time for clients in the plugin is 60 minutes which means that plugin can create and cache client, which will become unauthorized in 15 minutes. I've decrease cache expiry to 15 seconds and clients purge time to 120 seconds (to avoid keeping all those expired in memory) via system properties:

          {{ -Dorg.csanchez.jenkins.plugins.kubernetes.clients.expiredClientsPurgeTime=120}}
          {{ -Dorg.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=30}}

          Karol Gil added a comment - I think I solved it in my case, maybe it'll help others*.* TL;DR; If your API tokens have limited lifetime then decrease cache expiry time. As documented in README of this plugin EKS (it's our case) clusters use aws-iam-authenticator and tokens grantend to kubernetes clients have limited lifetime (15 minutes). The default expiry time for clients in the plugin is 60 minutes which means that plugin can create and cache client, which will become unauthorized in 15 minutes. I've decrease cache expiry to 15 seconds and clients purge time to 120 seconds (to avoid keeping all those expired in memory) via system properties: {{ -Dorg.csanchez.jenkins.plugins.kubernetes.clients.expiredClientsPurgeTime=120}} {{ -Dorg.csanchez.jenkins.plugins.kubernetes.clients.cacheExpiration=30}}

          Jesse Glick added a comment -

          karolgil that topic is discussed in JENKINS-58143. I am not aware of any mechanism by which it could cause the RejectedExecutionException, but perhaps that is one failure mode.

          Jesse Glick added a comment - karolgil that topic is discussed in JENKINS-58143 . I am not aware of any mechanism by which it could cause the RejectedExecutionException , but perhaps that is one failure mode.

          Karol Gil added a comment -

          I'm not sure if this is the case - the fact is that we saw RejectedExecutionException and log about increasing max connection, but of course it might be weird failure mode.
          I posted it here as maybe it'll be helpful to someone in similar environment. For now I won't open another issue as it seems to be resolved.

          Karol Gil added a comment - I'm not sure if this is the case - the fact is that we saw RejectedExecutionException and log about increasing max connection, but of course it might be weird failure mode. I posted it here as maybe it'll be helpful to someone in similar environment. For now I won't open another issue as it seems to be resolved.

            csanchez Carlos Sanchez
            dpacierpnik Damian Pacierpnik
            Votes:
            23 Vote for this issue
            Watchers:
            37 Start watching this issue

              Created:
              Updated:
              Resolved: