[JENKINS-42136] shared-library abstraction causing RejectedExecutionException when running sh() commands

Type: Bug
Resolution: Fixed
Priority: Major
Component/s: kubernetes-plugin
Labels:
None

Similar Issues:
Powered by SuggestiMate

Show

Hello,

When using this plug-in plus a rather abstracted shared-pipeline-library and tyring to run a sh() command on the kubernetes node/container, i get the following error

java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall$AsyncCall@4a046f09 rejected from java.util.concurrent.ThreadPoolExecutor@2314aa63[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 8]
	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
	at okhttp3.Dispatcher.enqueue(Dispatcher.java:129)
	at okhttp3.RealCall.enqueue(RealCall.java:78)
	at okhttp3.ws.WebSocketCall.enqueue(WebSocketCall.java:109)
	at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:210)
	at io.fabric8.kubernetes.client.dsl.internal.PodOperationsImpl.exec(PodOperationsImpl.java:54)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.ContainerExecDecorator$1.launch(ContainerExecDecorator.java:119)
	at hudson.Launcher$ProcStarter.start(Launcher.java:384)
	at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:157)
	at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:63)
	at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:172)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:184)
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:126)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:108)
	at groovy.lang.MetaClassImpl.invokeMethodOnGroovyObject(MetaClassImpl.java:1280)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1174)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:18)
	at deployDevBuild.call(/var/jenkins_home/jobs/trynew/builds/52/libs/sugarcrm-pipeline-library/vars/deployDevBuild.groovy:13)
	at helmNode.call(/var/jenkins_home/jobs/trynew/builds/52/libs/sugarcrm-pipeline-library/vars/helmNode.groovy:10)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
	at sun.reflect.GeneratedMethodAccessor806.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
	at com.cloudbees.groovy.cps.Next.step(Next.java:74)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:154)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:33)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:30)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:163)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:328)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:80)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:240)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:228)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:63)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I have my shared library setups and attached here

helmTemplate.groovy

#!/usr/bin/groovy
def call(Map parameters = [:], body) {

    def defaultLabel = "helm.${env.JOB_NAME}.${env.BUILD_NUMBER}".replace('-', '_').replace('/', '_')
    def defaultImage = "my.repo.com/kubernetes/helm-kubectl:latest"
    def label = parameters.get('label', defaultLabel)
    def helmImage = parameters.get('image', defaultImage)
    def inheritFrom = parameters.get('inheritFrom', 'base')

    podTemplate(label: label,  inheritFrom: "${inheritFrom}",
            containers: [
                    [name: 'helm', image: "${helmImage}", command: 'cat', ttyEnabled: true]
            ]) {
        body()
    }
}

helmNode.groovy

#!/usr/bin/groovy
def call(Map parameters = [:], body) {

    def defaultLabel = "helm.${env.JOB_NAME}.${env.BUILD_NUMBER}".replace('-', '_').replace('/', '_')
    def label = parameters.get('label', defaultLabel)

    helmTemplate(parameters) {
        node(label) {
            container('helm') {
                body()
            }
        }
    }
}

and then I call these in my Jenkins file by doing this

helmNode(parameters) {
        sh "helm version"
}

and it just bombs out with that error. It doesn't matter if the jenkins host in my kubernetes cluster or not.

What is weird is that if i don't abstract it out and just put everything in my Jenkinsfile, it works fine, but that isn't an option for us as we want as much reusable code as possible.

is duplicated by

JENKINS-55178 java.util.concurrent.RejectedExecutionException: Task okhttp3.RealCall

Closed

is related to

JENKINS-55392 java.util.concurrent.RejectedExecutionException AGAIN

Resolved

JENKINS-55138 Exec Failure: HTTP:0. Message:No response

Resolved

relates to

JENKINS-49700 nested pod template in library doesn't work as documented

Open

Jon Whitcraft created issue - 2017-02-17 05:15

Carlos Sanchez added a comment - 2017-03-12 09:32

This has to be a problem in the fabric8 kubernetes client or okhttp, have you tried building the git master version that has an updated k8s client ?

Carlos Sanchez added a comment - 2017-03-12 09:32 This has to be a problem in the fabric8 kubernetes client or okhttp, have you tried building the git master version that has an updated k8s client ?

Jon Whitcraft added a comment - 2017-03-12 16:01

I had not, I will try this and report back.

Jon Whitcraft added a comment - 2017-03-12 16:01 I had not, I will try this and report back.

Jon Whitcraft added a comment - 2017-03-12 17:16

csanchez that seems to have fixed it, but I didn't do extensive testing yet. I'll be able to do that tomorrow am. The only issue i'm seeing now is ~~JENKINS-42048~~.

Tomorrow I'll do more testing and let you know what I find. I can work around ~~JENKINS-42048~~ for now, but this issues seems to be fixed with the latest master.

Jon Whitcraft added a comment - 2017-03-12 17:16 csanchez that seems to have fixed it, but I didn't do extensive testing yet. I'll be able to do that tomorrow am. The only issue i'm seeing now is JENKINS-42048 . Tomorrow I'll do more testing and let you know what I find. I can work around JENKINS-42048 for now, but this issues seems to be fixed with the latest master.

Jon Whitcraft added a comment - 2017-03-22 17:56

csanchez,

From my extensive testing over the past week, this seems fixed with the latest master from the repo

Jon Whitcraft added a comment - 2017-03-22 17:56 csanchez , From my extensive testing over the past week, this seems fixed with the latest master from the repo

Jon Whitcraft added a comment - 2017-04-06 13:04

csanchez I started see this again yesterday when i tried to use inherit From again, even one i reverted the piepline, i had to restart the jenkins hosts to make it work again.

Jon Whitcraft added a comment - 2017-04-06 13:04 csanchez I started see this again yesterday when i tried to use inherit From again, even one i reverted the piepline, i had to restart the jenkins hosts to make it work again.

Carlos Sanchez added a comment - 2018-05-25 09:17

so this is an inheritFrom issue ?

Carlos Sanchez added a comment - 2018-05-25 09:17 so this is an inheritFrom issue ?

Carlos Sanchez made changes - 2018-05-25 09:17

Link

New: This issue relates to JENKINS-49700 [ JENKINS-49700 ]

Carlos Sanchez added a comment - 2018-11-28 10:51

I see RejectedExecutionException when more jobs than "Max connections to Kubernetes API" are running

Carlos Sanchez added a comment - 2018-11-28 10:51 I see RejectedExecutionException when more jobs than "Max connections to Kubernetes API" are running

Hayden Ball added a comment - 2018-12-03 15:17 - edited

I'm seeing this intermittently when using a shared declarative pipeline (i.e. https://jenkins.io/doc/book/pipeline/shared-libraries/#defining-declarative-pipelines).

Task okhttp3.RealCall$AsyncCall@29e44de7 rejected from java.util.concurrent.ThreadPoolExecutor@19b44afc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 77]

I should be running at well under "Max connections to Kubernetes API" (although I have just bumped that to see if it makes a difference), so I'm not convinced this is the issue. I'm seeing this with 3 concurrent jobs, with max connections set to 48.

Hayden Ball added a comment - 2018-12-03 15:17 - edited I'm seeing this intermittently when using a shared declarative pipeline (i.e. https://jenkins.io/doc/book/pipeline/shared-libraries/#defining-declarative-pipelines) . Task okhttp3.RealCall$AsyncCall@29e44de7 rejected from java.util.concurrent.ThreadPoolExecutor@19b44afc[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 77] I should be running at well under "Max connections to Kubernetes API" (although I have just bumped that to see if it makes a difference), so I'm not convinced this is the issue. I'm seeing this with 3 concurrent jobs, with max connections set to 48.

Assignee:: Carlos Sanchez

Reporter:: Jon Whitcraft

Votes:: 4 Vote for this issue

Watchers:: 15 Start watching this issue

Created:: 2017-02-17 05:15

Updated:: 2019-01-17 15:36

Resolved:: 2018-12-13 21:57

Jenkins

Details

Description

Attachments

Issue Links

Activity

Collapse comment: Carlos Sanchez added a comment - 2017-03-12 09:32

Expand comment: Carlos Sanchez added a comment - 2017-03-12 09:32

Collapse comment: Jon Whitcraft added a comment - 2017-03-12 16:01

Expand comment: Jon Whitcraft added a comment - 2017-03-12 16:01

Collapse comment: Jon Whitcraft added a comment - 2017-03-12 17:16

Expand comment: Jon Whitcraft added a comment - 2017-03-12 17:16

Collapse comment: Jon Whitcraft added a comment - 2017-03-22 17:56

Expand comment: Jon Whitcraft added a comment - 2017-03-22 17:56

Collapse comment: Jon Whitcraft added a comment - 2017-04-06 13:04

Expand comment: Jon Whitcraft added a comment - 2017-04-06 13:04

Collapse comment: Carlos Sanchez added a comment - 2018-05-25 09:17

Expand comment: Carlos Sanchez added a comment - 2018-05-25 09:17

Collapse comment: Carlos Sanchez added a comment - 2018-11-28 10:51

Expand comment: Carlos Sanchez added a comment - 2018-11-28 10:51

Collapse comment: Hayden Ball added a comment - 2018-12-03 15:17, Edited by Hayden Ball - 2018-12-03 15:24

Expand comment: Hayden Ball added a comment - 2018-12-03 15:17, Edited by Hayden Ball - 2018-12-03 15:24

People

Dates