-
Bug
-
Resolution: Fixed
-
Major
-
-
2.362
We regularly see issues with the jenkins/inbound-agent in our Jenkins logs on Kubernetes. It seems to occur in around 1% of all jobs.
The error message is below.
Whilst the error message refers to java.lang.OutOfMemoryError: and unable to create new native thread we have checked the pods and nodes in the cluster and there is always sufficient memory or threads available at the time of the error.
The specific versions for this specific error message are:
jenkins/inbound-agent:4.3-4
Jenkins 2.263.4
However we have also seen this error occur with different versions of both the inbound-agent and Jenkins.
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:1001) at hudson.FilePath.act(FilePath.java:1157) at hudson.FilePath.act(FilePath.java:1146) at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603) at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334) at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) at jenkins.util.InterceptingScheduledExecutorService.schedule(InterceptingScheduledExecutorService.java:49) at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.reschedule(DelayBufferedOutputStream.java:72) at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.<init>(DelayBufferedOutputStream.java:68) at org.jenkinsci.plugins.workflow.log.BufferedBuildListener$Replacement.readResolve(BufferedBuildListener.java:77) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1260) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2133) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423) at hudson.remoting.UserRequest.deserialize(UserRequest.java:290) at hudson.remoting.UserRequest.perform(UserRequest.java:189) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117) Caused: java.io.IOException: Remote call on JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138 failed at hudson.remoting.Channel.call(Channel.java:1007) at hudson.FilePath.act(FilePath.java:1157) at hudson.FilePath.act(FilePath.java:1146) at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
- is duplicated by
-
JENKINS-70047 k8s cloud slave pod unable to create native thread: possibly out of memory or process/resource limits reached
-
- Closed
-
- is related to
-
JENKINS-64904 kubernetes agent: sometimes, java.lang.OutOfMemoryError: unable to create new native thread
-
- Open
-
- relates to
-
JENKINS-68199 java.lang.OutOfMemoryError: unable to create new native thread
-
- Closed
-
-
JENKINS-68954 Agent with remoting 4.14+ hangs up or show response time >5s
-
- Fixed but Unreleased
-
- links to
FWIW I've seen the same error sporadically since 2019, but with Swarm on the client side. It seems to occur once every few months in my company, so less than 1% of all builds (we do thousands of builds a month). In case it's helpful I've attached the stack trace from our internal bug tracker. Needless to say this is not reproducible. I do wonder if setting -Xmx or -Xms might help but I've never tried. I also wonder if this might be helped by JENKINS-61103.