Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65873

java.lang.OutOfMemoryError: unable to create new native thread

XMLWordPrintable

    • 2.362

      We regularly see issues with the jenkins/inbound-agent in our Jenkins logs on Kubernetes. It seems to occur in around 1% of all jobs.

      The error message is below.

      Whilst the error message refers to java.lang.OutOfMemoryError: and unable to create new native thread we have checked the pods and nodes in the cluster and there is always sufficient memory or threads available at the time of the error.

      The specific versions for this specific error message are:

      jenkins/inbound-agent:4.3-4

      Jenkins 2.263.4

      However we have also seen this error occur with different versions of both the inbound-agent and Jenkins.

      Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
	at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
	at hudson.remoting.Channel.call(Channel.java:1001)
	at hudson.FilePath.act(FilePath.java:1157)
	at hudson.FilePath.act(FilePath.java:1146)
	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603)
	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
	at jenkins.util.InterceptingScheduledExecutorService.schedule(InterceptingScheduledExecutorService.java:49)
	at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.reschedule(DelayBufferedOutputStream.java:72)
	at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.<init>(DelayBufferedOutputStream.java:68)
	at org.jenkinsci.plugins.workflow.log.BufferedBuildListener$Replacement.readResolve(BufferedBuildListener.java:77)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1260)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2133)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
	at hudson.remoting.UserRequest.deserialize(UserRequest.java:290)
	at hudson.remoting.UserRequest.perform(UserRequest.java:189)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
Caused: java.io.IOException: Remote call on JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138 failed
	at hudson.remoting.Channel.call(Channel.java:1007)
	at hudson.FilePath.act(FilePath.java:1157)
	at hudson.FilePath.act(FilePath.java:1146)
	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

        1. jstack_sample.txt
          2.42 MB

            jglick Jesse Glick
            wasimj Wasim
            Votes:
            9 Vote for this issue
            Watchers:
            30 Start watching this issue

              Created:
              Updated:
              Resolved: