[JENKINS-65873] java.lang.OutOfMemoryError: unable to create new native thread - Jenkins Jira

Details

Type: Bug
Status: Closed (View Workflow)
Priority: Major
Resolution: Fixed
Component/s: remoting
Labels:

Similar Issues:

Show
Released As:
2.362

Description

We regularly see issues with the jenkins/inbound-agent in our Jenkins logs on Kubernetes. It seems to occur in around 1% of all jobs.

The error message is below.

Whilst the error message refers to java.lang.OutOfMemoryError: and unable to create new native thread we have checked the pods and nodes in the cluster and there is always sufficient memory or threads available at the time of the error.

The specific versions for this specific error message are:

jenkins/inbound-agent:4.3-4

Jenkins 2.263.4

However we have also seen this error occur with different versions of both the inbound-agent and Jenkins.

Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
	at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
	at hudson.remoting.Channel.call(Channel.java:1001)
	at hudson.FilePath.act(FilePath.java:1157)
	at hudson.FilePath.act(FilePath.java:1146)
	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603)
	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
	at jenkins.util.InterceptingScheduledExecutorService.schedule(InterceptingScheduledExecutorService.java:49)
	at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.reschedule(DelayBufferedOutputStream.java:72)
	at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.<init>(DelayBufferedOutputStream.java:68)
	at org.jenkinsci.plugins.workflow.log.BufferedBuildListener$Replacement.readResolve(BufferedBuildListener.java:77)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1260)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2133)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
	at hudson.remoting.UserRequest.deserialize(UserRequest.java:290)
	at hudson.remoting.UserRequest.perform(UserRequest.java:189)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
Caused: java.io.IOException: Remote call on JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138 failed
	at hudson.remoting.Channel.call(Channel.java:1007)
	at hudson.FilePath.act(FilePath.java:1157)
	at hudson.FilePath.act(FilePath.java:1146)
	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

jstack_sample.txt
2.42 MB
2021-09-09 12:45

Issue Links

is duplicated by

JENKINS-70047 k8s cloud slave pod unable to create native thread: possibly out of memory or process/resource limits reached

Closed

is related to

JENKINS-64904 kubernetes agent: sometimes, java.lang.OutOfMemoryError: unable to create new native thread

Open

relates to

JENKINS-68199 java.lang.OutOfMemoryError: unable to create new native thread

Closed

JENKINS-68954 Agent with remoting 4.14+ hangs up or show response time >5s

Fixed but Unreleased

links to

remoting #559

Remoting PR #505

(1 links to)

Activity

Ascending order - Click to sort in descending order

View 59 older comments

Basil Crow added a comment - 2022-08-16 16:10

I upgraded to […] 4.10-3-jdk11

kryan90 You aren't running with the fix. The fix is in Remoting 4.13.3 (LTS line) and Remoting 3044.vb_940a_a_e4f72e (weekly line). Java 11.0.16 is also recommended.

Basil Crow added a comment - 2022-08-16 16:10 I upgraded to […] 4.10-3-jdk11 kryan90 You aren't running with the fix. The fix is in Remoting 4.13.3 (LTS line) and Remoting 3044.vb_940a_a_e4f72e (weekly line). Java 11.0.16 is also recommended.

Donald Gobin added a comment - 2022-09-01 14:43

basil Definitely interested in this fix. We run the Docker containers for the controller and agent side, can you share which tags of these include this fix ? For instance, I don't see a 4.13 release for the agent side here - https://hub.docker.com/r/jenkins/inbound-agent/tags?page=1&name=jdk11. Is an update required on both the controller and agent side ? As Kevin tried above assuming that the fix is on the controller side only and used the latest available tagged image for the agent - 4.10. So, just need some clarity on which Docker tags we need to use here. Thanks.

Donald Gobin added a comment - 2022-09-01 14:43 basil Definitely interested in this fix. We run the Docker containers for the controller and agent side, can you share which tags of these include this fix ? For instance, I don't see a 4.13 release for the agent side here - https://hub.docker.com/r/jenkins/inbound-agent/tags?page=1&name=jdk11 . Is an update required on both the controller and agent side ? As Kevin tried above assuming that the fix is on the controller side only and used the latest available tagged image for the agent - 4.10. So, just need some clarity on which Docker tags we need to use here. Thanks.

Basil Crow added a comment - 2022-09-01 14:47

I don't know anything about how the Docker images for agents are built or what versions of Remoting they include in them.

Basil Crow added a comment - 2022-09-01 14:47 I don't know anything about how the Docker images for agents are built or what versions of Remoting they include in them.

Donald Gobin added a comment - 2022-09-01 15:01 - edited

basil Another question - do both the controller and agent have to have this fix/version ? Also, which component do I raise a ticket for to address the inbound-agent Docker image side of this ?

Donald Gobin added a comment - 2022-09-01 15:01 - edited basil Another question - do both the controller and agent have to have this fix/version ? Also, which component do I raise a ticket for to address the inbound-agent Docker image side of this ?

Basil Crow added a comment - 2022-09-01 15:36

The fix is in remoting.jar which is the main JAR for running the agent process. That having been said, that JAR file gets shipped over from the controller in some scenarios (e.g. SSH Build Agents, where the controller uses SSH to connect to the agent, ship over its copy of remoting.jar, and then start the agent process), so in some sense the controller version is relevant in the sense that it does bundle remoting.jar to be used in certain agent scenarios. That having been said I think your Docker image use case bundles its own copy of remoting.jar completely separate from the controller's version. I believe the maintainers of the Docker images use GitHub issues on the corresponding GitHub repositories to track issues.

Basil Crow added a comment - 2022-09-01 15:36 The fix is in remoting.jar which is the main JAR for running the agent process. That having been said, that JAR file gets shipped over from the controller in some scenarios (e.g. SSH Build Agents, where the controller uses SSH to connect to the agent, ship over its copy of remoting.jar , and then start the agent process), so in some sense the controller version is relevant in the sense that it does bundle remoting.jar to be used in certain agent scenarios. That having been said I think your Docker image use case bundles its own copy of remoting.jar completely separate from the controller's version. I believe the maintainers of the Docker images use GitHub issues on the corresponding GitHub repositories to track issues.

Jenkins

java.lang.OutOfMemoryError: unable to create new native thread

Details

Description

Attachments

Attachments

Issue Links

Activity

People

Dates