Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65873

java.lang.OutOfMemoryError: unable to create new native thread

    XMLWordPrintable

Details

    Description

      We regularly see issues with the jenkins/inbound-agent in our Jenkins logs on Kubernetes. It seems to occur in around 1% of all jobs.

      The error message is below.

      Whilst the error message refers to java.lang.OutOfMemoryError: and unable to create new native thread we have checked the pods and nodes in the cluster and there is always sufficient memory or threads available at the time of the error.

      The specific versions for this specific error message are:

      jenkins/inbound-agent:4.3-4

      Jenkins 2.263.4

      However we have also seen this error occur with different versions of both the inbound-agent and Jenkins.

      Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
	at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
	at hudson.remoting.Channel.call(Channel.java:1001)
	at hudson.FilePath.act(FilePath.java:1157)
	at hudson.FilePath.act(FilePath.java:1146)
	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Thread.java:717)
	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
	at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603)
	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
	at jenkins.util.InterceptingScheduledExecutorService.schedule(InterceptingScheduledExecutorService.java:49)
	at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.reschedule(DelayBufferedOutputStream.java:72)
	at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.<init>(DelayBufferedOutputStream.java:68)
	at org.jenkinsci.plugins.workflow.log.BufferedBuildListener$Replacement.readResolve(BufferedBuildListener.java:77)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1260)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2133)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
	at hudson.remoting.UserRequest.deserialize(UserRequest.java:290)
	at hudson.remoting.UserRequest.perform(UserRequest.java:189)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
Caused: java.io.IOException: Remote call on JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138 failed
	at hudson.remoting.Channel.call(Channel.java:1007)
	at hudson.FilePath.act(FilePath.java:1157)
	at hudson.FilePath.act(FilePath.java:1146)
	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904)
	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

      Attachments

        Issue Links

          is related to

          Improvement - An improvement or enhancement to an existing feature or task. JENKINS-64904 kubernetes agent: sometimes, java.lang.OutOfMemoryError: unable to create new native thread

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. JENKINS-68199 java.lang.OutOfMemoryError: unable to create new native thread

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          links to

          Web Link Remoting PR #505

          Activity

            Ascending order - Click to sort in descending order
            View 53 older comments
            matthewrgomes Matthew Gomes added a comment -

            rhinoceros Where did you set the git sleep? is this the git plugin

            matthewrgomes Matthew Gomes added a comment - rhinoceros Where did you set the git sleep? is this the git plugin
            rhinoceros rhinoceros.xn added a comment -

            matthewrgomes 

             

            before git OR checkout step:

            // code placeholder
    node(label) {
        stage('xxzzxasd') {
            container('xxxx') {
                        stage('git clone') {

                            sleep(10) //** HERE: adding sleep(10) before git checkout **

                            git branch: 'master', credentialsId: '', url: 'git@xxx.com:xxx/xxxxxxx.git'
            rhinoceros rhinoceros.xn added a comment - matthewrgomes     before git OR checkout step: // code placeholder node(label) { stage( 'xxzzxasd' ) { container( 'xxxx' ) { stage( 'git clone' ) { sleep(10) //** HERE: adding sleep(10) before git checkout ** git branch: 'master' , credentialsId: '', url: ' git@xxx.com:xxx/xxxxxxx.git'
            dg424 Donald Gobin added a comment -

            rhinoceros so if this is true, then it indicates that the issue is with some kind of race condition within the jenkins pipeline flow that is causing this to happen. your sleep essentially looks like waiting for all jenkins threads for that pipeline to all complete before continuing ?

            dg424 Donald Gobin added a comment - rhinoceros so if this is true, then it indicates that the issue is with some kind of race condition within the jenkins pipeline flow that is causing this to happen. your sleep essentially looks like waiting for all jenkins threads for that pipeline to all complete before continuing ?
            rhinoceros rhinoceros.xn added a comment -

            dg424 yes. I think so.

             

            I had updated agent.jar two weeks ago with https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/remoting/4.14-rc3000.5949ea_7370a_f/remoting-4.14-rc3000.5949ea_7370a_f.jar for hours ,  the problem had not diminished.

             

            I found that the problem is only at git checkout, so try to add sleep before git checkout

            rhinoceros rhinoceros.xn added a comment - dg424 yes. I think so.   I had updated agent.jar two weeks ago with https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/remoting/4.14-rc3000.5949ea_7370a_f/remoting-4.14-rc3000.5949ea_7370a_f.jar for hours ,  the problem had not diminished.   I found that the problem is only at git checkout, so try to add sleep before git checkout
            matthewrgomes Matthew Gomes added a comment -

            Issue is only reproducible on Agent/ECS jobs that use Git plug-in https://plugins.jenkins.io/git/

             

            matthewrgomes Matthew Gomes added a comment - Issue is only reproducible on Agent/ECS jobs that use Git plug-in https://plugins.jenkins.io/git/  

            People

              vlatombe Vincent Latombe
              wasimj Wasim
              Votes:
              9 Vote for this issue
              Watchers:
              25 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: