Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65873

java.lang.OutOfMemoryError: unable to create new native thread

    • 2.362

      We regularly see issues with the jenkins/inbound-agent in our Jenkins logs on Kubernetes. It seems to occur in around 1% of all jobs.

      The error message is below.

      Whilst the error message refers to java.lang.OutOfMemoryError: and unable to create new native thread we have checked the pods and nodes in the cluster and there is always sufficient memory or threads available at the time of the error.

      The specific versions for this specific error message are:

      jenkins/inbound-agent:4.3-4

      Jenkins 2.263.4

      However we have also seen this error occur with different versions of both the inbound-agent and Jenkins.

      Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138
      	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
      	at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
      	at hudson.remoting.Channel.call(Channel.java:1001)
      	at hudson.FilePath.act(FilePath.java:1157)
      	at hudson.FilePath.act(FilePath.java:1146)
      	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121)
      	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904)
      	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      java.lang.OutOfMemoryError: unable to create new native thread
      	at java.lang.Thread.start0(Native Method)
      	at java.lang.Thread.start(Thread.java:717)
      	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
      	at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603)
      	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
      	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
      	at jenkins.util.InterceptingScheduledExecutorService.schedule(InterceptingScheduledExecutorService.java:49)
      	at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.reschedule(DelayBufferedOutputStream.java:72)
      	at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.<init>(DelayBufferedOutputStream.java:68)
      	at org.jenkinsci.plugins.workflow.log.BufferedBuildListener$Replacement.readResolve(BufferedBuildListener.java:77)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1260)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2133)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:290)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:189)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
      Caused: java.io.IOException: Remote call on JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138 failed
      	at hudson.remoting.Channel.call(Channel.java:1007)
      	at hudson.FilePath.act(FilePath.java:1157)
      	at hudson.FilePath.act(FilePath.java:1146)
      	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121)
      	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904)
      	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

          [JENKINS-65873] java.lang.OutOfMemoryError: unable to create new native thread

          Wasim created issue -

          Basil Crow added a comment -

          FWIW I've seen the same error sporadically since 2019, but with Swarm on the client side. It seems to occur once every few months in my company, so less than 1% of all builds (we do thousands of builds a month). In case it's helpful I've attached the stack trace from our internal bug tracker. Needless to say this is not reproducible. I do wonder if setting -Xmx or -Xms might help but I've never tried. I also wonder if this might be helped by JENKINS-61103.

          ERROR: Execution failed
          Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 1.2.3.4/1.2.3.4:34136
          		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
          		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
          		at hudson.remoting.Channel.call(Channel.java:957)
          		at hudson.FilePath.act(FilePath.java:1072)
          		at hudson.FilePath.act(FilePath.java:1061)
          		at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144)
          		at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822)
          		at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813)
          		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
          		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
          		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
          		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
          		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
          		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          java.lang.OutOfMemoryError: unable to create new native thread
          	at java.lang.Thread.start0(Native Method)
          	at java.lang.Thread.start(Thread.java:717)
          	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
          	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
          	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
          	at hudson.remoting.DelegatingExecutorService.submit(DelegatingExecutorService.java:42)
          	at hudson.remoting.InterceptingExecutorService.submit(InterceptingExecutorService.java:46)
          	at hudson.remoting.InterceptingExecutorService.submit(InterceptingExecutorService.java:41)
          	at org.jenkinsci.remoting.util.AnonymousClassWarnings.check(AnonymousClassWarnings.java:65)
          	at org.jenkinsci.remoting.util.AnonymousClassWarnings$1.annotateClass(AnonymousClassWarnings.java:121)
          	at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1290)
          	at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231)
          	at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1294)
          	at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231)
          	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1427)
          	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
          	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
          	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
          	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
          	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
          	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
          	at hudson.remoting.Command.writeTo(Command.java:109)
          	at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:287)
          	at hudson.remoting.Channel.send(Channel.java:723)
          	at hudson.remoting.Request.callAsync(Request.java:238)
          	at hudson.remoting.Channel.callAsync(Channel.java:987)
          	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:282)
          	at com.sun.proxy.$Proxy3.notifyJarPresence(Unknown Source)
          	at hudson.remoting.FileSystemJarCache.lookInCache(FileSystemJarCache.java:80)
          	at hudson.remoting.JarCacheSupport.resolve(JarCacheSupport.java:46)
          	at hudson.remoting.ResourceImageInJar._resolveJarURL(ResourceImageInJar.java:90)
          	at hudson.remoting.ResourceImageInJar.resolve(ResourceImageInJar.java:43)
          	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:304)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          	at java.lang.Class.getDeclaredFields0(Native Method)
          	at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
          	at java.lang.Class.getDeclaredField(Class.java:2068)
          	at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1857)
          	at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:79)
          	at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:506)
          	at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:494)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:494)
          	at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:391)
          	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:681)
          	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
          	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
          	at java.util.ArrayList.readObject(ArrayList.java:797)
          	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
          	at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:190)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
          	at hudson.remoting.Request$2.run(Request.java:369)
          	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
          Caused: java.io.IOException: Remote call on JNLP4-connect connection from 1.2.3.4/1.2.3.4:34136 failed
          	at hudson.remoting.Channel.call(Channel.java:963)
          	at hudson.FilePath.act(FilePath.java:1072)
          	at hudson.FilePath.act(FilePath.java:1061)
          	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144)
          	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822)
          	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813)
          	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
          	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          

          Basil Crow added a comment - FWIW I've seen the same error sporadically since 2019, but with Swarm on the client side. It seems to occur once every few months in my company, so less than 1% of all builds (we do thousands of builds a month). In case it's helpful I've attached the stack trace from our internal bug tracker. Needless to say this is not reproducible. I do wonder if setting -Xmx or -Xms might help but I've never tried. I also wonder if this might be helped by JENKINS-61103 . ERROR: Execution failed Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 1.2.3.4/1.2.3.4:34136 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:957) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at hudson.remoting.DelegatingExecutorService.submit(DelegatingExecutorService.java:42) at hudson.remoting.InterceptingExecutorService.submit(InterceptingExecutorService.java:46) at hudson.remoting.InterceptingExecutorService.submit(InterceptingExecutorService.java:41) at org.jenkinsci.remoting.util.AnonymousClassWarnings.check(AnonymousClassWarnings.java:65) at org.jenkinsci.remoting.util.AnonymousClassWarnings$1.annotateClass(AnonymousClassWarnings.java:121) at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1290) at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231) at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1294) at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1427) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at hudson.remoting.Command.writeTo(Command.java:109) at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:287) at hudson.remoting.Channel.send(Channel.java:723) at hudson.remoting.Request.callAsync(Request.java:238) at hudson.remoting.Channel.callAsync(Channel.java:987) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:282) at com.sun.proxy.$Proxy3.notifyJarPresence(Unknown Source) at hudson.remoting.FileSystemJarCache.lookInCache(FileSystemJarCache.java:80) at hudson.remoting.JarCacheSupport.resolve(JarCacheSupport.java:46) at hudson.remoting.ResourceImageInJar._resolveJarURL(ResourceImageInJar.java:90) at hudson.remoting.ResourceImageInJar.resolve(ResourceImageInJar.java:43) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:304) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2583) at java.lang.Class.getDeclaredField(Class.java:2068) at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1857) at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:79) at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:506) at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:494) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:494) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:391) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:681) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at java.util.ArrayList.readObject(ArrayList.java:797) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at hudson.remoting.UserRequest.deserialize(UserRequest.java:291) at hudson.remoting.UserRequest.perform(UserRequest.java:190) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) Caused: java.io.IOException: Remote call on JNLP4-connect connection from 1.2.3.4/1.2.3.4:34136 failed at hudson.remoting.Channel.call(Channel.java:963) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

          Kevin added a comment -

          We have been seeing this as well at about the same rates. Makes it extremely difficult to test fixes since only a very small portion of jobs fail. It appears to always happen during the checkout scm stage.

          Here is generally how our pod templates / jobs are built using a shared library

          public void myTemplate(body) {
              def name = "my-node"
              def label = "${name}-${uuid}"
              podTemplate(
                  cloud: 'my-cloud',
                  label: label,
                  name: name,
                  containers: [
                      containerTemplate(
                          name: name,
                          image: "myimage",
                          command: 'cat',
                          ttyEnabled: true,
                          workingDir: '/home/jenkins/agent',
                          alwaysPullImage: false,
                          resourceRequestCpu: '500m',
                          resourceRequestMemory: '512Mi',
                      )
                  ],
                  workspaceVolume: emptyDirWorkspaceVolume(false),
              ) {
                  node(label) {
                      container(name) {
                          body.call()
                      }
                  }
              }
          }
          

          and calling it like this:

          import org.myorg.PodTemplates
          
          agentTemplates = new PodTemplates()
          agentTemplates.myTemplate{
              checkout scm
              stage('Do some stuff') {
                sh 'echo test'
              }
          }
          

          In this case the checkout scm command is being run in the context of the container block. One thing I plan on trying is to move that stage to run at the node level vs the container level.

          Kevin added a comment - We have been seeing this as well at about the same rates. Makes it extremely difficult to test fixes since only a very small portion of jobs fail. It appears to always happen during the checkout scm stage. Here is generally how our pod templates / jobs are built using a shared library public void myTemplate(body) { def name = "my-node" def label = "${name}-${uuid}" podTemplate( cloud: 'my-cloud' , label: label, name: name, containers: [ containerTemplate( name: name, image: "myimage" , command: 'cat' , ttyEnabled: true , workingDir: '/home/jenkins/agent' , alwaysPullImage: false , resourceRequestCpu: '500m' , resourceRequestMemory: '512Mi' , ) ], workspaceVolume: emptyDirWorkspaceVolume( false ), ) { node(label) { container(name) { body.call() } } } } and calling it like this: import org.myorg.PodTemplates agentTemplates = new PodTemplates() agentTemplates.myTemplate{ checkout scm stage( 'Do some stuff' ) { sh 'echo test' } } In this case the checkout scm command is being run in the context of the container block. One thing I plan on trying is to move that stage to run at the node level vs the container level.

          Jeff Thompson added a comment -

          I have no ideas or suggestions to help anyone out here. We would need more reproducibility or troubleshooting info to do anything.

          I will note that sometimes thinks like this are because of the behavior of some plugin or interactions between multiple ones.

          Jeff Thompson added a comment - I have no ideas or suggestions to help anyone out here. We would need more reproducibility or troubleshooting info to do anything. I will note that sometimes thinks like this are because of the behavior of some plugin or interactions between multiple ones.

          Simon added a comment -

          Updated to Jenkins to LTS 2.289.1 which includes:

          We have also updated the version of the inboud-agent image to 4.7, and are no longer seeing this issue.

           

          Simon added a comment - Updated to Jenkins to LTS 2.289.1 which includes: Upgrade from Remoting 4.6 to Remoting 4.7 with bugfixes and dependency updates. ( pull 5292 ,  Remoting 4.7 changelog ) We have also updated the version of the inboud-agent image to 4.7, and are no longer seeing this issue.  

          John added a comment -

          Simon - Good to know that latest version resolving issue.
          I have been seeing similar issue for long time. Can you please share following parameters?

          • OS ulimit (any other tuning parameters to consider)
          • Jenkins JVM parameters
          • Jenkins slave (any new changes)

          John added a comment - Simon - Good to know that latest version resolving issue. I have been seeing similar issue for long time. Can you please share following parameters? OS ulimit (any other tuning parameters to consider) Jenkins JVM parameters Jenkins slave (any new changes)

          Simon added a comment -
          • OS ulimit - we are running the Docker image from Docker Hub in AWS EKS service, and have not made any changes from the defaults provided by those services.
          • JVM parameters - -XX:MaxRAMPercentage=50.0
          • The only changes to the slave in K8s has been the adoption of the new inbound-agent version 4.7-1

          Simon added a comment - OS ulimit - we are running the Docker image from Docker Hub in AWS EKS service, and have not made any changes from the defaults provided by those services. JVM parameters - -XX:MaxRAMPercentage=50.0 The only changes to the slave in K8s has been the adoption of the new inbound-agent version 4.7-1

          Wasim added a comment -

          This error was seen again today with 
          inbound-agent:4.7-1

          Wasim added a comment - This error was seen again today with  inbound-agent:4.7-1

          Kevin added a comment -

          I have tested up through inbound-agent:4.9-1 w/ controller versions 2.289.2 and 2.304 – the issue is still happening at the same rate.

          I created a job on a dev controller deployment that only checks out scm. I have it set to run once per minute and usually get ~1 failure per hour.

          Kevin added a comment - I have tested up through inbound-agent:4.9-1 w/ controller versions 2.289.2 and 2.304 – the issue is still happening at the same rate. I created a job on a dev controller deployment that only checks out scm. I have it set to run once per minute and usually get ~1 failure per hour.
          Christopher made changes -
          Attachment New: jstack_sample.txt [ 56295 ]

            jglick Jesse Glick
            wasimj Wasim
            Votes:
            9 Vote for this issue
            Watchers:
            30 Start watching this issue

              Created:
              Updated:
              Resolved: