Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65873

java.lang.OutOfMemoryError: unable to create new native thread

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      We regularly see issues with the jenkins/inbound-agent in our Jenkins logs on Kubernetes. It seems to occur in around 1% of all jobs. 

      The error message is below.

      Whilst the error message refers to java.lang.OutOfMemoryError: and unable to create new native thread we have checked the pods and nodes in the cluster and there is always sufficient memory or threads available at the time of the error.

       

      The specific versions for this specific error message are:

      jenkins/inbound-agent:4.3-4

      Jenkins 2.263.4

      However we have also seen this error occur with different versions of both the inbound-agent and Jenkins.

       

      Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138
      {{ at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)}}
      {{ at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)}}
      {{ at hudson.remoting.Channel.call(Channel.java:1001)}}
      {{ at hudson.FilePath.act(FilePath.java:1157)}}
      {{ at hudson.FilePath.act(FilePath.java:1146)}}
      {{ at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121)}}
      {{ at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904)}}
      {{ at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835)}}
      {{ at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288)}}
      {{ at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)}}
      {{ at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)}}
      {{ at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)}}
      {{ at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)}}
      {{ at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}
      {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
      {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
      java.lang.OutOfMemoryError: unable to create new native thread
      {{ at java.lang.Thread.start0(Native Method)}}
      {{ at java.lang.Thread.start(Thread.java:717)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603)}}
      {{ at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)}}
      {{ at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)}}
      {{ at jenkins.util.InterceptingScheduledExecutorService.schedule(InterceptingScheduledExecutorService.java:49)}}
      {{ at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.reschedule(DelayBufferedOutputStream.java:72)}}
      {{ at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.<init>(DelayBufferedOutputStream.java:68)}}
      {{ at org.jenkinsci.plugins.workflow.log.BufferedBuildListener$Replacement.readResolve(BufferedBuildListener.java:77)}}
      {{ at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)}}
      {{ at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)}}
      {{ at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)}}
      {{ at java.lang.reflect.Method.invoke(Method.java:498)}}
      {{ at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1260)}}
      {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2133)}}
      {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)}}
      {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)}}
      {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)}}
      {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)}}
      {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)}}
      {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)}}
      {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)}}
      {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)}}
      {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)}}
      {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)}}
      {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)}}
      {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)}}
      {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)}}
      {{ at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)}}
      {{ at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)}}
      {{ at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)}}
      {{ at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)}}
      {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)}}
      {{ at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)}}
      {{ at hudson.remoting.UserRequest.deserialize(UserRequest.java:290)}}
      {{ at hudson.remoting.UserRequest.perform(UserRequest.java:189)}}
      {{ at hudson.remoting.UserRequest.perform(UserRequest.java:54)}}
      {{ at hudson.remoting.Request$2.run(Request.java:369)}}
      {{ at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)}}
      {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
      {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
      {{ at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)}}
      Caused: java.io.IOException: Remote call on JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138 failed
      {{ at hudson.remoting.Channel.call(Channel.java:1007)}}
      {{ at hudson.FilePath.act(FilePath.java:1157)}}
      {{ at hudson.FilePath.act(FilePath.java:1146)}}
      {{ at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121)}}
      {{ at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904)}}
      {{ at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835)}}
      {{ at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288)}}
      {{ at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)}}
      {{ at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)}}
      {{ at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)}}
      {{ at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)}}
      {{ at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)}}
      {{ at java.util.concurrent.FutureTask.run(FutureTask.java:266)}}
      {{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
      {{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
      {{ at java.lang.Thread.run(Thread.java:748)}}

       

        Attachments

          Activity

          Hide
          basil Basil Crow added a comment -

          FWIW I've seen the same error sporadically since 2019, but with Swarm on the client side. It seems to occur once every few months in my company, so less than 1% of all builds (we do thousands of builds a month). In case it's helpful I've attached the stack trace from our internal bug tracker. Needless to say this is not reproducible. I do wonder if setting -Xmx or -Xms might help but I've never tried. I also wonder if this might be helped by JENKINS-61103.

          ERROR: Execution failed
          Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 1.2.3.4/1.2.3.4:34136
          		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743)
          		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
          		at hudson.remoting.Channel.call(Channel.java:957)
          		at hudson.FilePath.act(FilePath.java:1072)
          		at hudson.FilePath.act(FilePath.java:1061)
          		at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144)
          		at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822)
          		at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813)
          		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
          		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
          		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
          		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
          		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
          		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          java.lang.OutOfMemoryError: unable to create new native thread
          	at java.lang.Thread.start0(Native Method)
          	at java.lang.Thread.start(Thread.java:717)
          	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
          	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
          	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
          	at hudson.remoting.DelegatingExecutorService.submit(DelegatingExecutorService.java:42)
          	at hudson.remoting.InterceptingExecutorService.submit(InterceptingExecutorService.java:46)
          	at hudson.remoting.InterceptingExecutorService.submit(InterceptingExecutorService.java:41)
          	at org.jenkinsci.remoting.util.AnonymousClassWarnings.check(AnonymousClassWarnings.java:65)
          	at org.jenkinsci.remoting.util.AnonymousClassWarnings$1.annotateClass(AnonymousClassWarnings.java:121)
          	at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1290)
          	at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231)
          	at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1294)
          	at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231)
          	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1427)
          	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
          	at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
          	at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
          	at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
          	at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
          	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
          	at hudson.remoting.Command.writeTo(Command.java:109)
          	at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:287)
          	at hudson.remoting.Channel.send(Channel.java:723)
          	at hudson.remoting.Request.callAsync(Request.java:238)
          	at hudson.remoting.Channel.callAsync(Channel.java:987)
          	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:282)
          	at com.sun.proxy.$Proxy3.notifyJarPresence(Unknown Source)
          	at hudson.remoting.FileSystemJarCache.lookInCache(FileSystemJarCache.java:80)
          	at hudson.remoting.JarCacheSupport.resolve(JarCacheSupport.java:46)
          	at hudson.remoting.ResourceImageInJar._resolveJarURL(ResourceImageInJar.java:90)
          	at hudson.remoting.ResourceImageInJar.resolve(ResourceImageInJar.java:43)
          	at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:304)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
          	at java.lang.Class.getDeclaredFields0(Native Method)
          	at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
          	at java.lang.Class.getDeclaredField(Class.java:2068)
          	at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1857)
          	at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:79)
          	at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:506)
          	at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:494)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:494)
          	at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:391)
          	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:681)
          	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
          	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
          	at java.util.ArrayList.readObject(ArrayList.java:797)
          	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:498)
          	at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
          	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
          	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
          	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
          	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
          	at hudson.remoting.UserRequest.deserialize(UserRequest.java:291)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:190)
          	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
          	at hudson.remoting.Request$2.run(Request.java:369)
          	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
          Caused: java.io.IOException: Remote call on JNLP4-connect connection from 1.2.3.4/1.2.3.4:34136 failed
          	at hudson.remoting.Channel.call(Channel.java:963)
          	at hudson.FilePath.act(FilePath.java:1072)
          	at hudson.FilePath.act(FilePath.java:1061)
          	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144)
          	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822)
          	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813)
          	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
          	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
          	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          	at java.lang.Thread.run(Thread.java:748)
          
          Show
          basil Basil Crow added a comment - FWIW I've seen the same error sporadically since 2019, but with Swarm on the client side. It seems to occur once every few months in my company, so less than 1% of all builds (we do thousands of builds a month). In case it's helpful I've attached the stack trace from our internal bug tracker. Needless to say this is not reproducible. I do wonder if setting -Xmx or -Xms might help but I've never tried. I also wonder if this might be helped by JENKINS-61103 . ERROR: Execution failed Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 1.2.3.4/1.2.3.4:34136 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1743) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357) at hudson.remoting.Channel.call(Channel.java:957) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) at hudson.remoting.DelegatingExecutorService.submit(DelegatingExecutorService.java:42) at hudson.remoting.InterceptingExecutorService.submit(InterceptingExecutorService.java:46) at hudson.remoting.InterceptingExecutorService.submit(InterceptingExecutorService.java:41) at org.jenkinsci.remoting.util.AnonymousClassWarnings.check(AnonymousClassWarnings.java:65) at org.jenkinsci.remoting.util.AnonymousClassWarnings$1.annotateClass(AnonymousClassWarnings.java:121) at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1290) at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231) at java.io.ObjectOutputStream.writeNonProxyDesc(ObjectOutputStream.java:1294) at java.io.ObjectOutputStream.writeClassDesc(ObjectOutputStream.java:1231) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1427) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348) at hudson.remoting.Command.writeTo(Command.java:109) at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:287) at hudson.remoting.Channel.send(Channel.java:723) at hudson.remoting.Request.callAsync(Request.java:238) at hudson.remoting.Channel.callAsync(Channel.java:987) at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:282) at com.sun.proxy.$Proxy3.notifyJarPresence(Unknown Source) at hudson.remoting.FileSystemJarCache.lookInCache(FileSystemJarCache.java:80) at hudson.remoting.JarCacheSupport.resolve(JarCacheSupport.java:46) at hudson.remoting.ResourceImageInJar._resolveJarURL(ResourceImageInJar.java:90) at hudson.remoting.ResourceImageInJar.resolve(ResourceImageInJar.java:43) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:304) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.getDeclaredFields0(Native Method) at java.lang.Class.privateGetDeclaredFields(Class.java:2583) at java.lang.Class.getDeclaredField(Class.java:2068) at java.io.ObjectStreamClass.getDeclaredSUID(ObjectStreamClass.java:1857) at java.io.ObjectStreamClass.access$700(ObjectStreamClass.java:79) at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:506) at java.io.ObjectStreamClass$3.run(ObjectStreamClass.java:494) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:494) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:391) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:681) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at java.util.ArrayList.readObject(ArrayList.java:797) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) at hudson.remoting.UserRequest.deserialize(UserRequest.java:291) at hudson.remoting.UserRequest.perform(UserRequest.java:190) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:369) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93) Caused: java.io.IOException: Remote call on JNLP4-connect connection from 1.2.3.4/1.2.3.4:34136 failed at hudson.remoting.Channel.call(Channel.java:963) at hudson.FilePath.act(FilePath.java:1072) at hudson.FilePath.act(FilePath.java:1061) at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:144) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:822) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:813) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:124) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
          Hide
          kryan90 Kevin added a comment -

          We have been seeing this as well at about the same rates. Makes it extremely difficult to test fixes since only a very small portion of jobs fail. It appears to always happen during the checkout scm stage.

          Here is generally how our pod templates / jobs are built using a shared library

          public void myTemplate(body) {
              def name = "my-node"
              def label = "${name}-${uuid}"
              podTemplate(
                  cloud: 'my-cloud',
                  label: label,
                  name: name,
                  containers: [
                      containerTemplate(
                          name: name,
                          image: "myimage",
                          command: 'cat',
                          ttyEnabled: true,
                          workingDir: '/home/jenkins/agent',
                          alwaysPullImage: false,
                          resourceRequestCpu: '500m',
                          resourceRequestMemory: '512Mi',
                      )
                  ],
                  workspaceVolume: emptyDirWorkspaceVolume(false),
              ) {
                  node(label) {
                      container(name) {
                          body.call()
                      }
                  }
              }
          }
          

          and calling it like this:

          import org.myorg.PodTemplates
          
          agentTemplates = new PodTemplates()
          agentTemplates.myTemplate{
              checkout scm
              stage('Do some stuff') {
                sh 'echo test'
              }
          }
          

          In this case the checkout scm command is being run in the context of the container block. One thing I plan on trying is to move that stage to run at the node level vs the container level.

          Show
          kryan90 Kevin added a comment - We have been seeing this as well at about the same rates. Makes it extremely difficult to test fixes since only a very small portion of jobs fail. It appears to always happen during the checkout scm stage. Here is generally how our pod templates / jobs are built using a shared library public void myTemplate(body) { def name = "my-node" def label = "${name}-${uuid}" podTemplate( cloud: 'my-cloud' , label: label, name: name, containers: [ containerTemplate( name: name, image: "myimage" , command: 'cat' , ttyEnabled: true , workingDir: '/home/jenkins/agent' , alwaysPullImage: false , resourceRequestCpu: '500m' , resourceRequestMemory: '512Mi' , ) ], workspaceVolume: emptyDirWorkspaceVolume( false ), ) { node(label) { container(name) { body.call() } } } } and calling it like this: import org.myorg.PodTemplates agentTemplates = new PodTemplates() agentTemplates.myTemplate{ checkout scm stage( 'Do some stuff' ) { sh 'echo test' } } In this case the checkout scm command is being run in the context of the container block. One thing I plan on trying is to move that stage to run at the node level vs the container level.
          Hide
          jthompson Jeff Thompson added a comment -

          I have no ideas or suggestions to help anyone out here. We would need more reproducibility or troubleshooting info to do anything.

          I will note that sometimes thinks like this are because of the behavior of some plugin or interactions between multiple ones.

          Show
          jthompson Jeff Thompson added a comment - I have no ideas or suggestions to help anyone out here. We would need more reproducibility or troubleshooting info to do anything. I will note that sometimes thinks like this are because of the behavior of some plugin or interactions between multiple ones.
          Hide
          simepo Simon added a comment -

          Updated to Jenkins to LTS 2.289.1 which includes:

          We have also updated the version of the inboud-agent image to 4.7, and are no longer seeing this issue.

           

          Show
          simepo Simon added a comment - Updated to Jenkins to LTS 2.289.1 which includes: Upgrade from Remoting 4.6 to Remoting 4.7 with bugfixes and dependency updates. ( pull 5292 ,  Remoting 4.7 changelog ) We have also updated the version of the inboud-agent image to 4.7, and are no longer seeing this issue.  
          Hide
          johnny25 John added a comment -

          Simon - Good to know that latest version resolving issue.
          I have been seeing similar issue for long time. Can you please share following parameters?

          • OS ulimit (any other tuning parameters to consider)
          • Jenkins JVM parameters
          • Jenkins slave (any new changes)
          Show
          johnny25 John added a comment - Simon - Good to know that latest version resolving issue. I have been seeing similar issue for long time. Can you please share following parameters? OS ulimit (any other tuning parameters to consider) Jenkins JVM parameters Jenkins slave (any new changes)

            People

            Assignee:
            jthompson Jeff Thompson
            Reporter:
            wasimj Wasim
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Dates

              Created:
              Updated: