Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-65873

java.lang.OutOfMemoryError: unable to create new native thread

    • 2.362

      We regularly see issues with the jenkins/inbound-agent in our Jenkins logs on Kubernetes. It seems to occur in around 1% of all jobs.

      The error message is below.

      Whilst the error message refers to java.lang.OutOfMemoryError: and unable to create new native thread we have checked the pods and nodes in the cluster and there is always sufficient memory or threads available at the time of the error.

      The specific versions for this specific error message are:

      jenkins/inbound-agent:4.3-4

      Jenkins 2.263.4

      However we have also seen this error occur with different versions of both the inbound-agent and Jenkins.

      Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138
      	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
      	at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
      	at hudson.remoting.Channel.call(Channel.java:1001)
      	at hudson.FilePath.act(FilePath.java:1157)
      	at hudson.FilePath.act(FilePath.java:1146)
      	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121)
      	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904)
      	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      java.lang.OutOfMemoryError: unable to create new native thread
      	at java.lang.Thread.start0(Native Method)
      	at java.lang.Thread.start(Thread.java:717)
      	at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
      	at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1603)
      	at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:334)
      	at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
      	at jenkins.util.InterceptingScheduledExecutorService.schedule(InterceptingScheduledExecutorService.java:49)
      	at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.reschedule(DelayBufferedOutputStream.java:72)
      	at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.<init>(DelayBufferedOutputStream.java:68)
      	at org.jenkinsci.plugins.workflow.log.BufferedBuildListener$Replacement.readResolve(BufferedBuildListener.java:77)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at java.io.ObjectStreamClass.invokeReadResolve(ObjectStreamClass.java:1260)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2133)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
      	at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2342)
      	at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2266)
      	at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2124)
      	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1625)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:465)
      	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
      	at hudson.remoting.UserRequest.deserialize(UserRequest.java:290)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:189)
      	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
      	at hudson.remoting.Request$2.run(Request.java:369)
      	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
      Caused: java.io.IOException: Remote call on JNLP4-connect connection from ip-100-64-244-120.eu-west-1.compute.internal/100.64.244.120:39138 failed
      	at hudson.remoting.Channel.call(Channel.java:1007)
      	at hudson.FilePath.act(FilePath.java:1157)
      	at hudson.FilePath.act(FilePath.java:1146)
      	at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:121)
      	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:904)
      	at hudson.plugins.git.GitSCM.createClient(GitSCM.java:835)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1288)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
      	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      

          [JENKINS-65873] java.lang.OutOfMemoryError: unable to create new native thread

          rhinoceros.xn added a comment -

          matthewrgomes 

           

          before git OR checkout step:

          // code placeholder
              node(label) {
                  stage('xxzzxasd') {
                      container('xxxx') {
                                  stage('git clone') {
          
                                      sleep(10) //** HERE: adding sleep(10) before git checkout **
          
                                      git branch: 'master', credentialsId: '', url: 'git@xxx.com:xxx/xxxxxxx.git'
           

          rhinoceros.xn added a comment - matthewrgomes     before git OR checkout step: // code placeholder node(label) { stage( 'xxzzxasd' ) { container( 'xxxx' ) { stage( 'git clone' ) { sleep(10) //** HERE: adding sleep(10) before git checkout ** git branch: 'master' , credentialsId: '', url: ' git@xxx.com:xxx/xxxxxxx.git'

          Donald Gobin added a comment -

          rhinoceros so if this is true, then it indicates that the issue is with some kind of race condition within the jenkins pipeline flow that is causing this to happen. your sleep essentially looks like waiting for all jenkins threads for that pipeline to all complete before continuing ?

          Donald Gobin added a comment - rhinoceros so if this is true, then it indicates that the issue is with some kind of race condition within the jenkins pipeline flow that is causing this to happen. your sleep essentially looks like waiting for all jenkins threads for that pipeline to all complete before continuing ?

          rhinoceros.xn added a comment -

          dg424 yes. I think so.

           

          I had updated agent.jar two weeks ago with https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/remoting/4.14-rc3000.5949ea_7370a_f/remoting-4.14-rc3000.5949ea_7370a_f.jar for hours ,  the problem had not diminished.

           

          I found that the problem is only at git checkout, so try to add sleep before git checkout

          rhinoceros.xn added a comment - dg424 yes. I think so.   I had updated agent.jar two weeks ago with https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/main/remoting/4.14-rc3000.5949ea_7370a_f/remoting-4.14-rc3000.5949ea_7370a_f.jar for hours ,  the problem had not diminished.   I found that the problem is only at git checkout, so try to add sleep before git checkout

          Matthew Gomes added a comment -

          Issue is only reproducible on Agent/ECS jobs that use Git plug-in https://plugins.jenkins.io/git/

           

          Matthew Gomes added a comment - Issue is only reproducible on Agent/ECS jobs that use Git plug-in https://plugins.jenkins.io/git/  

          Kevin added a comment - - edited

          I upgraded to 2.363-jdk11 w/ agents on 4.10-3-jdk11 and still getting the same error sporadically:

          Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.32.11.76/10.32.11.76:34066
          at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
          at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
          at hudson.remoting.Channel.call(Channel.java:1000)
          at hudson.FilePath.act(FilePath.java:1186)
          at hudson.FilePath.act(FilePath.java:1175)
          at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:140)
          at hudson.plugins.git.GitSCM.createClient(GitSCM.java:916)
          at hudson.plugins.git.GitSCM.createClient(GitSCM.java:847)
          at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1297)
          at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
          at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
          at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
          at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:829)
          java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
          at java.base/java.lang.Thread.start0(Native Method)
          at java.base/java.lang.Thread.start(Unknown Source)
          at hudson.remoting.AtmostOneThreadExecutor.execute(AtmostOneThreadExecutor.java:104)
          at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source)
          at org.jenkinsci.remoting.util.ExecutorServiceUtils.submitAsync(ExecutorServiceUtils.java:58)
          at hudson.remoting.JarCacheSupport.resolve(JarCacheSupport.java:66)
          at hudson.remoting.ResourceImageInJar._resolveJarURL(ResourceImageInJar.java:93)
          at hudson.remoting.ResourceImageInJar.resolve(ResourceImageInJar.java:45)
          at hudson.remoting.RemoteClassLoader.loadRemoteClass(RemoteClassLoader.java:284)
          at hudson.remoting.RemoteClassLoader.loadWithMultiClassLoader(RemoteClassLoader.java:264)
          at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:223)
          at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
          at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
          at jenkins.util.Timer.get(Timer.java:47)
          at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.reschedule(DelayBufferedOutputStream.java:74)
          at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.<init>(DelayBufferedOutputStream.java:70)
          at org.jenkinsci.plugins.workflow.log.BufferedBuildListener$Replacement.readResolve(BufferedBuildListener.java:79)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
          at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
          at java.base/java.lang.reflect.Method.invoke(Unknown Source)
          at java.base/java.io.ObjectStreamClass.invokeReadResolve(Unknown Source)
          at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
          at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
          at java.base/java.io.ObjectInputStream.defaultReadFields(Unknown Source)
          at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source)
          at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
          at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
          at java.base/java.io.ObjectInputStream.defaultReadFields(Unknown Source)
          at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source)
          at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
          at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
          at java.base/java.io.ObjectInputStream.defaultReadFields(Unknown Source)
          at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source)
          at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
          at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
          at java.base/java.io.ObjectInputStream.defaultReadFields(Unknown Source)
          at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source)
          at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source)
          at java.base/java.io.ObjectInputStream.readObject0(Unknown Source)
          at java.base/java.io.ObjectInputStream.readObject(Unknown Source)
          at java.base/java.io.ObjectInputStream.readObject(Unknown Source)
          at hudson.remoting.UserRequest.deserialize(UserRequest.java:289)
          at hudson.remoting.UserRequest.perform(UserRequest.java:189)
          at hudson.remoting.UserRequest.perform(UserRequest.java:54)
          at hudson.remoting.Request$2.run(Request.java:376)
          at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
          at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
          at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:122)
          at java.base/java.lang.Thread.run(Unknown Source)
          Caused: java.io.IOException: Remote call on JNLP4-connect connection from 10.32.11.76/10.32.11.76:34066 failed
          at hudson.remoting.Channel.call(Channel.java:1004)
          at hudson.FilePath.act(FilePath.java:1186)
          at hudson.FilePath.act(FilePath.java:1175)
          at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:140)
          at hudson.plugins.git.GitSCM.createClient(GitSCM.java:916)
          at hudson.plugins.git.GitSCM.createClient(GitSCM.java:847)
          at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1297)
          at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
          at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
          at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
          at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:829)
          Finished: FAILURE
          

          Kevin added a comment - - edited I upgraded to 2.363-jdk11 w/ agents on 4.10-3-jdk11 and still getting the same error sporadically: Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 10.32.11.76/10.32.11.76:34066 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784) at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356) at hudson.remoting.Channel.call(Channel.java:1000) at hudson.FilePath.act(FilePath.java:1186) at hudson.FilePath.act(FilePath.java:1175) at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:140) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:916) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:847) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1297) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang. Thread .run( Thread .java:829) java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached at java.base/java.lang. Thread .start0(Native Method) at java.base/java.lang. Thread .start(Unknown Source) at hudson.remoting.AtmostOneThreadExecutor.execute(AtmostOneThreadExecutor.java:104) at java.base/java.util.concurrent.AbstractExecutorService.submit(Unknown Source) at org.jenkinsci.remoting.util.ExecutorServiceUtils.submitAsync(ExecutorServiceUtils.java:58) at hudson.remoting.JarCacheSupport.resolve(JarCacheSupport.java:66) at hudson.remoting.ResourceImageInJar._resolveJarURL(ResourceImageInJar.java:93) at hudson.remoting.ResourceImageInJar.resolve(ResourceImageInJar.java:45) at hudson.remoting.RemoteClassLoader.loadRemoteClass(RemoteClassLoader.java:284) at hudson.remoting.RemoteClassLoader.loadWithMultiClassLoader(RemoteClassLoader.java:264) at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:223) at java.base/java.lang. ClassLoader .loadClass(Unknown Source) at java.base/java.lang. ClassLoader .loadClass(Unknown Source) at jenkins.util.Timer.get(Timer.java:47) at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.reschedule(DelayBufferedOutputStream.java:74) at org.jenkinsci.plugins.workflow.log.DelayBufferedOutputStream.<init>(DelayBufferedOutputStream.java:70) at org.jenkinsci.plugins.workflow.log.BufferedBuildListener$Replacement.readResolve(BufferedBuildListener.java:79) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source) at java.base/java.io.ObjectStreamClass.invokeReadResolve(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.defaultReadFields(Unknown Source) at java.base/java.io.ObjectInputStream.readSerialData(Unknown Source) at java.base/java.io.ObjectInputStream.readOrdinaryObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject0(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at java.base/java.io.ObjectInputStream.readObject(Unknown Source) at hudson.remoting.UserRequest.deserialize(UserRequest.java:289) at hudson.remoting.UserRequest.perform(UserRequest.java:189) at hudson.remoting.UserRequest.perform(UserRequest.java:54) at hudson.remoting.Request$2.run(Request.java:376) at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78) at java.base/java.util.concurrent.FutureTask.run(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:122) at java.base/java.lang. Thread .run(Unknown Source) Caused: java.io.IOException: Remote call on JNLP4-connect connection from 10.32.11.76/10.32.11.76:34066 failed at hudson.remoting.Channel.call(Channel.java:1004) at hudson.FilePath.act(FilePath.java:1186) at hudson.FilePath.act(FilePath.java:1175) at org.jenkinsci.plugins.gitclient.Git.getClient(Git.java:140) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:916) at hudson.plugins.git.GitSCM.createClient(GitSCM.java:847) at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1297) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97) at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84) at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang. Thread .run( Thread .java:829) Finished: FAILURE

          Basil Crow added a comment -

          I upgraded to […] 4.10-3-jdk11

          kryan90 You aren't running with the fix. The fix is in Remoting 4.13.3 (LTS line) and Remoting 3044.vb_940a_a_e4f72e (weekly line). Java 11.0.16 is also recommended.

          Basil Crow added a comment - I upgraded to […] 4.10-3-jdk11 kryan90 You aren't running with the fix. The fix is in Remoting 4.13.3 (LTS line) and Remoting 3044.vb_940a_a_e4f72e (weekly line). Java 11.0.16 is also recommended.

          Donald Gobin added a comment -

          basil Definitely interested in this fix. We run the Docker containers for the controller and agent side, can you share which tags of these include this fix ? For instance, I don't see a 4.13 release for the agent side here - https://hub.docker.com/r/jenkins/inbound-agent/tags?page=1&name=jdk11. Is an update required on both the controller and agent side ? As Kevin tried above assuming that the fix is on the controller side only and used the latest available tagged image for the agent - 4.10. So, just need some clarity on which Docker tags we need to use here. Thanks.

          Donald Gobin added a comment - basil Definitely interested in this fix. We run the Docker containers for the controller and agent side, can you share which tags of these include this fix ? For instance, I don't see a 4.13 release for the agent side here - https://hub.docker.com/r/jenkins/inbound-agent/tags?page=1&name=jdk11 . Is an update required on both the controller and agent side ? As Kevin tried above assuming that the fix is on the controller side only and used the latest available tagged image for the agent - 4.10. So, just need some clarity on which Docker tags we need to use here. Thanks.

          Basil Crow added a comment -

          I don't know anything about how the Docker images for agents are built or what versions of Remoting they include in them.

          Basil Crow added a comment - I don't know anything about how the Docker images for agents are built or what versions of Remoting they include in them.

          Donald Gobin added a comment - - edited

          basil Another question - do both the controller and agent have to have this fix/version ? Also, which component do I raise a ticket for to address the inbound-agent Docker image side of this ?

          Donald Gobin added a comment - - edited basil Another question - do both the controller and agent have to have this fix/version ? Also, which component do I raise a ticket for to address the inbound-agent Docker image side of this ?

          Basil Crow added a comment -

          The fix is in remoting.jar which is the main JAR for running the agent process. That having been said, that JAR file gets shipped over from the controller in some scenarios (e.g. SSH Build Agents, where the controller uses SSH to connect to the agent, ship over its copy of remoting.jar, and then start the agent process), so in some sense the controller version is relevant in the sense that it does bundle remoting.jar to be used in certain agent scenarios. That having been said I think your Docker image use case bundles its own copy of remoting.jar completely separate from the controller's version. I believe the maintainers of the Docker images use GitHub issues on the corresponding GitHub repositories to track issues.

          Basil Crow added a comment - The fix is in remoting.jar which is the main JAR for running the agent process. That having been said, that JAR file gets shipped over from the controller in some scenarios (e.g. SSH Build Agents, where the controller uses SSH to connect to the agent, ship over its copy of remoting.jar , and then start the agent process), so in some sense the controller version is relevant in the sense that it does bundle remoting.jar to be used in certain agent scenarios. That having been said I think your Docker image use case bundles its own copy of remoting.jar completely separate from the controller's version. I believe the maintainers of the Docker images use GitHub issues on the corresponding GitHub repositories to track issues.

            jglick Jesse Glick
            wasimj Wasim
            Votes:
            9 Vote for this issue
            Watchers:
            30 Start watching this issue

              Created:
              Updated:
              Resolved: