Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-14307

Ping timeout to slave

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Incomplete
    • Icon: Major Major
    • ssh-plugin, ssh-slaves-plugin
    • None

      Unable to complete a build. We end up with Java crashes.

      at hudson.FilePath.act(FilePath.java:838)
      at hudson.FilePath.act(FilePath.java:824)
      at hudson.FilePath.delete(FilePath.java:1129)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
      at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
      at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)
      at hudson.model.Build$BuildExecution.build(Build.java:199)
      at hudson.model.Build$BuildExecution.doRun(Build.java:160)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
      at hudson.model.Run.execute(Run.java:1488)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:236)
      Caused by: hudson.remoting.ChannelClosedException: channel is already closed
      at hudson.remoting.Channel.send(Channel.java:492)
      at hudson.remoting.Request.call(Request.java:129)
      at hudson.remoting.Channel.call(Channel.java:663)
      at hudson.FilePath.act(FilePath.java:831)
      ... 13 more
      Caused by: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
      at hudson.remoting.Command.readFrom(Command.java:90)
      at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)
      FATAL: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
      hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Request.call(Request.java:174)
      at hudson.remoting.Channel.call(Channel.java:663)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
      at $Proxy37.join(Unknown Source)
      at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:861)
      at hudson.Launcher$ProcStarter.join(Launcher.java:345)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
      at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
      at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:717)
      at hudson.model.Build$BuildExecution.build(Build.java:199)
      at hudson.model.Build$BuildExecution.doRun(Build.java:160)
      at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
      at hudson.model.Run.execute(Run.java:1488)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:236)
      Caused by: hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.Request.abort(Request.java:299)
      at hudson.remoting.Channel.terminate(Channel.java:719)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:69)
      Caused by: java.io.IOException: Unexpected termination of the channel
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:50)
      Caused by: java.io.EOFException
      at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2570)
      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1314)
      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:368)
      at hudson.remoting.Command.readFrom(Command.java:90)
      at hudson.remoting.ClassicCommandTransport.read(ClassicCommandTransport.java:59)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:48)

      We have also seen other crashes on this configuration...

      Design Caused by: hudson.remoting.Channel$OrderlyShutdown: java.util.concurrent.TimeoutException: Ping started on 1341254668579 hasn't completed at 1341254908585
      at hudson.remoting.Channel$CloseCommand.execute(Channel.java:818)
      at hudson.remoting.Channel$1.handle(Channel.java:416)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
      Caused by: Command close created at
      at hudson.remoting.Command.<init>(Command.java:54)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:812)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:810)
      at hudson.remoting.Channel.close(Channel.java:877)
      at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:110)
      at hudson.remoting.PingThread.ping(PingThread.java:114)
      at hudson.remoting.PingThread.run(PingThread.java:81)
      Caused by: java.util.concurrent.TimeoutException: Ping started on 1341254668579 hasn't completed at 1341254908585
      ... 2 more
      Caused by: java.util.concurrent.TimeoutException
      at hudson.remoting.Request$1.get(Request.java:249)
      at hudson.remoting.Request$1.get(Request.java:184)
      at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
      at hudson.remoting.PingThread.ping(PingThread.java:107)
      ... 1 more
      FATAL: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown: java.util.concurrent.TimeoutException: Ping started on 1341254668579 hasn't completed at 1341254908585
      hudson.remoting.RequestAbortedException: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown: java.util.concurrent.TimeoutException: Ping started on 1341254668579 hasn't completed at 1341254908585
      at hudson.remoting.Request.call(Request.java:149)
      at hudson.remoting.Channel.call(Channel.java:646)
      at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:158)
      at $Proxy33.join(Unknown Source)
      at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:861)
      at hudson.Launcher$ProcStarter.join(Launcher.java:345)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:82)
      at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:58)
      at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19)
      at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:705)
      at hudson.model.Build$RunnerImpl.build(Build.java:178)
      at hudson.model.Build$RunnerImpl.doRun(Build.java:139)
      at hudson.model.AbstractBuild$AbstractRunner.run(AbstractBuild.java:475)
      at hudson.model.Run.run(Run.java:1434)
      at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46)
      at hudson.model.ResourceController.execute(ResourceController.java:88)
      at hudson.model.Executor.run(Executor.java:239)
      Caused by: hudson.remoting.RequestAbortedException: hudson.remoting.Channel$OrderlyShutdown: java.util.concurrent.TimeoutException: Ping started on 1341254668579 hasn't completed at 1341254908585
      at hudson.remoting.Request.abort(Request.java:273)
      at hudson.remoting.Channel.terminate(Channel.java:702)
      at hudson.remoting.Channel$CloseCommand.execute(Channel.java:818)
      at hudson.remoting.Channel$1.handle(Channel.java:416)
      at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:60)
      Caused by: hudson.remoting.Channel$OrderlyShutdown: java.util.concurrent.TimeoutException: Ping started on 1341254668579 hasn't completed at 1341254908585
      ... 3 more
      Caused by: Command close created at
      at hudson.remoting.Command.<init>(Command.java:54)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:812)
      at hudson.remoting.Channel$CloseCommand.<init>(Channel.java:810)
      at hudson.remoting.Channel.close(Channel.java:877)
      at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:110)
      at hudson.remoting.PingThread.ping(PingThread.java:114)
      at hudson.remoting.PingThread.run(PingThread.java:81)
      Caused by: java.util.concurrent.TimeoutException: Ping started on 1341254668579 hasn't completed at 1341254908585
      ... 2 more
      Caused by: java.util.concurrent.TimeoutException
      at hudson.remoting.Request$1.get(Request.java:249)
      at hudson.remoting.Request$1.get(Request.java:184)
      at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59)
      at hudson.remoting.PingThread.ping(PingThread.java:107)
      ... 1 more

      is this a setup issue or a genuine crash? If you need any more info please ask!

      Thanks

      Christian

          [JENKINS-14307] Ping timeout to slave

          Kohsuke Kawaguchi added a comment -

          The ping thread had killed the connection because it didn't hear back for 4 minutes.

          This kind of problems normally only happen when the network route to the slave is failing to route packets without reporting a failure. Are your master and slave far apart? Do you have interesting network setup in between?

          Kohsuke Kawaguchi added a comment - The ping thread had killed the connection because it didn't hear back for 4 minutes. This kind of problems normally only happen when the network route to the slave is failing to route packets without reporting a failure. Are your master and slave far apart? Do you have interesting network setup in between?

          Kohsuke Kawaguchi added a comment -

          As I said, a ping error indicates that the master didn't hear back a ping response from the slave after 4 minutes. Aside from a network problem, there's another possibility I failed to consider in my previous comment — that the TCP/SSH connection to the slave is intact, but the slave JVM is choking and being unable to respond.

          For example, if a process is paused (such as "kill -STOP"), you could see this. Or if the slave has a massive heap and it's taking more than 4 minutes for GC, you could see this. Or theoretically, if the heap is so saturated that VM is doing one full GC after another, you can see this, too.

          See https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError and try to diagnose the slave heap problem. -verbose:gc or other means to monitor the GC performance could be useful, too.

          ERROR: Failed to evaluate groovy script. 
hudson.remoting.ChannelClosedException: channel is already closed 
at hudson.remoting.Channel.send(Channel.java:475) 
at hudson.remoting.Request.call(Request.java:110) 
at hudson.remoting.Channel.call(Channel.java:646) 
at hudson.EnvVars.getRemote(EnvVars.java:202) 
at hudson.model.Computer.getEnvironment(Computer.java:837) 
at hudson.model.Run.getEnvironment(Run.java:1887) 
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:837) 
at hudson.model.Run.getEnvironment(Run.java:1864) 
at sun.reflect.GeneratedMethodAccessor3345.invoke(Unknown Source) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
at java.lang.reflect.Method.invoke(Method.java:597) 
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) 
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) 
at groovy.lang.MetaClassImpl$GetBeanMethodMetaProperty.getProperty(MetaClassImpl.java:3457) 
at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:61) 
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:227) 
at Script1.run(Script1.groovy:1) 
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:580) 
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:618) 
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:589) 
at org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildRecorder.perform(GroovyPostbuildRecorder.java:273) 
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) 
at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:711) 
at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:686) 
at hudson.model.Build$RunnerImpl.post2(Build.java:162) 
at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:633) 
at hudson.model.Run.run(Run.java:1463) 
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) 
at hudson.model.ResourceController.execute(ResourceController.java:88) 
at hudson.model.Executor.run(Executor.java:239) 
Caused by: java.io.IOException 
at hudson.remoting.Channel.close(Channel.java:878) 
at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:110) 
at hudson.remoting.PingThread.ping(PingThread.java:114) 
at hudson.remoting.PingThread.run(PingThread.java:81) 
Caused by: java.util.concurrent.TimeoutException: Ping started on 1374740621714 hasn't completed at 1374740861714 
... 2 more 
Caused by: java.util.concurrent.TimeoutException 
at hudson.remoting.Request$1.get(Request.java:249) 
at hudson.remoting.Request$1.get(Request.java:184) 
at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) 
at hudson.remoting.PingThread.ping(PingThread.java:107) 
... 1 more

          Kohsuke Kawaguchi added a comment - As I said, a ping error indicates that the master didn't hear back a ping response from the slave after 4 minutes. Aside from a network problem, there's another possibility I failed to consider in my previous comment — that the TCP/SSH connection to the slave is intact, but the slave JVM is choking and being unable to respond. For example, if a process is paused (such as "kill -STOP"), you could see this. Or if the slave has a massive heap and it's taking more than 4 minutes for GC, you could see this. Or theoretically, if the heap is so saturated that VM is doing one full GC after another, you can see this, too. See https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError and try to diagnose the slave heap problem. -verbose:gc or other means to monitor the GC performance could be useful, too. ERROR: Failed to evaluate groovy script. hudson.remoting.ChannelClosedException: channel is already closed at hudson.remoting.Channel.send(Channel.java:475) at hudson.remoting.Request.call(Request.java:110) at hudson.remoting.Channel.call(Channel.java:646) at hudson.EnvVars.getRemote(EnvVars.java:202) at hudson.model.Computer.getEnvironment(Computer.java:837) at hudson.model.Run.getEnvironment(Run.java:1887) at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:837) at hudson.model.Run.getEnvironment(Run.java:1864) at sun.reflect.GeneratedMethodAccessor3345.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233) at groovy.lang.MetaClassImpl$GetBeanMethodMetaProperty.getProperty(MetaClassImpl.java:3457) at org.codehaus.groovy.runtime.callsite.GetEffectivePojoPropertySite.getProperty(GetEffectivePojoPropertySite.java:61) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGetProperty(AbstractCallSite.java:227) at Script1.run(Script1.groovy:1) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:580) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:618) at groovy.lang.GroovyShell.evaluate(GroovyShell.java:589) at org.jvnet.hudson.plugins.groovypostbuild.GroovyPostbuildRecorder.perform(GroovyPostbuildRecorder.java:273) at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:19) at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:711) at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:686) at hudson.model.Build$RunnerImpl.post2(Build.java:162) at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:633) at hudson.model.Run.run(Run.java:1463) at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:46) at hudson.model.ResourceController.execute(ResourceController.java:88) at hudson.model.Executor.run(Executor.java:239) Caused by: java.io.IOException at hudson.remoting.Channel.close(Channel.java:878) at hudson.slaves.ChannelPinger$1.onDead(ChannelPinger.java:110) at hudson.remoting.PingThread.ping(PingThread.java:114) at hudson.remoting.PingThread.run(PingThread.java:81) Caused by: java.util.concurrent.TimeoutException: Ping started on 1374740621714 hasn't completed at 1374740861714 ... 2 more Caused by: java.util.concurrent.TimeoutException at hudson.remoting.Request$1.get(Request.java:249) at hudson.remoting.Request$1.get(Request.java:184) at hudson.remoting.FutureAdapter.get(FutureAdapter.java:59) at hudson.remoting.PingThread.ping(PingThread.java:107) ... 1 more

          Kohsuke Kawaguchi added a comment -

          We are not hearing back additional details, so marking as closed until someone posts more details.

          Kohsuke Kawaguchi added a comment - We are not hearing back additional details, so marking as closed until someone posts more details.

          Łukasz Jąder added a comment -

          Set issue to Closed (no details provided for nearly 4 years)

          Łukasz Jąder added a comment - Set issue to Closed (no details provided for nearly 4 years)

            kohsuke Kohsuke Kawaguchi
            hurricanechristian christian symmons
            Votes:
            1 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: