Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-68346

jenkins agent disconnected unexpected termination of channel

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • remoting
    • None
    • jenkins contoller running on rhel7 at Jenkins 2.332.1 lts version, upgrded to 2.346.1
      openstack cloud rhel7 & rhel9 and local linux rhel7 & rhel8 agents both disconnecting

      jenkins controller upgrade on Monday a week ago and agent reliability is very bad
      sometimes they stop immediately, sometimes they build for 3+ hours
      I've reviewd the remoting info for ideas, and will include the agent log
      this morning I doubled the controller memory Jenkins
      I updated my local and openstack agents to run remoting.jar under the same jdk-11.0.14.1+1 as the controller

      the build fails with 
      16:21:36 FATAL: command execution failed
      16:24:49 java.io.EOFException
      16:24:49  at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2905)
      16:24:49  at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3400)
      16:24:49  at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:936)
      16:24:49  at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
      16:24:49  at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
      16:24:49  at hudson.remoting.Command.readFrom(Command.java:142)
      16:24:49  at hudson.remoting.Command.readFrom(Command.java:128)
      16:24:49  at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
      16:24:49  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
      16:24:49 Caused: java.io.IOException: Unexpected termination of the channel
      16:24:49  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:75)
      16:24:49 Caused: java.io.IOException: Backing channel 'mhub-linux9' is disconnected.
      16:24:49  at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)
      16:24:49  at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
      16:24:49  at com.sun.proxy.$Proxy105.isAlive(Unknown Source)
      16:24:49  at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1215)
      16:24:49  at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1207)
      16:24:49  at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:195)
      16:24:49  at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:145)
      16:24:49  at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
      16:24:49  at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
      16:24:49  at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:814)
      16:24:49  at hudson.model.Build$BuildExecution.build(Build.java:199)
      16:24:49  at hudson.model.Build$BuildExecution.doRun(Build.java:164)
      16:24:49  at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:522)
      16:24:49  at hudson.model.Run.execute(Run.java:1896)
      16:24:49  at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
      16:24:49  at hudson.model.ResourceController.execute(ResourceController.java:101)
      16:24:49  at hudson.model.Executor.run(Executor.java:442)
      16:24:49 FATAL: Unable to delete script file /tmp/jenkins15763002140024398353.sh
      16:24:49 java.io.EOFException
      16:24:49  at java.base/java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2905)
      16:24:49  at java.base/java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3400)
      16:24:49  at java.base/java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:936)
      16:24:49  at java.base/java.io.ObjectInputStream.<init>(ObjectInputStream.java:379)
      16:24:49  at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:49)
      16:24:49  at hudson.remoting.Command.readFrom(Command.java:142)
      16:24:49  at hudson.remoting.Command.readFrom(Command.java:128)
      16:24:49  at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
      16:24:49  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:61)
      16:24:49 Caused: java.io.IOException: Unexpected termination of the channel
      16:24:49  at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:75)
      16:24:49 Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@50af1e95:mhub-linux9": Remote call on mhub-linux9 failed. The channel is closing down or has closed down
      

      I reviewed a number of the similarly named issues, but none of them were as recent as this LTS, with JDK-11.0.14

        1. agent-remoting-log.txt
          12 kB
          Heather Mardis
        2. build-console.log
          6 kB
          Heather Mardis
        3. controller-log.txt
          10 kB
          Heather Mardis
        4. mhub-jenkins.log
          6 kB
          Heather Mardis
        5. mhub-linux9-slave.log.1
          10 kB
          Heather Mardis

            Unassigned Unassigned
            abinitio_user Heather Mardis
            Votes:
            2 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: