• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • customtools-plugin
    • None
    • core 2.46.2
      slave.jar 3.7
      JNLP 4
      custom tools plugin 0.5

      I`m running slaves on Univa Grid Engine with "Launch slave agents via SSH". It works great until I add "Install Custom Tool" option.  After that I catch error.

      If I use "Launch agent via Java Web Start" - everything is fine.
       
      SlaveLog.txt MasterConsoleOutput.txtConsoleOutput.txtog.txt^^

        1. 2.36.log
          1.03 MB
        2. 2.38.log
          572 kB
        3. 3.7.log
          207 kB
        4. 3.7.nonSGE.log
          2.02 MB
        5. 3.7.SGE.log
          663 kB
        6. ConsoleOutput.txt
          2 kB
        7. MasterConsoleOutput.txt
          6 kB
        8. slave_3.14.log
          93 kB
        9. SlaveLog.txt
          222 kB

          [JENKINS-44229] Problem with Custom Tools plugin

          Problem starts from slave.jar 2.38+. With 2.37 - everything is fine.

          Ilya Evseenkov added a comment - Problem starts from slave.jar 2.38+. With 2.37 - everything is fine.

          Oleg Nenashev added a comment -

          This slave.jar version is too old to investigate unfortunately. I'd guess there is something being printed to STDERR/STDOUT, which corrupts the remoting protocol over SSH

          Oleg Nenashev added a comment - This slave.jar version is too old to investigate unfortunately. I'd guess there is something being printed to STDERR/STDOUT, which corrupts the remoting protocol over SSH

          Oleg Nenashev added a comment -

          Please provide...

          • Version of the SSH server on the agent side
          • Welcome message you receive after the connection. Maybe this welcome message interferes somehow

          Oleg Nenashev added a comment - Please provide... Version of the SSH server on the agent side Welcome message you receive after the connection. Maybe this welcome message interferes somehow

          I`ve added 3 logs from different slave.jar`s

          2.36 - OK

          2.38 - problem

          3.7 - problem

           

          I see, that no-problem log doesn`t have "hudson.remoting.RemoteInvocationHandler$RPCRequest"

          Ilya Evseenkov added a comment - I`ve added 3 logs from different slave.jar`s 2.36 - OK 2.38 - problem 3.7 - problem   I see, that no-problem log doesn`t have "hudson.remoting.RemoteInvocationHandler$RPCRequest"

          2 files added.

          3.7 SGE and nonSGE logs. Difference from sting 241

          Ilya Evseenkov added a comment - 2 files added. 3.7 SGE and nonSGE logs. Difference from sting 241

          Some more information

           

          SEVERE: I/O error in channel channel java.io.UTFDataFormatException at java.io.ObjectInputStream$BlockDataInputStream.readUTFSpan(ObjectInputStream.java:3432) at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3357) at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3169) at java.io.ObjectInputStream.readString(ObjectInputStream.java:1849) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1526) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at java.util.HashMap.readObject(HashMap.java:1402) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2122) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.Command.readFrom(Command.java:97) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)

          Ilya Evseenkov added a comment - Some more information   SEVERE: I/O error in channel channel java.io.UTFDataFormatException at java.io.ObjectInputStream$BlockDataInputStream.readUTFSpan(ObjectInputStream.java:3432) at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3357) at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3169) at java.io.ObjectInputStream.readString(ObjectInputStream.java:1849) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1526) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at java.util.HashMap.readObject(HashMap.java:1402) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2122) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.Command.readFrom(Command.java:97) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)

          Oleg Nenashev added a comment -

          Please run the connection with FINER logging for ChannelBuilder, let's see what happens with the channel mode.

          Oleg Nenashev added a comment - Please run the connection with FINER logging for ChannelBuilder, let's see what happens with the channel mode.

          Ilya Evseenkov added a comment - Here is slave_3.14.log

          Oleg Nenashev added a comment -

          Now the error is more interesting.

          ERROR: ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
          ERROR: Failed to install restarter
          Connection terminated
          hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
          	at hudson.remoting.Request.abort(Request.java:329)
          	at hudson.remoting.Channel.terminate(Channel.java:992)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:96)
          	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to de02-sgeslot-linux26
          		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
          		at hudson.remoting.Request.call(Request.java:192)
          java.io.EOFException
          	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2624)
          	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3099)
          	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
          	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
          	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
          	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
          Caused: java.io.IOException: Unexpected termination of the channel
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
          		at hudson.remoting.Channel.call(Channel.java:907)
          		at org.jenkinsci.modules.slave_installer.impl.ComputerListenerImpl.onOnline(ComputerListenerImpl.java:32)
          		at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:611)
          		at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:413)
          		at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1074)
          		at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:145)
          		at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:818)
          		at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:793)
          

          Master tries to propagate installers to the SSH slaves. It should not do that (I knew about this issue, created JENKINS-48574 just now). We could fix that, but it should not be a root cause on the issue.

          OTOH I see that:

          FINER: fetch3(org.jenkinsci.modules.slave_installer.impl.InstallerGui)
          Dec 15, 2017 8:32:15 AM hudson.remoting.Channel send
          FINE: Send RPCRequest(2,fetch3)
          Dec 15, 2017 8:32:50 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run
          SEVERE: I/O error in channel channel
          

          So the installer GUI call takes 35 seconds before it gets interrupted.... needs to be fixed

          Oleg Nenashev added a comment - Now the error is more interesting. ERROR: ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins. ERROR: Failed to install restarter Connection terminated hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:329) at hudson.remoting.Channel.terminate(Channel.java:992) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:96) Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to de02-sgeslot-linux26 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693) at hudson.remoting.Request.call(Request.java:192) java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2624) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3099) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) at hudson.remoting.Channel.call(Channel.java:907) at org.jenkinsci.modules.slave_installer.impl.ComputerListenerImpl.onOnline(ComputerListenerImpl.java:32) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:611) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:413) at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1074) at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:145) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:818) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:793) Master tries to propagate installers to the SSH slaves. It should not do that (I knew about this issue, created JENKINS-48574 just now). We could fix that, but it should not be a root cause on the issue. OTOH I see that: FINER: fetch3(org.jenkinsci.modules.slave_installer.impl.InstallerGui) Dec 15, 2017 8:32:15 AM hudson.remoting.Channel send FINE: Send RPCRequest(2,fetch3) Dec 15, 2017 8:32:50 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel channel So the installer GUI call takes 35 seconds before it gets interrupted.... needs to be fixed

          Oleg Nenashev added a comment -

          I have marked the plugin for adoption recently, see https://groups.google.com/d/msg/jenkinsci-dev/uc6NsMoCFQI/AIO4WG1UCwAJ for the context. Unassigning so that anybody else can work on it.

           

          Oleg Nenashev added a comment - I have marked the plugin for adoption recently, see https://groups.google.com/d/msg/jenkinsci-dev/uc6NsMoCFQI/AIO4WG1UCwAJ for the context. Unassigning so that anybody else can work on it.  

            Unassigned Unassigned
            btk3000 Ilya Evseenkov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: