• Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • customtools-plugin
    • None
    • core 2.46.2
      slave.jar 3.7
      JNLP 4
      custom tools plugin 0.5

      I`m running slaves on Univa Grid Engine with "Launch slave agents via SSH". It works great until I add "Install Custom Tool" option.  After that I catch error.

      If I use "Launch agent via Java Web Start" - everything is fine.
       
      SlaveLog.txt MasterConsoleOutput.txtConsoleOutput.txtog.txt^^

        1. SlaveLog.txt
          222 kB
          Ilya Evseenkov
        2. MasterConsoleOutput.txt
          6 kB
          Ilya Evseenkov
        3. ConsoleOutput.txt
          2 kB
          Ilya Evseenkov
        4. 2.38.log
          572 kB
          Ilya Evseenkov
        5. 3.7.log
          207 kB
          Ilya Evseenkov
        6. 2.36.log
          1.03 MB
          Ilya Evseenkov
        7. 3.7.nonSGE.log
          2.02 MB
          Ilya Evseenkov
        8. 3.7.SGE.log
          663 kB
          Ilya Evseenkov
        9. slave_3.14.log
          93 kB
          Ilya Evseenkov

          [JENKINS-44229] Problem with Custom Tools plugin

          Problem starts from slave.jar 2.38+. With 2.37 - everything is fine.

          Ilya Evseenkov added a comment - Problem starts from slave.jar 2.38+. With 2.37 - everything is fine.

          Oleg Nenashev added a comment -

          This slave.jar version is too old to investigate unfortunately. I'd guess there is something being printed to STDERR/STDOUT, which corrupts the remoting protocol over SSH

          Oleg Nenashev added a comment - This slave.jar version is too old to investigate unfortunately. I'd guess there is something being printed to STDERR/STDOUT, which corrupts the remoting protocol over SSH

          Oleg Nenashev added a comment -

          Please provide...

          • Version of the SSH server on the agent side
          • Welcome message you receive after the connection. Maybe this welcome message interferes somehow

          Oleg Nenashev added a comment - Please provide... Version of the SSH server on the agent side Welcome message you receive after the connection. Maybe this welcome message interferes somehow

          I`ve added 3 logs from different slave.jar`s

          2.36 - OK

          2.38 - problem

          3.7 - problem

           

          I see, that no-problem log doesn`t have "hudson.remoting.RemoteInvocationHandler$RPCRequest"

          Ilya Evseenkov added a comment - I`ve added 3 logs from different slave.jar`s 2.36 - OK 2.38 - problem 3.7 - problem   I see, that no-problem log doesn`t have "hudson.remoting.RemoteInvocationHandler$RPCRequest"

          2 files added.

          3.7 SGE and nonSGE logs. Difference from sting 241

          Ilya Evseenkov added a comment - 2 files added. 3.7 SGE and nonSGE logs. Difference from sting 241

          Some more information

           

          SEVERE: I/O error in channel channel java.io.UTFDataFormatException at java.io.ObjectInputStream$BlockDataInputStream.readUTFSpan(ObjectInputStream.java:3432) at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3357) at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3169) at java.io.ObjectInputStream.readString(ObjectInputStream.java:1849) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1526) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at java.util.HashMap.readObject(HashMap.java:1402) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2122) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.Command.readFrom(Command.java:97) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)

          Ilya Evseenkov added a comment - Some more information   SEVERE: I/O error in channel channel java.io.UTFDataFormatException at java.io.ObjectInputStream$BlockDataInputStream.readUTFSpan(ObjectInputStream.java:3432) at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3357) at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3169) at java.io.ObjectInputStream.readString(ObjectInputStream.java:1849) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1526) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at java.util.HashMap.readObject(HashMap.java:1402) at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2122) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2231) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2155) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2013) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) at hudson.remoting.Command.readFrom(Command.java:97) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)

          Oleg Nenashev added a comment -

          Please run the connection with FINER logging for ChannelBuilder, let's see what happens with the channel mode.

          Oleg Nenashev added a comment - Please run the connection with FINER logging for ChannelBuilder, let's see what happens with the channel mode.

          Ilya Evseenkov added a comment - Here is slave_3.14.log

          Oleg Nenashev added a comment -

          Now the error is more interesting.

          ERROR: ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins.
          ERROR: Failed to install restarter
          Connection terminated
          hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel
          	at hudson.remoting.Request.abort(Request.java:329)
          	at hudson.remoting.Channel.terminate(Channel.java:992)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:96)
          	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to de02-sgeslot-linux26
          		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693)
          		at hudson.remoting.Request.call(Request.java:192)
          java.io.EOFException
          	at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2624)
          	at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3099)
          	at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853)
          	at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349)
          	at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
          	at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35)
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63)
          Caused: java.io.IOException: Unexpected termination of the channel
          	at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77)
          		at hudson.remoting.Channel.call(Channel.java:907)
          		at org.jenkinsci.modules.slave_installer.impl.ComputerListenerImpl.onOnline(ComputerListenerImpl.java:32)
          		at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:611)
          		at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:413)
          		at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1074)
          		at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:145)
          		at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:818)
          		at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:793)
          

          Master tries to propagate installers to the SSH slaves. It should not do that (I knew about this issue, created JENKINS-48574 just now). We could fix that, but it should not be a root cause on the issue.

          OTOH I see that:

          FINER: fetch3(org.jenkinsci.modules.slave_installer.impl.InstallerGui)
          Dec 15, 2017 8:32:15 AM hudson.remoting.Channel send
          FINE: Send RPCRequest(2,fetch3)
          Dec 15, 2017 8:32:50 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run
          SEVERE: I/O error in channel channel
          

          So the installer GUI call takes 35 seconds before it gets interrupted.... needs to be fixed

          Oleg Nenashev added a comment - Now the error is more interesting. ERROR: ERROR: Unexpected error in launching a slave. This is probably a bug in Jenkins. ERROR: Failed to install restarter Connection terminated hudson.remoting.RequestAbortedException: java.io.IOException: Unexpected termination of the channel at hudson.remoting.Request.abort(Request.java:329) at hudson.remoting.Channel.terminate(Channel.java:992) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:96) Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to de02-sgeslot-linux26 at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1693) at hudson.remoting.Request.call(Request.java:192) java.io.EOFException at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2624) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3099) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853) at java.io.ObjectInputStream.<init>(ObjectInputStream.java:349) at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48) at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:35) at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:63) Caused: java.io.IOException: Unexpected termination of the channel at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:77) at hudson.remoting.Channel.call(Channel.java:907) at org.jenkinsci.modules.slave_installer.impl.ComputerListenerImpl.onOnline(ComputerListenerImpl.java:32) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:611) at hudson.slaves.SlaveComputer.setChannel(SlaveComputer.java:413) at hudson.plugins.sshslaves.SSHLauncher.startSlave(SSHLauncher.java:1074) at hudson.plugins.sshslaves.SSHLauncher.access$500(SSHLauncher.java:145) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:818) at hudson.plugins.sshslaves.SSHLauncher$2.call(SSHLauncher.java:793) Master tries to propagate installers to the SSH slaves. It should not do that (I knew about this issue, created JENKINS-48574 just now). We could fix that, but it should not be a root cause on the issue. OTOH I see that: FINER: fetch3(org.jenkinsci.modules.slave_installer.impl.InstallerGui) Dec 15, 2017 8:32:15 AM hudson.remoting.Channel send FINE: Send RPCRequest(2,fetch3) Dec 15, 2017 8:32:50 AM hudson.remoting.SynchronousCommandTransport$ReaderThread run SEVERE: I/O error in channel channel So the installer GUI call takes 35 seconds before it gets interrupted.... needs to be fixed

          Oleg Nenashev added a comment -

          I have marked the plugin for adoption recently, see https://groups.google.com/d/msg/jenkinsci-dev/uc6NsMoCFQI/AIO4WG1UCwAJ for the context. Unassigning so that anybody else can work on it.

           

          Oleg Nenashev added a comment - I have marked the plugin for adoption recently, see https://groups.google.com/d/msg/jenkinsci-dev/uc6NsMoCFQI/AIO4WG1UCwAJ for the context. Unassigning so that anybody else can work on it.  

            Unassigned Unassigned
            btk3000 Ilya Evseenkov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: