• Icon: Bug Bug
    • Resolution: Not A Defect
    • Icon: Minor Minor
    • core
    • None
    • CentOS Linux, version 7

      I upgraded from Jenkins 2.319 to 2.320, and all agents failed to start up. I have agents running on Linux, Mac OS, Solaris, and Windows. Below is the error I got.

      The work around is to visit every agent and remove the remoting.jar from the previous  start up. Without an existing remoting.jar, the agents started up just fine. However, in 2.319 and earlier, I didn't need to manually delete the old file.

      java.io.IOException: Could not copy remoting.jar into '/u/build/jenkins/build2' on agent
             at hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:715)
             at hudson.plugins.sshslaves.SSHLauncher.access$300(SSHLauncher.java:112)
             at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:455)
             at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422)
             at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
             at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
             at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
             at java.base/java.lang.Thread.run(Unknown Source)
      Caused by: java.lang.IllegalArgumentException: invalid len argument
             at com.trilead.ssh2.SFTPv3Client.read(SFTPv3Client.java:1232)
             at com.trilead.ssh2.jenkins.SFTPClient$SFTPInputStream.read(SFTPClient.java:172)
             at com.google.common.io.ByteStreams.toByteArrayInternal(ByteStreams.java:184)
             at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:224)
             at hudson.plugins.sshslaves.SSHLauncher.readInputStreamIntoByteArrayAndClose(SSHLauncher.java:773)
             at hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:684)
             ... 7 more
      
      [11/17/21 19:55:59] Launch failed - cleaning up connection
      

          [JENKINS-67165] Jenkins 2.320, Agent Start up Error

          Maciej added a comment -

          Seeing the same issue with agent startup (Could not copy remoting.jar / invalid len argument).
          Jenkins updated yesterday to 2.321 (cannot tell from which version).

          After instance restart all nodes are down with log as above. Some nodes can be started after deleting remoting.jar from their folder, some show no improvement.

          Worth noting that agent-roots are on nfs. Permissions have not changed recently and look to be in line. And most nodes do get working.

          The permanently failing nodes are debian9,centos7. However, some centos7 nodes come up. Centos8 come up.

           

          {{
          [11/18/21 18:03:50] [SSH] Checking java version of java

          [11/18/21 18:03:50] [SSH] java -version returned 1.8.0_242.
          [11/18/21 18:03:50] [SSH] Starting sftp client.
          [11/18/21 18:03:51] [SSH] Copying latest remoting.jar...
          java.io.IOException: Could not copy remoting.jar into '/home/xcb-jenkins-onload/agent-roots/dellr210t' on agent
          at hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:715)
          at hudson.plugins.sshslaves.SSHLauncher.access$300(SSHLauncher.java:112)
          at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:455)
          at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
          Caused by: java.lang.IllegalArgumentException: invalid len argument
          at com.trilead.ssh2.SFTPv3Client.read(SFTPv3Client.java:1232)
          at com.trilead.ssh2.jenkins.SFTPClient$SFTPInputStream.read(SFTPClient.java:172)
          at com.google.common.io.ByteStreams.toByteArrayInternal(ByteStreams.java:184)
          at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:224)
          at hudson.plugins.sshslaves.SSHLauncher.readInputStreamIntoByteArrayAndClose(SSHLauncher.java:773)
          at hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:684)
          ... 7 more
          [11/18/21 18:03:51] Launch failed - cleaning up connection
          [11/18/21 18:03:51] [SSH] Connection closed.
          }}

          Maciej added a comment - Seeing the same issue with agent startup (Could not copy remoting.jar / invalid len argument). Jenkins updated yesterday to 2.321 (cannot tell from which version). After instance restart all nodes are down with log as above. Some nodes can be started after deleting remoting.jar from their folder, some show no improvement. Worth noting that agent-roots are on nfs. Permissions have not changed recently and look to be in line. And most nodes do get working. The permanently failing nodes are debian9,centos7. However, some centos7 nodes come up. Centos8 come up.   {{ [11/18/21 18:03:50] [SSH] Checking java version of java [11/18/21 18:03:50] [SSH] java -version returned 1.8.0_242. [11/18/21 18:03:50] [SSH] Starting sftp client. [11/18/21 18:03:51] [SSH] Copying latest remoting.jar... java.io.IOException: Could not copy remoting.jar into '/home/xcb-jenkins-onload/agent-roots/dellr210t' on agent at hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:715) at hudson.plugins.sshslaves.SSHLauncher.access$300(SSHLauncher.java:112) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:455) at hudson.plugins.sshslaves.SSHLauncher$1.call(SSHLauncher.java:422) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalArgumentException: invalid len argument at com.trilead.ssh2.SFTPv3Client.read(SFTPv3Client.java:1232) at com.trilead.ssh2.jenkins.SFTPClient$SFTPInputStream.read(SFTPClient.java:172) at com.google.common.io.ByteStreams.toByteArrayInternal(ByteStreams.java:184) at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:224) at hudson.plugins.sshslaves.SSHLauncher.readInputStreamIntoByteArrayAndClose(SSHLauncher.java:773) at hudson.plugins.sshslaves.SSHLauncher.copyAgentJar(SSHLauncher.java:684) ... 7 more [11/18/21 18:03:51] Launch failed - cleaning up connection [11/18/21 18:03:51] [SSH] Connection closed. }}

          Shun Cheung added a comment -

          Actually I discovered this problem upgrading from 2.319 to 2.321, and then I went back to 2.320 and I had the same issue. I rolled back to 2.319 and that was fine. Prior to 2.319, I had installed 2.315 and 2.317 ... without any problems. In fact I had never seen this issue before. It looks like the problem was introduced in 2.320 and it continues onto 2.321, which is the latest for this week.

          Shun Cheung added a comment - Actually I discovered this problem upgrading from 2.319 to 2.321, and then I went back to 2.320 and I had the same issue. I rolled back to 2.319 and that was fine. Prior to 2.319, I had installed 2.315 and 2.317 ... without any problems. In fact I had never seen this issue before. It looks like the problem was introduced in 2.320 and it continues onto 2.321, which is the latest for this week.

          Maciej added a comment -

          We have uninstalled and reinstalled jenkins (jenkins-2.321-1.1.noarch), updated plugins (we did before this step but did not help on its own) and issue went away.

          Maciej added a comment - We have uninstalled and reinstalled jenkins (jenkins-2.321-1.1.noarch), updated plugins (we did before this step but did not help on its own) and issue went away.

          Basil Crow added a comment -

          If you see com.google.common.io.ByteStreams.toByteArray (rather than org.apache.commons.io.IOUtils.toByteArray) after hudson.plugins.sshslaves.SSHLauncher.readInputStreamIntoByteArrayAndClose in the stack trace, it means you aren't running jenkinsci/ssh-slaves-plugin#228 (which shipped in 1.32.0). Please ensure you are running the latest version of SSH Build Agents. Disconnect and reconnect the node after upgrading if necessary.

          Basil Crow added a comment - If you see com.google.common.io.ByteStreams.toByteArray (rather than org.apache.commons.io.IOUtils.toByteArray ) after hudson.plugins.sshslaves.SSHLauncher.readInputStreamIntoByteArrayAndClose in the stack trace, it means you aren't running jenkinsci/ssh-slaves-plugin#228 (which shipped in 1.32.0 ). Please ensure you are running the latest version of SSH Build Agents. Disconnect and reconnect the node after upgrading if necessary.

          Shun Cheung added a comment -

          Thank you Basil Crow. I updated the ssh-slaves-plugin and the problem is resolved. Previously I used a script to remove the various remoting.jar files, and that worked also.

          Shun Cheung added a comment - Thank you Basil Crow. I updated the ssh-slaves-plugin and the problem is resolved. Previously I used a script to remove the various remoting.jar files, and that worked also.

            Unassigned Unassigned
            shuncheung Shun Cheung
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: