Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61006

AWS EC2 plugin failing to connect to Windows Slave

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Minor Minor
    • ec2-plugin
    • Jenkins - 2.222.1
      ec2-plugin - 1.49.1

      Jenkins EC2 Plugin fails to launch Windows Slave after upgrading the plugin version from 1.46.1 to 1.49.1. While the Linux agents come up perfectly, the Windows agents fail to come up and fail with the below error. 

      I believe this issue is due to the update of the smbj version as mentioned in the release notes. In order to get my agents working, I had to revert to the previous version of the plugin (1.46.1).

       

      Please investigate and advise on the fix.

      WARNING h.plugins.ec2.win.WinConnection#ping: Failed to verify connectivity to Windows slave java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.readFully(DirectTcpPacketReader.java:70) at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.readTcpHeader(DirectTcpPacketReader.java:59) at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.doRead(DirectTcpPacketReader.java:48) Caused: com.hierynomus.protocol.transport.TransportException at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.doRead(DirectTcpPacketReader.java:53) at com.hierynomus.smbj.transport.PacketReader.readPacket(PacketReader.java:70) at com.hierynomus.smbj.transport.PacketReader.run(PacketReader.java:48) Caused: com.hierynomus.smbj.common.SMBRuntimeException at com.hierynomus.smbj.common.SMBRuntimeException$1.wrap(SMBRuntimeException.java:27) at com.hierynomus.smbj.common.SMBRuntimeException$1.wrap(SMBRuntimeException.java:21) at com.hierynomus.protocol.commons.concurrent.Promise.deliverError(Promise.java:95) at com.hierynomus.smbj.connection.OutstandingRequests.handleError(OutstandingRequests.java:88) at com.hierynomus.smbj.connection.Connection.handleError(Connection.java:465) at com.hierynomus.smbj.transport.PacketReader.run(PacketReader.java:54) Caused: java.util.concurrent.ExecutionException at com.hierynomus.protocol.commons.concurrent.PromiseBackedFuture.get(PromiseBackedFuture.java:60) at com.hierynomus.protocol.commons.concurrent.CancellableFuture.get(CancellableFuture.java:84) at com.hierynomus.protocol.commons.concurrent.Futures.get(Futures.java:38) Caused: com.hierynomus.protocol.transport.TransportException at com.hierynomus.protocol.transport.TransportException$1.wrap(TransportException.java:29) at com.hierynomus.protocol.transport.TransportException$1.wrap(TransportException.java:23) at com.hierynomus.protocol.commons.concurrent.Futures.get(Futures.java:43) at com.hierynomus.smbj.connection.Connection.sendAndReceive(Connection.java:305) at com.hierynomus.smbj.connection.Connection.initiateSessionSetup(Connection.java:244) at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:181) Caused: com.hierynomus.smbj.common.SMBRuntimeException at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:215) at hudson.plugins.ec2.win.WinConnection.ping(WinConnection.java:112) at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:181) at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:39) at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:291) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)


WARNING h.plugins.ec2.win.WinConnection#ping: Failed to verify connectivity to Windows slave java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:75) Caused: com.hierynomus.protocol.transport.TransportException at com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:78) at com.hierynomus.smbj.connection.Connection.send(Connection.java:297) at com.hierynomus.smbj.connection.Connection.sendAndReceive(Connection.java:305) at com.hierynomus.smbj.connection.Connection.initiateSessionSetup(Connection.java:244) at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:181) Caused: com.hierynomus.smbj.common.SMBRuntimeException at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:215) at hudson.plugins.ec2.win.WinConnection.ping(WinConnection.java:112) at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:169) at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:39) at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:291) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        1. Ec2-linux-error.JPG
          Ec2-linux-error.JPG
          109 kB
        2. Ec2-win-1.46.1.jpg
          Ec2-win-1.46.1.jpg
          176 kB
        3. Ec2-win-error.jpg
          Ec2-win-error.jpg
          59 kB
        4. image-2020-07-24-13-33-11-072.png
          image-2020-07-24-13-33-11-072.png
          3 kB
        5. image-2020-07-24-13-35-10-259.png
          image-2020-07-24-13-35-10-259.png
          4 kB
        6. Jenkins-manage-error.JPG
          Jenkins-manage-error.JPG
          139 kB

          [JENKINS-61006] AWS EC2 plugin failing to connect to Windows Slave

          Mark added a comment -

          v 1.58

          Was able to setup windows node with winrm over https and self-signed ssl and accept-new strategy. 

          Node comes up and shuts down after setup period of inactivity, but will not launch again.

          Mark added a comment - v 1.58 Was able to setup windows node with winrm over https and self-signed ssl and accept-new strategy.  Node comes up and shuts down after setup period of inactivity, but will not launch again.

          walid boussafa added a comment - - edited

          I tried with 1.60.1  and was getting the same error.

           

          2021-06-09 14:05:58.257+0000 [id=126] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agent2021-06-09 14:05:58.257+0000 [id=126] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agentjava.lang.UnsupportedOperationException

           

           

          The latest version that worked for me is 1.45.

           

          After a lot of debugging I found out that the issue is caused by smbj. I recompiled ec2-plugin v 1.60.1 myself but using smbj 0.9.1. 

          Now everything work as expected.

           

          walid boussafa added a comment - - edited I tried with 1.60.1  and was getting the same error.   2021-06-09 14:05:58.257+0000 [id=126] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agent2021-06-09 14:05:58.257+0000 [id=126] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agentjava.lang.UnsupportedOperationException     The latest version that worked for me is 1.45.   After a lot of debugging I found out that the issue is caused by  smbj.  I recompiled ec2-plugin v 1.60.1 myself but using  smbj 0.9.1.   Now everything work as expected.  

          Guy Davis added a comment -

          Good day Walid!  We're facing the exact issue you describe with latest v1.60.1.  Would you mind providing some more details about the steps you took to build a working version of the plugin?  Any tips you can provide would be most appreciated!

          Guy Davis added a comment - Good day Walid!  We're facing the exact issue you describe with latest v1.60.1.  Would you mind providing some more details about the steps you took to build a working version of the plugin?  Any tips you can provide would be most appreciated!

          Raihaan Shouhell added a comment -

          boussaffawalid could you provide a longer stacktrace?

          guy_davis which version of the plugin was last working for you?

          Raihaan Shouhell added a comment - boussaffawalid could you provide a longer stacktrace? guy_davis which version of the plugin was last working for you?

          Raihaan Shouhell added a comment -

          Possibly Walid's and Guy's issue is caused by https://github.com/hierynomus/smbj/issues/638

          Raihaan Shouhell added a comment - Possibly Walid's and Guy's issue is caused by https://github.com/hierynomus/smbj/issues/638

          walid boussafa added a comment -

          raihaan Probably yes, here is a longer trace.

           

          2021-06-09 07:32:33.471+0000 [id=116] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agent2021-06-09 07:32:33.471+0000 [id=116] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agentjava.lang.UnsupportedOperationException at com.hierynomus.security.jce.JceSecurityProvider.getDerivationFunction(JceSecurityProvider.java:64) at com.hierynomus.smbj.connection.SMBSessionBuilder.deriveKey(SMBSessionBuilder.java:302) at com.hierynomus.smbj.connection.SMBSessionBuilder.deriveKeys(SMBSessionBuilder.java:270) at com.hierynomus.smbj.connection.SMBSessionBuilder.setupSession(SMBSessionBuilder.java:162) at com.hierynomus.smbj.connection.SMBSessionBuilder.setupSession(SMBSessionBuilder.java:142) at com.hierynomus.smbj.connection.SMBSessionBuilder.establish(SMBSessionBuilder.java:109) at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:192) at hudson.plugins.ec2.win.WinConnection.pingFailingIfSSHHandShakeError(WinConnection.java:127) at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:220) at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:53) at hudson.plugins.ec2.EC2JNLPLauncher.launch(EC2JNLPLauncher.java:48) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:294) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

          walid boussafa added a comment - raihaan  Probably yes, here is a longer trace.   2021-06-09 07:32:33.471+0000 [id=116] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agent2021-06-09 07:32:33.471+0000 [id=116] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agentjava.lang.UnsupportedOperationException at com.hierynomus.security.jce.JceSecurityProvider.getDerivationFunction(JceSecurityProvider.java:64) at com.hierynomus.smbj.connection.SMBSessionBuilder.deriveKey(SMBSessionBuilder.java:302) at com.hierynomus.smbj.connection.SMBSessionBuilder.deriveKeys(SMBSessionBuilder.java:270) at com.hierynomus.smbj.connection.SMBSessionBuilder.setupSession(SMBSessionBuilder.java:162) at com.hierynomus.smbj.connection.SMBSessionBuilder.setupSession(SMBSessionBuilder.java:142) at com.hierynomus.smbj.connection.SMBSessionBuilder.establish(SMBSessionBuilder.java:109) at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:192) at hudson.plugins.ec2.win.WinConnection.pingFailingIfSSHHandShakeError(WinConnection.java:127) at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:220) at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:53) at hudson.plugins.ec2.EC2JNLPLauncher.launch(EC2JNLPLauncher.java:48) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:294) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang. Thread .run( Thread .java:829)

          Raihaan Shouhell added a comment -

          Could you test https://ci.jenkins.io/job/Plugins/job/ec2-plugin/job/PR-630/1/artifact/org/jenkins-ci/plugins/ec2/1.61-rc1345.42cd2c44cdfd/ec2-1.61-rc1345.42cd2c44cdfd.hpi to see if this resolves your issue?
          Change is https://github.com/jenkinsci/ec2-plugin/pull/630

          Raihaan Shouhell added a comment - Could you test https://ci.jenkins.io/job/Plugins/job/ec2-plugin/job/PR-630/1/artifact/org/jenkins-ci/plugins/ec2/1.61-rc1345.42cd2c44cdfd/ec2-1.61-rc1345.42cd2c44cdfd.hpi to see if this resolves your issue? Change is https://github.com/jenkinsci/ec2-plugin/pull/630

          walid boussafa added a comment -

          raihaan I tested your fix and it resolves the issue. Thanks!

          walid boussafa added a comment - raihaan  I tested your fix and it resolves the issue. Thanks!

          Raihaan Shouhell added a comment -

          guy_davis boussaffawalid I'll be releasing 1.61 which will resolve the issue in the next half hour or so.

          Raihaan Shouhell added a comment - guy_davis boussaffawalid I'll be releasing 1.61 which will resolve the issue in the next half hour or so.

          Guy Davis added a comment -

          raihaan  Thanks!  Yes, I can confirm that the now EC2 plugin at version 1.61 has corrected the launch issue I replicated yesterday in 1.60.1.  We're able to launch Windows agents again!  Quick turnaround on this fix is much appreciated.

          Guy Davis added a comment - raihaan   Thanks!  Yes, I can confirm that the now EC2 plugin at version 1.61 has corrected the launch issue I replicated yesterday in 1.60.1.  We're able to launch Windows agents again!  Quick turnaround on this fix is much appreciated.

            thoulen FABRIZIO MANFREDI
            jpara29 PJ
            Votes:
            2 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: