-
Bug
-
Resolution: Fixed
-
Minor
-
Jenkins - 2.222.1
ec2-plugin - 1.49.1
Jenkins EC2 Plugin fails to launch Windows Slave after upgrading the plugin version from 1.46.1 to 1.49.1. While the Linux agents come up perfectly, the Windows agents fail to come up and fail with the below error.
I believe this issue is due to the update of the smbj version as mentioned in the release notes. In order to get my agents working, I had to revert to the previous version of the plugin (1.46.1).
Please investigate and advise on the fix.
WARNING h.plugins.ec2.win.WinConnection#ping: Failed to verify connectivity to Windows slave java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.readFully(DirectTcpPacketReader.java:70) at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.readTcpHeader(DirectTcpPacketReader.java:59) at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.doRead(DirectTcpPacketReader.java:48) Caused: com.hierynomus.protocol.transport.TransportException at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.doRead(DirectTcpPacketReader.java:53) at com.hierynomus.smbj.transport.PacketReader.readPacket(PacketReader.java:70) at com.hierynomus.smbj.transport.PacketReader.run(PacketReader.java:48) Caused: com.hierynomus.smbj.common.SMBRuntimeException at com.hierynomus.smbj.common.SMBRuntimeException$1.wrap(SMBRuntimeException.java:27) at com.hierynomus.smbj.common.SMBRuntimeException$1.wrap(SMBRuntimeException.java:21) at com.hierynomus.protocol.commons.concurrent.Promise.deliverError(Promise.java:95) at com.hierynomus.smbj.connection.OutstandingRequests.handleError(OutstandingRequests.java:88) at com.hierynomus.smbj.connection.Connection.handleError(Connection.java:465) at com.hierynomus.smbj.transport.PacketReader.run(PacketReader.java:54) Caused: java.util.concurrent.ExecutionException at com.hierynomus.protocol.commons.concurrent.PromiseBackedFuture.get(PromiseBackedFuture.java:60) at com.hierynomus.protocol.commons.concurrent.CancellableFuture.get(CancellableFuture.java:84) at com.hierynomus.protocol.commons.concurrent.Futures.get(Futures.java:38) Caused: com.hierynomus.protocol.transport.TransportException at com.hierynomus.protocol.transport.TransportException$1.wrap(TransportException.java:29) at com.hierynomus.protocol.transport.TransportException$1.wrap(TransportException.java:23) at com.hierynomus.protocol.commons.concurrent.Futures.get(Futures.java:43) at com.hierynomus.smbj.connection.Connection.sendAndReceive(Connection.java:305) at com.hierynomus.smbj.connection.Connection.initiateSessionSetup(Connection.java:244) at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:181) Caused: com.hierynomus.smbj.common.SMBRuntimeException at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:215) at hudson.plugins.ec2.win.WinConnection.ping(WinConnection.java:112) at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:181) at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:39) at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:291) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) WARNING h.plugins.ec2.win.WinConnection#ping: Failed to verify connectivity to Windows slave java.net.SocketException: Broken pipe (Write failed) at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:75) Caused: com.hierynomus.protocol.transport.TransportException at com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:78) at com.hierynomus.smbj.connection.Connection.send(Connection.java:297) at com.hierynomus.smbj.connection.Connection.sendAndReceive(Connection.java:305) at com.hierynomus.smbj.connection.Connection.initiateSessionSetup(Connection.java:244) at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:181) Caused: com.hierynomus.smbj.common.SMBRuntimeException at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:215) at hudson.plugins.ec2.win.WinConnection.ping(WinConnection.java:112) at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:169) at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:39) at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:291) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[JENKINS-61006] AWS EC2 plugin failing to connect to Windows Slave
Raihaan Shouhell
added a comment - Hey PJ, how do you reproduce this issue? Which version of windows server are you using? I could not replicate this issue on Server 2016 and server 2019. Do you have port 445 open?
Raihaan Shouhell
added a comment - Hey PJ, how do you reproduce this issue? Which version of windows server are you using? I could not replicate this issue on Server 2016 and server 2019. Do you have port 445 open? HI raihaan We are using Win 2016.
In order to reproduce all I need to do was update the EC2 plugin from Plugin manager and then start an EC2 jenkins slave.
It fails to get the password for WinRM and in the jenkins.log we notice the above error.
Dont see any such issues with the 1.46.1 version.
Raihaan Shouhell
added a comment - The strange bit for me is that I do not see this issue on my windows server 2016 instances.
Could you provide the entire agent log?
If you have a fixed password when you update can you ensure that it is set to use the fixed password instead of retrieving the password?
Raihaan Shouhell
added a comment - The strange bit for me is that I do not see this issue on my windows server 2016 instances.
Could you provide the entire agent log?
If you have a fixed password when you update can you ensure that it is set to use the fixed password instead of retrieving the password? HI raihaan
Today I updated the ec2 plugin to the latest (1.50.3) and noticed the following behaviour:
1. The linux Ec2 agents came up online adter 9 mins of launching it. They did throw the error message as in the screenshot 1 attached before connecting to it successfully.
2. The Windows agents continue to the Wait to getting connected as mentioned in the screenshot 2. There are no errors relative to them in the jenkins logs.
In addition to that I see the following Warning messages on the Manage Jenkins screen after uploading the plugin.
As per the instructions for this update, I am not going to dismiss them for now. Since, I already backed-up the previous ec2.jpi for ver 1.46.1, I shall rollback my changes and update you.
I was able to successfully roll back the plugin to 1.46.1 version and was able to launch the linux agents without any issues.
However, I see that the Windows agents, even though were able to connect successfully, left the following errors before establishing connectivity. Don't remember seeing this error in the past when I rolled them back from the trials to update to higher versions.
2020-05-20 20:53:59.135+0000 [id=336] INFO c.h.smbj.transport.PacketReader#run: PacketReader error, got exception.
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.readFully(DirectTcpPacketReader.java:70)
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.readTcpHeader(DirectTcpPacketReader.java:59)
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.doRead(DirectTcpPacketReader.java:48)
Caused: com.hierynomus.protocol.transport.TransportException
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.doRead(DirectTcpPacketReader.java:53)
at com.hierynomus.smbj.transport.PacketReader.readPacket(PacketReader.java:70)
at com.hierynomus.smbj.transport.PacketReader.run(PacketReader.java:48)
at java.lang.Thread.run(Thread.java:748)
2020-05-20 20:53:59.136+0000 [id=336] INFO c.h.smbj.session.Session#logoff: Logging off session 255086697644033 from host IP-address
2020-05-20 20:53:59.136+0000 [id=197] SEVERE c.h.p.commons.concurrent.Promise#tryRetrieve: << 4 >> woke to: {}
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.readFully(DirectTcpPacketReader.java:70)
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.readTcpHeader(DirectTcpPacketReader.java:59)
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.doRead(DirectTcpPacketReader.java:48)
Caused: com.hierynomus.protocol.transport.TransportException
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.doRead(DirectTcpPacketReader.java:53)
at com.hierynomus.smbj.transport.PacketReader.readPacket(PacketReader.java:70)
at com.hierynomus.smbj.transport.PacketReader.run(PacketReader.java:48)
Caused: com.hierynomus.smbj.common.SMBRuntimeException
at com.hierynomus.smbj.common.SMBRuntimeException$1.wrap(SMBRuntimeException.java:27)
at com.hierynomus.smbj.common.SMBRuntimeException$1.wrap(SMBRuntimeException.java:21)
at com.hierynomus.protocol.commons.concurrent.Promise.deliverError(Promise.java:95)
at com.hierynomus.smbj.connection.OutstandingRequests.handleError(OutstandingRequests.java:88)
at com.hierynomus.smbj.connection.Connection.handleError(Connection.java:445)
at com.hierynomus.smbj.transport.PacketReader.run(PacketReader.java:54)
at java.lang.Thread.run(Thread.java:748)
2020-05-20 20:53:59.139+0000 [id=197] WARNING h.plugins.ec2.win.WinConnection#ping: Failed to verify connectivity to Windows slave
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.readFully(DirectTcpPacketReader.java:70)
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.readTcpHeader(DirectTcpPacketReader.java:59)
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.doRead(DirectTcpPacketReader.java:48)
Caused: com.hierynomus.protocol.transport.TransportException
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpPacketReader.doRead(DirectTcpPacketReader.java:53)
at com.hierynomus.smbj.transport.PacketReader.readPacket(PacketReader.java:70)
at com.hierynomus.smbj.transport.PacketReader.run(PacketReader.java:48)
Caused: com.hierynomus.smbj.common.SMBRuntimeException
at com.hierynomus.smbj.common.SMBRuntimeException$1.wrap(SMBRuntimeException.java:27)
at com.hierynomus.smbj.common.SMBRuntimeException$1.wrap(SMBRuntimeException.java:21)
at com.hierynomus.protocol.commons.concurrent.Promise.deliverError(Promise.java:95)
at com.hierynomus.smbj.connection.OutstandingRequests.handleError(OutstandingRequests.java:88)
at com.hierynomus.smbj.connection.Connection.handleError(Connection.java:445)
at com.hierynomus.smbj.transport.PacketReader.run(PacketReader.java:54)
Caused: java.util.concurrent.ExecutionException
at com.hierynomus.protocol.commons.concurrent.PromiseBackedFuture.get(PromiseBackedFuture.java:60)
at com.hierynomus.protocol.commons.concurrent.CancellableFuture.get(CancellableFuture.java:84)
at com.hierynomus.protocol.commons.concurrent.Futures.get(Futures.java:38)
Caused: com.hierynomus.protocol.transport.TransportException
at com.hierynomus.protocol.transport.TransportException$1.wrap(TransportException.java:29)
at com.hierynomus.protocol.transport.TransportException$1.wrap(TransportException.java:23)
at com.hierynomus.protocol.commons.concurrent.Futures.get(Futures.java:43)
at com.hierynomus.smbj.connection.Connection.sendAndReceive(Connection.java:285)
at com.hierynomus.smbj.connection.Connection.initiateSessionSetup(Connection.java:232)
at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:169)
Caused: com.hierynomus.smbj.common.SMBRuntimeException
at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:203)
at hudson.plugins.ec2.win.WinConnection.ping(WinConnection.java:120)
at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:181)
at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:39)
at hudson.plugins.ec2.EC2ComputerLauncher.launch(EC2ComputerLauncher.java:48)
at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:292)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-05-20 20:53:59.141+0000 [id=336] SEVERE c.h.smbj.session.Session#logoff: Caught exception while closing TreeConnect with id: 1
java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:75)
Caused: com.hierynomus.protocol.transport.TransportException
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:78)
at com.hierynomus.smbj.connection.Connection.send(Connection.java:277)
at com.hierynomus.smbj.session.Session.send(Session.java:285)
at com.hierynomus.smbj.share.TreeConnect.close(TreeConnect.java:66)
at com.hierynomus.smbj.share.Share.close(Share.java:115)
at com.hierynomus.smbj.session.Session.logoff(Session.java:226)
at com.hierynomus.smbj.session.Session.close(Session.java:267)
at com.hierynomus.smbj.connection.Connection.close(Connection.java:141)
at com.hierynomus.smbj.connection.Connection.close(Connection.java:126)
at com.hierynomus.smbj.connection.Connection.handleError(Connection.java:447)
at com.hierynomus.smbj.transport.PacketReader.run(PacketReader.java:54)
at java.lang.Thread.run(Thread.java:748)
2020-05-20 20:53:59.141+0000 [id=336] WARNING c.h.smbj.connection.Connection#close: Exception while closing session 255086697644033
java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:75)
Caused: com.hierynomus.protocol.transport.TransportException
at com.hierynomus.smbj.transport.tcp.direct.DirectTcpTransport.write(DirectTcpTransport.java:78)
at com.hierynomus.smbj.connection.Connection.send(Connection.java:277)
at com.hierynomus.smbj.session.Session.send(Session.java:285)
at com.hierynomus.smbj.session.Session.logoff(Session.java:240)
at com.hierynomus.smbj.session.Session.close(Session.java:267)
at com.hierynomus.smbj.connection.Connection.close(Connection.java:141)
at com.hierynomus.smbj.connection.Connection.close(Connection.java:126)
at com.hierynomus.smbj.connection.Connection.handleError(Connection.java:447)
at com.hierynomus.smbj.transport.PacketReader.run(PacketReader.java:54)
at java.lang.Thread.run(Thread.java:748)
2020-05-20 20:53:59.143+0000 [id=181] WARNING c.s.s.utils.LogConsumer#handleRetry: will resend the message:{raw data{thread_id=336, level=INFO, log_source=com.hierynomus.smbj.transport.PacketReader run, message=PacketReader error, got exception.}}
2020-05-20 20:54:10.051+0000 [id=197] INFO c.h.smbj.connection.Connection#connect: Successfully connected to: IP-address
2020-05-20 20:54:10.104+0000 [id=197] INFO c.h.smbj.connection.Connection#authenticate: Successfully authenticated jenkins on IP-address, session is 259484744155137
2020-05-20 20:54:10.104+0000 [id=197] INFO c.h.smbj.session.Session#connectTree: Connecting to \\IP-address\IPC$ on session 259484744155137
2020-05-20 20:54:11.880+0000 [id=197] INFO c.h.smbj.connection.Connection#authenticate: Successfully authenticated jenkins on IP-address, session is 259484744155141
2020-05-20 20:54:11.880+0000 [id=197] INFO c.h.smbj.session.Session#connectTree: Connecting to \\IP-address\C$ on session 259484744155141
2020-05-20 20:54:11.913+0000 [id=197] INFO c.h.smbj.connection.Connection#authenticate: Successfully authenticated jenkins on IP-address, session is 259484744155145
2020-05-20 20:54:11.914+0000 [id=197] INFO c.h.smbj.session.Session#connectTree: Connecting to \\IP-address\C$ on session 259484744155145
2020-05-20 20:54:14.670+0000 [id=197] INFO c.h.smbj.connection.Connection#authenticate: Successfully authenticated jenkins on IP-address, session is 259484744155149
2020-05-20 20:54:14.670+0000 [id=197] INFO c.h.smbj.session.Session#connectTree: Connecting to \\IP-address\C$ on session 259484744155149
2020-05-20 20:54:14.705+0000 [id=197] INFO c.h.smbj.connection.Connection#authenticate: Successfully authenticated jenkins on IP-address, session is 259484744155153
2020-05-20 20:54:14.705+0000 [id=197] INFO c.h.smbj.session.Session#connectTree: Connecting to \\IP-address\C$ on session 259484744155153
Adding the screenshot for the launch of this agent below:
Raihaan Shouhell
added a comment - The linux error is expected due to security issues there is a host key verifying strategy to prevent MitM attacks. You can turn this off(there is a new setting for this)
So you are telling me that there is no issue with windows agents in 1.50.3?
Raihaan Shouhell
added a comment - The linux error is expected due to security issues there is a host key verifying strategy to prevent MitM attacks. You can turn this off(there is a new setting for this)
So you are telling me that there is no issue with windows agents in 1.50.3? No, I see that none of the plugin versions are working since 1.46.1, atleast for the windows EC2 agents. With 1.50.3, the Windows agents could not proceed to use the WinRM step to login as mentioned in my screenshot here:
Raihaan Shouhell
added a comment - can you link me your whole configuration option when you migrate to 1.50.3?
Can you ensure that the password is specified as well.
Raihaan Shouhell
added a comment - can you link me your whole configuration option when you migrate to 1.50.3?
Can you ensure that the password is specified as well. We've actually hit this same bug and have been watching this issue for a while. Downgrading to 1.46.1 fixes the issue for us, every version that has been released after that (we tested all of them, since 1.47) end up in the issue as described. Windows Jenkins agents simply cannot hold/get a connection, for any version after 1.46.1.
Tested with several different WinRM settings, from memory to timeouts, nothing seems to resolve the issue beyond keeping it at 1.46.1. Even disabled pinging on both the master and the agent, didn't make any difference.
Ivo Teel
added a comment - - edited We've actually hit this same bug and have been watching this issue for a while. Downgrading to 1.46.1 fixes the issue for us, every version that has been released after that (we tested all of them, since 1.47) end up in the issue as described. Windows Jenkins agents simply cannot hold/get a connection, for any version after 1.46.1.
Tested with several different WinRM settings, from memory to timeouts, nothing seems to resolve the issue beyond keeping it at 1.46.1. Even disabled pinging on both the master and the agent, didn't make any difference. HI raihaan
Yes. Here is an XML config of the windows build-agent I am trying to bring up for all versions. As you can see the pwd is specified as well including a boot delay of 180 secs.
Let me know if this info helps.
This XML file does not appear to have any style information associated with it. The document tree is shown below. <hudson.plugins.ec2.EC2Slave plugin="ec2@1.46.1"> <name> EC2 (build-agent-cloud-4) - build-agent-windows-build-agent (i-123456789) </name> <remoteFS>C:\Users\jenkins</remoteFS> <numExecutors>3</numExecutors> <mode>EXCLUSIVE</mode> <retentionStrategy class="hudson.plugins.ec2.EC2RetentionStrategy"> <nextCheckAfter>1590533896339</nextCheckAfter> <idleTerminationMinutes>45</idleTerminationMinutes> </retentionStrategy> <launcher class="hudson.plugins.ec2.win.EC2WindowsLauncher"> <sleepBetweenAttempts>10000</sleepBetweenAttempts> </launcher> <label>windows-agent</label> <nodeProperties/> <instanceId>i-123456789</instanceId> <initScript> git config --global pack.windowsMemory 256m </initScript> <tmpDir/> <remoteAdmin>jenkins</remoteAdmin> <templateDescription>windows-build-agent</templateDescription> <jvmopts>-Djenkins.agentname=windows-build-agent </jvmopts> <stopOnTerminate>false</stopOnTerminate> <idleTerminationMinutes>45</idleTerminationMinutes> <useDedicatedTenancy>false</useDedicatedTenancy> <isConnected>false</isConnected> <tags class="linked-list"> <hudson.plugins.ec2.EC2Tag> <name>Name</name> <value>jenkins-windows-build-agent</value> </hudson.plugins.ec2.EC2Tag> <hudson.plugins.ec2.EC2Tag> <name>jenkins_server_url</name> <value>https://jenkins.example.com/</value> </hudson.plugins.ec2.EC2Tag> <hudson.plugins.ec2.EC2Tag> <name>jenkins_slave_type</name> <value>demand_build-agent-windows-build-agent</value> </hudson.plugins.ec2.EC2Tag> <hudson.plugins.ec2.EC2Tag> <name>backup_disable</name> <value>true</value> </hudson.plugins.ec2.EC2Tag> </tags> <cloudName>ec2-build-agent-cloud-4</cloudName> <amiType class="hudson.plugins.ec2.WindowsData"> <password> {$password} </password> <useHTTPS>false</useHTTPS> <bootDelay>180</bootDelay> <specifyPassword>true</specifyPassword> </amiType> <maxTotalUses>-1</maxTotalUses> <instanceType>c5.2xlarge</instanceType> <launchTimeout>2147483647</launchTimeout> </hudson.plugins.ec2.EC2Slave>
HI raihaan
Just checking in if you got a chance to review my last comment and/or have any updates.
Thanks a lot for your help !
Let me ask you some questions to try to reduce the factors implied:
- Do you allow self-signed certificates for this windows instance?
- You're not using https, right?
Try to allow self-signed certificates to see if it helps. The latest release tries to avoid them for security reasons. But let's start from the beginning and iterate to solve your issue.
Configuring a Windows instance to be totally secure is very cumbersome. Please check the documentation on the plugin GitHub repo: https://github.com/jenkinsci/ec2-plugin/#securing-the-connection-to-windows-amis and in the AMI Type field help, under the template configuration on your Jenkins instance.
Let's first try without security in place to figure out what's going on
Steven Astac
added a comment - I had the same issue, Windows 10 AMI, telnet 445 / 5985 / 5986 all succeeded, using Remote Desktop app to login using Administrator +password works.
Direct HTTP, non-verify cert is ticked. Firewall is not an issue.
I just see
Connecting to (<ip>) with WinRM as Administrator, and hanging. Eventually it got terminated due to timeout.
The plugin is what's on the master branch (1.51) right now.
Steven Astac
added a comment - I had the same issue, Windows 10 AMI, telnet 445 / 5985 / 5986 all succeeded, using Remote Desktop app to login using Administrator +password works.
Direct HTTP, non-verify cert is ticked. Firewall is not an issue.
I just see
Connecting to (<ip>) with WinRM as Administrator, and hanging. Eventually it got terminated due to timeout.
The plugin is what's on the master branch (1.51) right now. I too am experiencing the same issue as sastac.
An AMI that used to previously connect without issue is now failing to connect.
Jenkins immediately displays error "This agent is offline because Jenkins failed to launch the agent process on it"
Checking the log does not provide any insight into the cause of the failure. Jenkins says the AMI booted and that WinRM is connecting, however that just hangs indefinitely:
The AMI is configured to connect via HTTP and accept self-signed certificates.
EC2 Plugin is latest available version
Andrei Juc
added a comment - - edited I too am experiencing the same issue as sastac .
An AMI that used to previously connect without issue is now failing to connect.
Jenkins immediately displays error "This agent is offline because Jenkins failed to launch the agent process on it"
Checking the log does not provide any insight into the cause of the failure. Jenkins says the AMI booted and that WinRM is connecting, however that just hangs indefinitely:
The AMI is configured to connect via HTTP and accept self-signed certificates.
EC2 Plugin is latest available version I am seeing a similar behaviour. Nothing in the logs but the master hangs connecting. It used to work on 1.46.1.
Hung Vo
added a comment - Same issue from me. Downgrade to 1.46.1 work Hello Everyone !
Looks like the current version of the EC2 plugin ( 1.151) has fixed the issue with the SLF4J for the smbj.
I tested this yday ( over Jenkins 2.235.1 jenkins ver) and was able to launch Windows and linux agents without any issues.
Although I did receive longer agent-launch times, due to the MitM Security checks in place, I was able to go around them and launch agents successfully.
Here are more details of the fix as part of the 1.151 release notes - https://github.com/jenkinsci/ec2-plugin/pull/481.
Looks like the version of the EC2 plugin ( 1.151) has fixed the issue with the SLF4J for the smbj.
I tested this yday ( over Jenkins 2.235.1 jenkins ver) and was able to launch Windows and linux agents without any issues.
Although I did receive longer agent-launch times, due to the MitM Security checks in place, I was able to go around them and launch agents successfully.
Here are more details of the fix as part of the 1.151 release notes - https://github.com/jenkinsci/ec2-plugin/pull/481.
Mark
added a comment - v 1.58
Was able to setup windows node with winrm over https and self-signed ssl and accept-new strategy.
Node comes up and shuts down after setup period of inactivity, but will not launch again.
Mark
added a comment - v 1.58
Was able to setup windows node with winrm over https and self-signed ssl and accept-new strategy.
Node comes up and shuts down after setup period of inactivity, but will not launch again. I tried with 1.60.1 and was getting the same error.
2021-06-09 14:05:58.257+0000 [id=126] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agent2021-06-09 14:05:58.257+0000 [id=126] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agentjava.lang.UnsupportedOperationException
The latest version that worked for me is 1.45.
After a lot of debugging I found out that the issue is caused by smbj. I recompiled ec2-plugin v 1.60.1 myself but using smbj 0.9.1.
Now everything work as expected.
walid boussafa
added a comment - - edited I tried with 1.60.1 and was getting the same error.
2021-06-09 14:05:58.257+0000 [id=126] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agent2021-06-09 14:05:58.257+0000 [id=126] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agentjava.lang.UnsupportedOperationException
The latest version that worked for me is 1.45.
After a lot of debugging I found out that the issue is caused by smbj. I recompiled ec2-plugin v 1.60.1 myself but using smbj 0.9.1.
Now everything work as expected.
Guy Davis
added a comment - Good day Walid! We're facing the exact issue you describe with latest v1.60.1. Would you mind providing some more details about the steps you took to build a working version of the plugin? Any tips you can provide would be most appreciated!
Guy Davis
added a comment - Good day Walid! We're facing the exact issue you describe with latest v1.60.1. Would you mind providing some more details about the steps you took to build a working version of the plugin? Any tips you can provide would be most appreciated! Raihaan Shouhell
added a comment - boussaffawalid could you provide a longer stacktrace?
guy_davis which version of the plugin was last working for you?
Raihaan Shouhell
added a comment - boussaffawalid could you provide a longer stacktrace?
guy_davis which version of the plugin was last working for you? Raihaan Shouhell
added a comment - Possibly Walid's and Guy's issue is caused by https://github.com/hierynomus/smbj/issues/638
Raihaan Shouhell
added a comment - Possibly Walid's and Guy's issue is caused by https://github.com/hierynomus/smbj/issues/638 walid boussafa
added a comment - raihaan Probably yes, here is a longer trace.
2021-06-09 07:32:33.471+0000 [id=116] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agent2021-06-09 07:32:33.471+0000 [id=116] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agentjava.lang.UnsupportedOperationException at com.hierynomus.security.jce.JceSecurityProvider.getDerivationFunction(JceSecurityProvider.java:64) at com.hierynomus.smbj.connection.SMBSessionBuilder.deriveKey(SMBSessionBuilder.java:302) at com.hierynomus.smbj.connection.SMBSessionBuilder.deriveKeys(SMBSessionBuilder.java:270) at com.hierynomus.smbj.connection.SMBSessionBuilder.setupSession(SMBSessionBuilder.java:162) at com.hierynomus.smbj.connection.SMBSessionBuilder.setupSession(SMBSessionBuilder.java:142) at com.hierynomus.smbj.connection.SMBSessionBuilder.establish(SMBSessionBuilder.java:109) at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:192) at hudson.plugins.ec2.win.WinConnection.pingFailingIfSSHHandShakeError(WinConnection.java:127) at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:220) at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:53) at hudson.plugins.ec2.EC2JNLPLauncher.launch(EC2JNLPLauncher.java:48) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:294) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
walid boussafa
added a comment - raihaan Probably yes, here is a longer trace.
2021-06-09 07:32:33.471+0000 [id=116] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agent2021-06-09 07:32:33.471+0000 [id=116] WARNING h.plugins.ec2.win.WinConnection#pingFailingIfSSHHandShakeError: Failed to verify connectivity to Windows agentjava.lang.UnsupportedOperationException at com.hierynomus.security.jce.JceSecurityProvider.getDerivationFunction(JceSecurityProvider.java:64) at com.hierynomus.smbj.connection.SMBSessionBuilder.deriveKey(SMBSessionBuilder.java:302) at com.hierynomus.smbj.connection.SMBSessionBuilder.deriveKeys(SMBSessionBuilder.java:270) at com.hierynomus.smbj.connection.SMBSessionBuilder.setupSession(SMBSessionBuilder.java:162) at com.hierynomus.smbj.connection.SMBSessionBuilder.setupSession(SMBSessionBuilder.java:142) at com.hierynomus.smbj.connection.SMBSessionBuilder.establish(SMBSessionBuilder.java:109) at com.hierynomus.smbj.connection.Connection.authenticate(Connection.java:192) at hudson.plugins.ec2.win.WinConnection.pingFailingIfSSHHandShakeError(WinConnection.java:127) at hudson.plugins.ec2.win.EC2WindowsLauncher.connectToWinRM(EC2WindowsLauncher.java:220) at hudson.plugins.ec2.win.EC2WindowsLauncher.launchScript(EC2WindowsLauncher.java:53) at hudson.plugins.ec2.EC2JNLPLauncher.launch(EC2JNLPLauncher.java:48) at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:294) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:80) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang. Thread .run( Thread .java:829)
Raihaan Shouhell
added a comment - Could you test https://ci.jenkins.io/job/Plugins/job/ec2-plugin/job/PR-630/1/artifact/org/jenkins-ci/plugins/ec2/1.61-rc1345.42cd2c44cdfd/ec2-1.61-rc1345.42cd2c44cdfd.hpi to see if this resolves your issue?
Change is https://github.com/jenkinsci/ec2-plugin/pull/630
Raihaan Shouhell
added a comment - Could you test https://ci.jenkins.io/job/Plugins/job/ec2-plugin/job/PR-630/1/artifact/org/jenkins-ci/plugins/ec2/1.61-rc1345.42cd2c44cdfd/ec2-1.61-rc1345.42cd2c44cdfd.hpi to see if this resolves your issue?
Change is https://github.com/jenkinsci/ec2-plugin/pull/630 walid boussafa
added a comment - raihaan I tested your fix and it resolves the issue. Thanks! Raihaan Shouhell
added a comment - guy_davis boussaffawalid I'll be releasing 1.61 which will resolve the issue in the next half hour or so.
Raihaan Shouhell
added a comment - guy_davis boussaffawalid I'll be releasing 1.61 which will resolve the issue in the next half hour or so. Guy Davis
added a comment - raihaan Thanks! Yes, I can confirm that the now EC2 plugin at version 1.61 has corrected the launch issue I replicated yesterday in 1.60.1. We're able to launch Windows agents again! Quick turnaround on this fix is much appreciated.
Guy Davis
added a comment - raihaan Thanks! Yes, I can confirm that the now EC2 plugin at version 1.61 has corrected the launch issue I replicated yesterday in 1.60.1. We're able to launch Windows agents again! Quick turnaround on this fix is much appreciated. 
HI thoulen Could you please update on this ? We are seeing the same errors even after updating the plugin to the latest 1.50.1 version.