-
Bug
-
Resolution: Duplicate
-
Minor
-
None
When using idle retention strategy and setting shutting down only option, sometimes the agents may not be able to connect successfully. And it will stay as suspended all the time.
Some useful log from @Sander Bel ,
2019-05-09 09:20:05.878+0000 [id=1060] FINE hudson.model.AsyncPeriodicWork$1#run: Started Azure VM Maintainer Pool Size 2019-05-09 09:20:05.879+0000 [id=1060] FINE hudson.model.AsyncPeriodicWork$1#run: Finished Azure VM Maintainer Pool Size. 1 ms 2019-05-09 09:22:05.560+0000 [id=1165] FINE hudson.model.AsyncPeriodicWork$1#run: Started Azure VM Agents Clean Task 2019-05-09 09:22:05.560+0000 [id=1165] FINE c.m.a.v.AzureVMAgentCleanUpTask#execute: AzureVMAgentCleanUpTask: execute: start 2019-05-09 09:22:05.561+0000 [id=1165] FINE c.m.a.v.AzureVMAgentCleanUpTask#execute: AzureVMAgentCleanUpTask: execute: Running clean with 5 minute timeout 2019-05-09 09:22:05.561+0000 [id=1166] FINE c.m.a.v.AzureVMAgentCleanUpTask#cleanVMs: AzureVMAgentCleanUpTask: cleanVMs: node ubuntu-test3c8d10 blocked to cleanup 2019-05-09 09:22:05.561+0000 [id=1166] FINE c.m.a.v.AzureVMAgentCleanUpTask#cleanDeployments: AzureVMAgentCleanUpTask: cleanDeployments: Cleaning deployments 2019-05-09 09:22:05.561+0000 [id=1166] FINE c.m.a.v.AzureVMAgentCleanUpTask#cleanDeployments: AzureVMAgentCleanUpTask: cleanDeployments: Checking deployment ubuntu-test0509110640988 2019-05-09 09:22:07.340+0000 [id=1166] FINE c.m.a.v.AzureVMAgentCleanUpTask#cleanDeployments: AzureVMAgentCleanUpTask: cleanDeployments: Deployment created on 5/9/19 11:08 AM 2019-05-09 09:22:07.340+0000 [id=1166] FINE c.m.a.v.AzureVMAgentCleanUpTask#cleanDeployments: AzureVMAgentCleanUpTask: cleanDeployments: Deployment newer than timeout, keeping 2019-05-09 09:22:07.340+0000 [id=1166] FINE c.m.a.v.AzureVMAgentCleanUpTask#cleanDeployments: AzureVMAgentCleanUpTask: cleanDeployments: Done cleaning deployments 2019-05-09 09:22:07.439+0000 [id=1165] FINE c.m.a.v.AzureVMAgentCleanUpTask#execute: AzureVMAgentCleanUpTask: execute: end 2019-05-09 09:22:07.439+0000 [id=1165] FINE hudson.model.AsyncPeriodicWork$1#run: Finished Azure VM Agents Clean Task. 1,879 ms 2019-05-09 09:22:54.407+0000 [id=104] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Retrieving agent template with label mdb-sandbox-acs 2019-05-09 09:22:54.407+0000 [id=104] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template ubuntu-test 2019-05-09 09:22:54.407+0000 [id=104] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template test-jenkins-ubuntu 2019-05-09 09:22:54.407+0000 [id=104] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template acostest 2019-05-09 09:23:13.239+0000 [id=103] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Retrieving agent template with label chef 2019-05-09 09:23:13.239+0000 [id=103] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template ubuntu-test 2019-05-09 09:23:13.239+0000 [id=103] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: ubuntu-test matches! 2019-05-09 09:23:13.239+0000 [id=103] INFO c.m.azure.vmagent.AzureVMCloud#provision: AzureVMCloud: provision: start for label chef workLoad 1 2019-05-09 09:23:13.240+0000 [id=103] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Retrieving agent template with label chef 2019-05-09 09:23:13.240+0000 [id=103] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template ubuntu-test 2019-05-09 09:23:13.240+0000 [id=103] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: ubuntu-test matches! 2019-05-09 09:23:13.240+0000 [id=103] INFO c.m.azure.vmagent.AzureVMCloud#provision: AzureVMCloud: provision: checking for node reuse options 2019-05-09 09:23:13.240+0000 [id=103] INFO c.m.azure.vmagent.AzureVMCloud#provision: AzureVMCloud: provision: agent computer eligible for reuse ubuntu-test3c8d10 2019-05-09 09:23:13.240+0000 [id=103] INFO c.m.a.v.AzureVMManagementServiceDelegate#virtualMachineExists: AzureVMManagementServiceDelegate: virtualMachineExists: check for ubuntu-test3c8d10 2019-05-09 09:23:14.168+0000 [id=103] INFO c.m.a.v.AzureVMManagementServiceDelegate#virtualMachineExists: AzureVMManagementServiceDelegate: virtualMachineExists: ubuntu-test3c8d10 exists 2019-05-09 09:23:14.169+0000 [id=103] INFO c.m.azure.vmagent.AzureVMCloud#provision: AzureVMCloud: provision: asynchronous provision finished, returning 1 planned node(s) 2019-05-09 09:23:14.169+0000 [id=930] INFO c.m.azure.vmagent.AzureVMCloud#lambda$provision$0: Found existing node, starting VM ubuntu-test3c8d10 2019-05-09 09:23:14.170+0000 [id=930] INFO c.m.a.v.AzureVMManagementServiceDelegate#startVirtualMachine: AzureVMManagementServiceDelegate: startVirtualMachine: ubuntu-test3c8d10 2019-05-09 09:24:03.370+0000 [id=930] INFO c.m.a.v.AzureVMManagementServiceDelegate#setVirtualMachineDetails: The Azure agent doesn't have a public IP. Will use the private IP 2019-05-09 09:24:03.370+0000 [id=930] INFO c.m.a.v.AzureVMManagementServiceDelegate#setVirtualMachineDetails: Azure agent details: nodeNameubuntu-test3c8d10 adminUserName=jenkins_node shutdownOnIdle=true retentionTimeInMin=0 labels=chef 2019-05-09 09:24:03.370+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: launch method called for agent ubuntu-test3c8d10 2019-05-09 09:24:05.920+0000 [id=1348] INFO c.m.a.v.AzureVMManagementServiceDelegate#isVMAliveOrHealthy: AzureVMManagementServiceDelegate: isVMAliveOrHealthy: status PowerState/running 2019-05-09 09:24:05.921+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#connectToSsh: AzureVMAgentSSHLauncher: connectToSsh: start 2019-05-09 09:24:05.921+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#getRemoteSession: AzureVMAgentSSHLauncher: getRemoteSession: getting remote session for user jenkins to host 10.76.96.77:22 2019-05-09 09:24:37.558+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#getRemoteSession: AzureVMAgentSSHLauncher: getRemoteSession: Got remote session for user jenkins to host 10.76.96.77:22 2019-05-09 09:24:37.559+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#connectToSsh: AzureVMAgentSSHLauncher: connectToSsh: Got remote connection 2019-05-09 09:24:37.559+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#executeRemoteCommand: AzureVMAgentSSHLauncher: executeRemoteCommand: starting test -e ~/.azure-agent-init 2019-05-09 09:24:38.811+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#executeRemoteCommand: AzureVMAgentSSHLauncher: executeRemoteCommand: executed successfully 2019-05-09 09:24:38.811+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: checking for java runtime 2019-05-09 09:24:38.811+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#executeRemoteCommand: AzureVMAgentSSHLauncher: executeRemoteCommand: starting java -fullversion 2019-05-09 09:24:40.152+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#executeRemoteCommand: AzureVMAgentSSHLauncher: executeRemoteCommand: executed successfully 2019-05-09 09:24:40.152+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: java runtime present, copying slave.jar to remote 2019-05-09 09:24:40.154+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#copyFileToRemote: AzureVMAgentSSHLauncher: copyFileToRemote: Initiating file transfer to slave.jar 2019-05-09 09:24:51.720+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#copyFileToRemote: AzureVMAgentSSHLauncher: copyFileToRemote: copied file Successfully to slave.jar 2019-05-09 09:24:51.721+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: launching agent: java -jar slave.jar 2019-05-09 09:24:51.742+0000 [id=1348] INFO c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: Connected successfully 2019-05-09 09:24:56.983+0000 [id=97] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Retrieving agent template with label chef2 2019-05-09 09:24:56.983+0000 [id=97] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template ubuntu-test 2019-05-09 09:24:56.983+0000 [id=97] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template test-jenkins-ubuntu 2019-05-09 09:24:56.983+0000 [id=97] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: test-jenkins-ubuntu matches! 2019-05-09 09:24:56.983+0000 [id=97] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Retrieving agent template with label azure-acos-atlasco 2019-05-09 09:24:56.983+0000 [id=97] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template ubuntu-test 2019-05-09 09:24:56.983+0000 [id=97] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template test-jenkins-ubuntu 2019-05-09 09:24:56.983+0000 [id=97] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template acostest 2019-05-09 09:24:56.983+0000 [id=97] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: acostest matches! 2019-05-09 09:24:56.983+0000 [id=97] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Retrieving agent template with label mdb-sandbox-acs 2019-05-09 09:24:56.983+0000 [id=97] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template ubuntu-test 2019-05-09 09:24:56.983+0000 [id=97] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template test-jenkins-ubuntu 2019-05-09 09:24:56.983+0000 [id=97] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template acostest 2019-05-09 09:25:05.878+0000 [id=1474] FINE hudson.model.AsyncPeriodicWork$1#run: Started Azure VM Maintainer Pool Size 2019-05-09 09:25:05.879+0000 [id=1474] FINE hudson.model.AsyncPeriodicWork$1#run: Finished Azure VM Maintainer Pool Size. 1 ms 2019-05-09 09:25:13.239+0000 [id=101] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Retrieving agent template with label mdb-sandbox-acs 2019-05-09 09:25:13.239+0000 [id=101] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template ubuntu-test 2019-05-09 09:25:13.239+0000 [id=101] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template test-jenkins-ubuntu 2019-05-09 09:25:13.239+0000 [id=101] FINE c.m.azure.vmagent.AzureVMCloud#getAzureAgentTemplate: AzureVMCloud: getAzureAgentTemplate: Found agent template acostest 2019-05-09 09:27:05.560+0000 [id=1812] FINE hudson.model.AsyncPeriodicWork$1#run: Started Azure VM Agents Clean Task 2019-05-09 09:27:05.561+0000 [id=1812] FINE c.m.a.v.AzureVMAgentCleanUpTask#execute: AzureVMAgentCleanUpTask: execute: start 2019-05-09 09:27:05.561+0000 [id=1812] FINE c.m.a.v.AzureVMAgentCleanUpTask#execute: AzureVMAgentCleanUpTask: execute: Running clean with 5 minute timeout 2019-05-09 09:27:05.561+0000 [id=1813] FINE c.m.a.v.AzureVMAgentCleanUpTask#cleanDeployments: AzureVMAgentCleanUpTask: cleanDeployments: Cleaning deployments 2019-05-09 09:27:05.561+0000 [id=1813] FINE c.m.a.v.AzureVMAgentCleanUpTask#cleanDeployments: AzureVMAgentCleanUpTask: cleanDeployments: Checking deployment ubuntu-test0509110640988 2019-05-09 09:27:07.711+0000 [id=1813] FINE c.m.a.v.AzureVMAgentCleanUpTask#cleanDeployments: AzureVMAgentCleanUpTask: cleanDeployments: Deployment created on 5/9/19 11:08 AM 2019-05-09 09:27:07.711+0000 [id=1813] FINE c.m.a.v.AzureVMAgentCleanUpTask#cleanDeployments: AzureVMAgentCleanUpTask: cleanDeployments: Deployment newer than timeout, keeping 2019-05-09 09:27:07.712+0000 [id=1813] FINE c.m.a.v.AzureVMAgentCleanUpTask#cleanDeployments: AzureVMAgentCleanUpTask: cleanDeployments: Done cleaning deployments 2019-05-09 09:27:07.800+0000 [id=1812] FINE c.m.a.v.AzureVMAgentCleanUpTask#execute: AzureVMAgentCleanUpTask: execute: end 2019-05-09 09:27:07.800+0000 [id=1812] FINE hudson.model.AsyncPeriodicWork$1#run: Finished Azure VM Agents Clean Task. 2,240 ms 2019-05-09 09:28:46.078+0000 [id=101] INFO c.m.a.v.AzureVMCloudRetensionStrategy#check: AzureVMCloudRetensionStrategy: check: Idle timeout reached for agent: ubuntu-test3c8d10, action: shutdown 2019-05-09 09:28:46.079+0000 [id=1967] INFO c.m.a.v.AzureVMCloudRetensionStrategy$1#call: AzureVMCloudRetensionStrategy: going to idleTimeout