[JENKINS-54776] Node stays suspended after node is started

Type: Bug
Resolution: Duplicate
Priority: Major
Component/s: _unsorted
Labels:
- azure-jenkins
Environment:
Jenkins ver. 2.138.3 with Azure VM Agents Plugin ver. 0.7.4

Similar Issues:
Powered by SuggestiMate

Show

We are currently using the Azure VM agents plugin for our CI/CD environment, we use custom images.

Our images are one of:

Image 1
- Linux (With SSH launch)
- Idle retention time (10 min)
- Shutdown Only enabled
Image 2
- Windows (With JNLP launch)
- Idle retention time (20 min)
- Shutdown Only enabled

Our issue is that we have an agent that successfully gets provisioned launched and is usable, after the idle timeout the node successfully shuts down and applies the (offline) and (suspended) flags to it correctly. Once another build comes along and attempts to use it, the plugin correctly launches the previously shutdown VM it starts correctly, connects correctly but it maintains the (suspended) flag making it unusable. I can manually fix it by issuing the following command in the Jenkins script console

Jenkins.instance.getNode('azure-nodeXXXXXX').toComputer().setAcceptingTasks(true);

Then the build goes to the node and builds successfully, however if that command is not issued the build gets stuck waiting for the node to come online.

This issue happens with both types of images.

Daniel McAssey created issue - 2018-11-22 15:40

Jie Shen made changes - 2018-11-23 01:18

Assignee

Original: Azure DevOps [ azure_devops ]

New: Jie Shen [ jieshe ]

Jie Shen added a comment - 2018-11-23 01:19

Thanks for your reporting, I will investigate this issue.

Jie Shen added a comment - 2018-11-23 01:19 Thanks for your reporting, I will investigate this issue.

Jie Shen made changes - 2018-11-29 06:25

Status

Original: Open [ 1 ]

New: In Progress [ 3 ]

Niels Oke Schmäschke added a comment - 2018-12-03 14:05

Hi jieshe,
first up thanks for the work on the Azure VM Agents Plugin, we are enjoying using this plugin in our build environment.
We are also seeing the same issue during our CI/CD builds.

The Cloud configuration we are using looks essentially like:

new AzureVMCloudBuilder()
    .withCloudName("our Cloud Name")
    .withMaxVirtualMachinesLimit("3")
    .withDeploymentTimeout("7200")
    .addNewTemplate()
        .withName("our-name")
        .withDescription("Our Description")
        .withLabels("our-label")
        .withVirtualMachineSize("Standard_D4s_v3")
        .withStorageAccountType("Premium_LRS")
        .withExistingStorageAccount("uniquestorageaccountname")
        .withRetentionStrategy(new AzureVMCloudRetensionStrategy(30))
        .withShutdownOnIdle(true)
        .withWorkspace("d:\\")
        .addNewAdvancedImage()
            .withReferenceImage("MicrosoftWindowsDesktop", "Windows-10", "RS4-ProN", "latest")
            .withOsType("Windows")
            .withLaunchMethod("SSH")
            .withPreInstallSsh(true)
            .withRunScriptAsRoot(false)
            .withNumberOfExecutors("1")
            .withInitScript("Write-Host 'custom init script'")
            .withDoNotUseMachineIfInitFails(true)
            .withVirtualNetworkName("our-network")
            .withVirtualNetworkResourceGroupName("network-resource-group")
            .withSubnetName("subnet-name")
            .withUsePrivateIP(true)
            .withNetworkSecurityGroupName("")
            .withJvmOptions("")
            .withDisableTemplate(false)
        .endAdvancedImage()
        .withUsageMode(Node.Mode.EXCLUSIVE.getDescription())
        .withLocation("West Europe")
        .withDiskType(Constants.DISK_MANAGED)
        .withAdminCredential("cred-id")
    .endTemplate()
    .withAzureCredentialsId("azure-cred-id")
    .withExistingResourceGroupName("resource-group")
    .build();

But as reported I guess the crucial parameter for the issue is

.withShutdownOnIdle(true)

Niels Oke Schmäschke added a comment - 2018-12-03 14:05 Hi jieshe , first up thanks for the work on the Azure VM Agents Plugin, we are enjoying using this plugin in our build environment. We are also seeing the same issue during our CI/CD builds. The Cloud configuration we are using looks essentially like: new AzureVMCloudBuilder() .withCloudName( "our Cloud Name" ) .withMaxVirtualMachinesLimit( "3" ) .withDeploymentTimeout( "7200" ) .addNewTemplate() .withName( "our-name" ) .withDescription( "Our Description" ) .withLabels( "our-label" ) .withVirtualMachineSize( "Standard_D4s_v3" ) .withStorageAccountType( "Premium_LRS" ) .withExistingStorageAccount( "uniquestorageaccountname" ) .withRetentionStrategy( new AzureVMCloudRetensionStrategy(30)) .withShutdownOnIdle( true ) .withWorkspace( "d:\\" ) .addNewAdvancedImage() .withReferenceImage( "MicrosoftWindowsDesktop" , "Windows-10" , "RS4-ProN" , "latest" ) .withOsType( "Windows" ) .withLaunchMethod( "SSH" ) .withPreInstallSsh( true ) .withRunScriptAsRoot( false ) .withNumberOfExecutors( "1" ) .withInitScript( "Write-Host 'custom init script' " ) .withDoNotUseMachineIfInitFails( true ) .withVirtualNetworkName( "our-network" ) .withVirtualNetworkResourceGroupName( "network-resource-group" ) .withSubnetName( "subnet-name" ) .withUsePrivateIP( true ) .withNetworkSecurityGroupName("") .withJvmOptions("") .withDisableTemplate( false ) .endAdvancedImage() .withUsageMode(Node.Mode.EXCLUSIVE.getDescription()) .withLocation( "West Europe" ) .withDiskType(Constants.DISK_MANAGED) .withAdminCredential( "cred-id" ) .endTemplate() .withAzureCredentialsId( "azure-cred-id" ) .withExistingResourceGroupName( "resource-group" ) .build(); But as reported I guess the crucial parameter for the issue is .withShutdownOnIdle( true )

Jie Shen added a comment - 2018-12-04 01:51

niels_oke Thanks for providing your configuration. I was unable to reproduce this issue in my environment before. I will use your configuration to check whether I could get any luck.

Jie Shen added a comment - 2018-12-04 01:51 niels_oke Thanks for providing your configuration. I was unable to reproduce this issue in my environment before. I will use your configuration to check whether I could get any luck.

Jie Shen added a comment - 2018-12-14 08:07

Successfully reproduce this issue, will create a PR for soon.

Jie Shen added a comment - 2018-12-14 08:07 Successfully reproduce this issue, will create a PR for soon.

Balaal Ashraf added a comment - 2019-01-10 14:18

Hello, sorry to bother about this but i also have the same issue.. Hope you are close to resolving it and raising PR

Looking at the logs and the code i have noticed it goes into AzureVMManagementServiceDelegate : startVirtualMachine : <image>.

Within the code this is called within line 699 of AzureVMCloud, with azureComputer.setAcceptingTasks(true); being called shortly after (713), but stays in suspended state as mentioned above..

Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloud
AzureVMCloud: getAzureAgentTemplate: Retrieving agent template with label azureFresh
Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloud
AzureVMCloud: getAzureAgentTemplate: Found agent template vimbuild
Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloud
AzureVMCloud: getAzureAgentTemplate: vimbuild matches!
Jan 10, 2019 11:59:57 AM INFO com.microsoft.azure.vmagent.AzureVMCloud provision
AzureVMCloud: provision: start for label azureFresh workLoad 1
Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloud
AzureVMCloud: getAzureAgentTemplate: Retrieving agent template with label azureFresh
Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloud
AzureVMCloud: getAzureAgentTemplate: Found agent template vimbuild
Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloud
AzureVMCloud: getAzureAgentTemplate: vimbuild matches!
Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloudVerificationTask
AzureVMCloudVerificationTask: verify: verifying cloud vimBuild-0.7.6-snapshot
Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloudVerificationTask
AzureVMCloudVerificationTask: verifyConfiguration: start
Jan 10, 2019 11:59:58 AM FINE com.microsoft.azure.vmagent.AzureVMCloudVerificationTask
AzureVMCloudVerificationTask: validate: vimBuild-0.7.6-snapshot verified pass
Jan 10, 2019 11:59:58 AM FINE com.microsoft.azure.vmagent.AzureVMCloudVerificationTask
AzureVMCloudVerificationTask: getVirtualMachineCount: start
Jan 10, 2019 11:59:58 AM FINE com.microsoft.azure.vmagent.AzureVMCloudVerificationTask
AzureVMCloudVerificationTask: getVirtualMachineCount: end, cloud vimBuild-0.7.6-snapshot has currently 1 vms
Jan 10, 2019 11:59:59 AM FINE com.microsoft.azure.vmagent.AzureVMCloudVerificationTask
AzureVMCloudVerificationTask: verify: vimbuild verified successfully
Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMCloud provision
AzureVMCloud: provision: checking for node reuse options
Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMCloud provision
AzureVMCloud: provision: agent computer eligible for reuse vimbuild6a1390
Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
AzureVMManagementServiceDelegate: virtualMachineExists: check for vimbuild6a1390
Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists
AzureVMManagementServiceDelegate: virtualMachineExists: vimbuild6a1390 exists
Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMCloud$2 call
Found existing node, starting VM vimbuild6a1390
Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate startVirtualMachine
AzureVMManagementServiceDelegate: startVirtualMachine: vimbuild6a1390
Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMCloud provision
AzureVMCloud: provision: asynchronous provision finished, returning 1 planned node(s)
Jan 10, 2019 12:02:02 PM FINE com.microsoft.azure.vmagent.AzureVMMaintainPoolTask
Started Azure VM Maintainer Pool Size
Jan 10, 2019 12:02:02 PM FINE com.microsoft.azure.vmagent.AzureVMMaintainPoolTask
Finished Azure VM Maintainer Pool Size. 0 ms
Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Started Azure VM Agents Clean Task
Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
AzureVMAgentCleanUpTask: execute: start
Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
AzureVMAgentCleanUpTask: execute: Running clean with 5 minute timeout
Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
AzureVMAgentCleanUpTask: cleanVMs: node vimbuild6a1390 blocked to cleanup
Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
AzureVMAgentCleanUpTask: cleanDeployments: Cleaning deployments
Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
AzureVMAgentCleanUpTask: cleanDeployments: Checking deployment vimbuild0110113128980
Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
AzureVMAgentCleanUpTask: cleanDeployments: Deployment created on 1/10/19 11:36 AM
Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
AzureVMAgentCleanUpTask: cleanDeployments: Deployment newer than timeout, keeping
Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
AzureVMAgentCleanUpTask: cleanDeployments: Done cleaning deployments
Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
AzureVMAgentCleanUpTask: execute: end
Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask
Finished Azure VM Agents Clean Task. 219 ms
Jan 10, 2019 12:02:09 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate setVirtualMachineDetails
The Azure agent doesn't have a public IP. Will use the private IP
Jan 10, 2019 12:02:09 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate setVirtualMachineDetails
Azure agent details:
nodeNamevimbuild6a1390
adminUserName=localadmin
shutdownOnIdle=true
retentionTimeInMin=0
labels=azureFresh
Jan 10, 2019 12:02:09 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch
AzureVMAgentSSHLauncher: launch: launch method called for agent vimbuild6a1390
Jan 10, 2019 12:02:09 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate isVMAliveOrHealthy
AzureVMManagementServiceDelegate: isVMAliveOrHealthy: status PowerState/running
Jan 10, 2019 12:02:09 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher connectToSsh
AzureVMAgentSSHLauncher: connectToSsh: start
Jan 10, 2019 12:02:09 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher getRemoteSession
AzureVMAgentSSHLauncher: getRemoteSession: getting remote session for user localadmin to host 10.228.4.26:22
Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher getRemoteSession
AzureVMAgentSSHLauncher: getRemoteSession: Got remote session for user localadmin to host 10.228.4.26:22
Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher connectToSsh
AzureVMAgentSSHLauncher: connectToSsh: Got remote connection
Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand
AzureVMAgentSSHLauncher: executeRemoteCommand: starting dir C:\.azure-agent-init
Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand
AzureVMAgentSSHLauncher: executeRemoteCommand: executed successfully
Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch
AzureVMAgentSSHLauncher: launch: checking for java runtime
Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand
AzureVMAgentSSHLauncher: executeRemoteCommand: starting java -fullversion
Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand
AzureVMAgentSSHLauncher: executeRemoteCommand: executed successfully
Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch
AzureVMAgentSSHLauncher: launch: java runtime present, copying slave.jar to remote
Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher copyFileToRemote
AzureVMAgentSSHLauncher: copyFileToRemote: Initiating file transfer to slave.jar
Jan 10, 2019 12:02:20 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher copyFileToRemote
AzureVMAgentSSHLauncher: copyFileToRemote: copied file Successfully to slave.jar
Jan 10, 2019 12:02:20 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch
AzureVMAgentSSHLauncher: launch: launching agent: java -jar slave.jar
Jan 10, 2019 12:02:20 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch
AzureVMAgentSSHLauncher: launch: Connected successfully

Balaal Ashraf added a comment - 2019-01-10 14:18 Hello, sorry to bother about this but i also have the same issue.. Hope you are close to resolving it and raising PR Looking at the logs and the code i have noticed it goes into AzureVMManagementServiceDelegate : startVirtualMachine : <image>. Within the code this is called within line 699 of AzureVMCloud, with azureComputer.setAcceptingTasks(true); being called shortly after (713), but stays in suspended state as mentioned above.. Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloud AzureVMCloud: getAzureAgentTemplate: Retrieving agent template with label azureFresh Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloud AzureVMCloud: getAzureAgentTemplate: Found agent template vimbuild Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloud AzureVMCloud: getAzureAgentTemplate: vimbuild matches! Jan 10, 2019 11:59:57 AM INFO com.microsoft.azure.vmagent.AzureVMCloud provision AzureVMCloud: provision: start for label azureFresh workLoad 1 Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloud AzureVMCloud: getAzureAgentTemplate: Retrieving agent template with label azureFresh Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloud AzureVMCloud: getAzureAgentTemplate: Found agent template vimbuild Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloud AzureVMCloud: getAzureAgentTemplate: vimbuild matches! Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloudVerificationTask AzureVMCloudVerificationTask: verify: verifying cloud vimBuild-0.7.6-snapshot Jan 10, 2019 11:59:57 AM FINE com.microsoft.azure.vmagent.AzureVMCloudVerificationTask AzureVMCloudVerificationTask: verifyConfiguration: start Jan 10, 2019 11:59:58 AM FINE com.microsoft.azure.vmagent.AzureVMCloudVerificationTask AzureVMCloudVerificationTask: validate: vimBuild-0.7.6-snapshot verified pass Jan 10, 2019 11:59:58 AM FINE com.microsoft.azure.vmagent.AzureVMCloudVerificationTask AzureVMCloudVerificationTask: getVirtualMachineCount: start Jan 10, 2019 11:59:58 AM FINE com.microsoft.azure.vmagent.AzureVMCloudVerificationTask AzureVMCloudVerificationTask: getVirtualMachineCount: end, cloud vimBuild-0.7.6-snapshot has currently 1 vms Jan 10, 2019 11:59:59 AM FINE com.microsoft.azure.vmagent.AzureVMCloudVerificationTask AzureVMCloudVerificationTask: verify: vimbuild verified successfully Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMCloud provision AzureVMCloud: provision: checking for node reuse options Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMCloud provision AzureVMCloud: provision: agent computer eligible for reuse vimbuild6a1390 Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists AzureVMManagementServiceDelegate: virtualMachineExists: check for vimbuild6a1390 Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate virtualMachineExists AzureVMManagementServiceDelegate: virtualMachineExists: vimbuild6a1390 exists Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMCloud$2 call Found existing node, starting VM vimbuild6a1390 Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate startVirtualMachine AzureVMManagementServiceDelegate: startVirtualMachine: vimbuild6a1390 Jan 10, 2019 11:59:59 AM INFO com.microsoft.azure.vmagent.AzureVMCloud provision AzureVMCloud: provision: asynchronous provision finished, returning 1 planned node(s) Jan 10, 2019 12:02:02 PM FINE com.microsoft.azure.vmagent.AzureVMMaintainPoolTask Started Azure VM Maintainer Pool Size Jan 10, 2019 12:02:02 PM FINE com.microsoft.azure.vmagent.AzureVMMaintainPoolTask Finished Azure VM Maintainer Pool Size. 0 ms Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask Started Azure VM Agents Clean Task Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask AzureVMAgentCleanUpTask: execute: start Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask AzureVMAgentCleanUpTask: execute: Running clean with 5 minute timeout Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask AzureVMAgentCleanUpTask: cleanVMs: node vimbuild6a1390 blocked to cleanup Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask AzureVMAgentCleanUpTask: cleanDeployments: Cleaning deployments Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask AzureVMAgentCleanUpTask: cleanDeployments: Checking deployment vimbuild0110113128980 Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask AzureVMAgentCleanUpTask: cleanDeployments: Deployment created on 1/10/19 11:36 AM Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask AzureVMAgentCleanUpTask: cleanDeployments: Deployment newer than timeout, keeping Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask AzureVMAgentCleanUpTask: cleanDeployments: Done cleaning deployments Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask AzureVMAgentCleanUpTask: execute: end Jan 10, 2019 12:02:07 PM FINE com.microsoft.azure.vmagent.AzureVMAgentCleanUpTask Finished Azure VM Agents Clean Task. 219 ms Jan 10, 2019 12:02:09 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate setVirtualMachineDetails The Azure agent doesn't have a public IP. Will use the private IP Jan 10, 2019 12:02:09 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate setVirtualMachineDetails Azure agent details: nodeNamevimbuild6a1390 adminUserName=localadmin shutdownOnIdle= true retentionTimeInMin=0 labels=azureFresh Jan 10, 2019 12:02:09 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch AzureVMAgentSSHLauncher: launch: launch method called for agent vimbuild6a1390 Jan 10, 2019 12:02:09 PM INFO com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate isVMAliveOrHealthy AzureVMManagementServiceDelegate: isVMAliveOrHealthy: status PowerState/running Jan 10, 2019 12:02:09 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher connectToSsh AzureVMAgentSSHLauncher: connectToSsh: start Jan 10, 2019 12:02:09 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher getRemoteSession AzureVMAgentSSHLauncher: getRemoteSession: getting remote session for user localadmin to host 10.228.4.26:22 Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher getRemoteSession AzureVMAgentSSHLauncher: getRemoteSession: Got remote session for user localadmin to host 10.228.4.26:22 Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher connectToSsh AzureVMAgentSSHLauncher: connectToSsh: Got remote connection Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand AzureVMAgentSSHLauncher: executeRemoteCommand: starting dir C:\.azure-agent-init Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand AzureVMAgentSSHLauncher: executeRemoteCommand: executed successfully Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch AzureVMAgentSSHLauncher: launch: checking for java runtime Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand AzureVMAgentSSHLauncher: executeRemoteCommand: starting java -fullversion Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher executeRemoteCommand AzureVMAgentSSHLauncher: executeRemoteCommand: executed successfully Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch AzureVMAgentSSHLauncher: launch: java runtime present, copying slave.jar to remote Jan 10, 2019 12:02:10 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher copyFileToRemote AzureVMAgentSSHLauncher: copyFileToRemote: Initiating file transfer to slave.jar Jan 10, 2019 12:02:20 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher copyFileToRemote AzureVMAgentSSHLauncher: copyFileToRemote: copied file Successfully to slave.jar Jan 10, 2019 12:02:20 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch AzureVMAgentSSHLauncher: launch: launching agent: java -jar slave.jar Jan 10, 2019 12:02:20 PM INFO com.microsoft.azure.vmagent.remote.AzureVMAgentSSHLauncher launch AzureVMAgentSSHLauncher: launch: Connected successfully

Jie Shen added a comment - 2019-01-11 02:50

Hi balaal, you are right. This issue is caused by the codes. https://github.com/jenkinsci/azure-vm-agents-plugin/blob/ea8dc0b19184c381172afac0a41cac46435eaaf4/src/main/java/com/microsoft/azure/vmagent/AzureVMCloud.java#L707 may throw some exceptions in some cases and then skip the azureComputer.setAcceptingTasks(true). I made some changes to just catch and ignore the exceptions. But it seems not a proper way to fix it. I need to find out why the exceptions are thrown and how to avoid it.

Jie Shen added a comment - 2019-01-11 02:50 Hi balaal , you are right. This issue is caused by the codes. https://github.com/jenkinsci/azure-vm-agents-plugin/blob/ea8dc0b19184c381172afac0a41cac46435eaaf4/src/main/java/com/microsoft/azure/vmagent/AzureVMCloud.java#L707 may throw some exceptions in some cases and then skip the azureComputer.setAcceptingTasks(true). I made some changes to just catch and ignore the exceptions. But it seems not a proper way to fix it. I need to find out why the exceptions are thrown and how to avoid it.

Sander Bel added a comment - 2019-02-18 14:50 - edited

We are experiencing similar issues, but unlike the OP, executing:

Jenkins.instance.getNode('azure-nodeXXXXXX').toComputer().setAcceptingTasks(true);

does not solve anything.

When the node gets shutdown and de-allocated, it gets started again when a build with its label starts. The build however does not acquire the node. When we execute the statement above, we can see that the node gets acquired, but nothing more happens, the build does not continue.

Any help would be greatly appreciated! We are keeping the nodes online for now (idle timeout set to basically infinite), not shutting them down, make it a rather expensive solution for the moment...

EDIT: if any extra logging/debugging is needed, let me know!

Sander Bel added a comment - 2019-02-18 14:50 - edited We are experiencing similar issues, but unlike the OP, executing: Jenkins.instance.getNode( 'azure-nodeXXXXXX' ).toComputer().setAcceptingTasks( true ); does not solve anything. When the node gets shutdown and de-allocated, it gets started again when a build with its label starts. The build however does not acquire the node. When we execute the statement above, we can see that the node gets acquired, but nothing more happens, the build does not continue. Any help would be greatly appreciated! We are keeping the nodes online for now (idle timeout set to basically infinite), not shutting them down, make it a rather expensive solution for the moment... EDIT: if any extra logging/debugging is needed, let me know!

Assignee:: Jie Shen

Reporter:: Daniel McAssey

Votes:: 9 Vote for this issue

Watchers:: 15 Start watching this issue

Created:: 2018-11-22 15:40

Updated:: 2021-03-28 08:26

Resolved:: 2021-03-28 08:25

Jenkins

Details

Description

Attachments

Activity

Collapse comment: Jie Shen added a comment - 2018-11-23 01:19

Expand comment: Jie Shen added a comment - 2018-11-23 01:19

Collapse comment: Niels Oke Schmäschke added a comment - 2018-12-03 14:05

Expand comment: Niels Oke Schmäschke added a comment - 2018-12-03 14:05

Collapse comment: Jie Shen added a comment - 2018-12-04 01:51

Expand comment: Jie Shen added a comment - 2018-12-04 01:51

Collapse comment: Jie Shen added a comment - 2018-12-14 08:07

Expand comment: Jie Shen added a comment - 2018-12-14 08:07

Collapse comment: Balaal Ashraf added a comment - 2019-01-10 14:18

Expand comment: Balaal Ashraf added a comment - 2019-01-10 14:18

Collapse comment: Jie Shen added a comment - 2019-01-11 02:50

Expand comment: Jie Shen added a comment - 2019-01-11 02:50

Collapse comment: Sander Bel added a comment - 2019-02-18 14:50, Edited by Sander Bel - 2019-02-18 14:52

Expand comment: Sander Bel added a comment - 2019-02-18 14:50, Edited by Sander Bel - 2019-02-18 14:52

People

Dates