Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64225

shutdownOnIdle nodes failed to provision after Jenkins restart

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Minor
    • Resolution: Duplicate
    • Component/s: _unsorted
    • Labels:
      None
    • Environment:
      Jenkins 2.249.3
      azure-vm-agents 1.5.1
    • Similar Issues:

      Description

      Nodes configured for AzureVMCloudRetensionStrategy with shutdownOnIdle enabled.
      Nodes are suspended/shutdown due to idle.
      Upon Jenkins restart nodes cannot be brought back up due to NullPointerException at AzureVMCloud.getLockForAgent(AzureVMCloud.java:1017)

      2020-11-14 18:55:55.556+0000 [id=1]     INFO    o.e.j.s.handler.ContextHandler#doStart: Started w.@20bd8be5{Jenkins v2.249.3,/,file:///var/jenkins_home/war/,AVAILABLE}{/var/jenkins_home/war}
      [...]
      2020-11-14 18:58:52.103+0000 [id=27]    INFO    c.m.azure.vmagent.AzureVMAgent#createComputer: AzureVMAgent: createComputer: start for agent vs2019-b0ccdf0
      2020-11-14 18:58:52.190+0000 [id=27]    INFO    c.m.a.v.AzureVMCloudRetensionStrategy#start: AzureVMCloudRetensionStrategy: start: azureComputer name vs2019-b0ccdf0
      2020-11-14 18:58:52.193+0000 [id=27]    INFO    c.m.azure.vmagent.AzureVMAgent#createComputer: AzureVMAgent: createComputer: start for agent vs2019-b5ac440
      2020-11-14 18:58:52.194+0000 [id=85]    INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: Add suspended status for node vs2019-b0ccdf0
      2020-11-14 18:58:52.197+0000 [id=85]    INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: shutting down agent vs2019-b0ccdf0
      2020-11-14 18:58:52.201+0000 [id=27]    INFO    c.m.a.v.AzureVMCloudRetensionStrategy#start: AzureVMCloudRetensionStrategy: start: azureComputer name vs2019-b5ac440
      2020-11-14 18:58:52.202+0000 [id=27]    INFO    c.m.azure.vmagent.AzureVMAgent#createComputer: AzureVMAgent: createComputer: start for agent vs2019-bdba010
      2020-11-14 18:58:52.208+0000 [id=88]    INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: Add suspended status for node vs2019-b5ac440
      2020-11-14 18:58:52.208+0000 [id=27]    INFO    c.m.a.v.AzureVMCloudRetensionStrategy#start: AzureVMCloudRetensionStrategy: start: azureComputer name vs2019-bdba010
      2020-11-14 18:58:52.220+0000 [id=27]    INFO    c.m.azure.vmagent.AzureVMAgent#createComputer: AzureVMAgent: createComputer: start for agent vs2019-bdba011
      2020-11-14 18:58:52.221+0000 [id=91]    INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: Add suspended status for node vs2019-bdba010
      2020-11-14 18:58:52.225+0000 [id=88]    INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: shutting down agent vs2019-b5ac440
      2020-11-14 18:58:52.226+0000 [id=91]    INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: shutting down agent vs2019-bdba010
      2020-11-14 18:58:52.226+0000 [id=27]    INFO    c.m.a.v.AzureVMCloudRetensionStrategy#start: AzureVMCloudRetensionStrategy: start: azureComputer name vs2019-bdba011
      2020-11-14 18:58:52.228+0000 [id=27]    INFO    c.m.azure.vmagent.AzureVMAgent#createComputer: AzureVMAgent: createComputer: start for agent vs2019-bee4040
      2020-11-14 18:58:52.229+0000 [id=96]    INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: Add suspended status for node vs2019-bdba011
      2020-11-14 18:58:52.234+0000 [id=87]    INFO    c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: launch method called for agent vs2019-b5ac440
      2020-11-14 18:58:52.236+0000 [id=96]    INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: shutting down agent vs2019-bdba011
      2020-11-14 18:58:52.232+0000 [id=84]    INFO    c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: launch method called for agent vs2019-b0ccdf0
      2020-11-14 18:58:52.235+0000 [id=27]    INFO    c.m.a.v.AzureVMCloudRetensionStrategy#start: AzureVMCloudRetensionStrategy: start: azureComputer name vs2019-bee4040
      2020-11-14 18:58:52.239+0000 [id=27]    INFO    c.m.azure.vmagent.AzureVMAgent#createComputer: AzureVMAgent: createComputer: start for agent vs2019-bee4041
      2020-11-14 18:58:52.238+0000 [id=90]    INFO    c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: launch method called for agent vs2019-bdba010
      2020-11-14 18:58:52.241+0000 [id=93]    INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: Add suspended status for node vs2019-bee4040
      2020-11-14 18:58:52.243+0000 [id=93]    INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: shutting down agent vs2019-bee4040
      2020-11-14 18:58:52.245+0000 [id=27]    INFO    c.m.a.v.AzureVMCloudRetensionStrategy#start: AzureVMCloudRetensionStrategy: start: azureComputer name vs2019-bee4041
      2020-11-14 18:58:52.247+0000 [id=27]    INFO    c.m.azure.vmagent.AzureVMAgent#createComputer: AzureVMAgent: createComputer: start for agent vs2019-bee4042
      2020-11-14 18:58:52.249+0000 [id=102]   INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: Add suspended status for node vs2019-bee4041
      2020-11-14 18:58:52.252+0000 [id=102]   INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: shutting down agent vs2019-bee4041
      2020-11-14 18:58:52.253+0000 [id=27]    INFO    c.m.a.v.AzureVMCloudRetensionStrategy#start: AzureVMCloudRetensionStrategy: start: azureComputer name vs2019-bee4042
      2020-11-14 18:58:52.253+0000 [id=95]    INFO    c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: launch method called for agent vs2019-bdba011
      2020-11-14 18:58:52.260+0000 [id=107]   INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: Add suspended status for node vs2019-bee4042
      2020-11-14 18:58:52.261+0000 [id=107]   INFO    c.m.azure.vmagent.AzureVMAgent#shutdown: AzureVMAgent: shutdown: shutting down agent vs2019-bee4042
      [...]
      2020-11-14 18:58:55.589+0000 [id=85]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#shutdownVirtualMachine: AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for vs2019-b0ccdf0
      2020-11-14 18:58:55.592+0000 [id=107]   INFO    c.m.a.v.AzureVMManagementServiceDelegate#shutdownVirtualMachine: AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for vs2019-bee4042
      2020-11-14 18:58:55.592+0000 [id=102]   INFO    c.m.a.v.AzureVMManagementServiceDelegate#shutdownVirtualMachine: AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for vs2019-bee4041
      2020-11-14 18:58:55.593+0000 [id=88]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#shutdownVirtualMachine: AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for vs2019-b5ac440
      2020-11-14 18:58:55.593+0000 [id=91]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#shutdownVirtualMachine: AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for vs2019-bdba010
      2020-11-14 18:58:55.593+0000 [id=96]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#shutdownVirtualMachine: AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for vs2019-bdba011
      2020-11-14 18:58:55.593+0000 [id=93]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#shutdownVirtualMachine: AzureVMManagementServiceDelegate: shutdownVirtualMachine: called for vs2019-bee4040
      [...]
      2020-11-14 18:58:57.155+0000 [id=94]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#isVMAliveOrHealthy: AzureVMManagementServiceDelegate: isVMAliveOrHealthy: status PowerState/deallocated
      2020-11-14 18:58:57.156+0000 [id=94]    INFO    c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: Agent vs2019-bee4040 is shut down, deleted, etc. Not attempting to connect
      2020-11-14 18:58:57.156+0000 [id=106]   INFO    c.m.a.v.AzureVMManagementServiceDelegate#isVMAliveOrHealthy: AzureVMManagementServiceDelegate: isVMAliveOrHealthy: status PowerState/deallocated
      2020-11-14 18:58:57.156+0000 [id=87]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#isVMAliveOrHealthy: AzureVMManagementServiceDelegate: isVMAliveOrHealthy: status PowerState/deallocated
      2020-11-14 18:58:57.157+0000 [id=87]    INFO    c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: Agent vs2019-b5ac440 is shut down, deleted, etc. Not attempting to connect
      [...]
      2020-11-14 18:58:57.169+0000 [id=106]   INFO    c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: Agent vs2019-bee4042 is shut down, deleted, etc. Not attempting to connect
      2020-11-14 18:58:57.169+0000 [id=84]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#isVMAliveOrHealthy: AzureVMManagementServiceDelegate: isVMAliveOrHealthy: status PowerState/deallocated
      2020-11-14 18:58:57.169+0000 [id=84]    INFO    c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: Agent vs2019-b0ccdf0 is shut down, deleted, etc. Not attempting to connect
      2020-11-14 18:58:57.174+0000 [id=90]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#isVMAliveOrHealthy: AzureVMManagementServiceDelegate: isVMAliveOrHealthy: status PowerState/deallocated
      2020-11-14 18:58:57.174+0000 [id=90]    INFO    c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: Agent vs2019-bdba010 is shut down, deleted, etc. Not attempting to connect
      [...]
      2020-11-14 18:58:57.251+0000 [id=95]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#isVMAliveOrHealthy: AzureVMManagementServiceDelegate: isVMAliveOrHealthy: status PowerState/deallocated
      2020-11-14 18:58:57.255+0000 [id=95]    INFO    c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: Agent vs2019-bdba011 is shut down, deleted, etc. Not attempting to connect
      2020-11-14 18:58:57.266+0000 [id=101]   INFO    c.m.a.v.AzureVMManagementServiceDelegate#isVMAliveOrHealthy: AzureVMManagementServiceDelegate: isVMAliveOrHealthy: status PowerState/deallocated
      2020-11-14 18:58:57.267+0000 [id=101]   INFO    c.m.a.v.r.AzureVMAgentSSHLauncher#launch: AzureVMAgentSSHLauncher: launch: Agent vs2019-bee4041 is shut down, deleted, etc. Not attempting to connect
      [...]
      2020-11-14 18:59:00.005+0000 [id=19]    INFO    hudson.WebAppMain$3#run: Jenkins is fully up and running
      [...]
      2020-11-14 19:09:51.497+0000 [id=351]   INFO    o.j.plugins.ghprb.GhprbTrigger#start: Starting the ghprb trigger for the Windows/build job; newInstance is false
      [...]
      2020-11-14 19:10:19.860+0000 [id=28]    INFO    c.m.azure.vmagent.AzureVMCloud#provision: AzureVMCloud: provision: checking for node reuse options
      2020-11-14 19:10:19.860+0000 [id=28]    INFO    c.m.azure.vmagent.AzureVMCloud#provision: AzureVMCloud: provision: agent computer eligible for reuse vs2019-b0ccdf0
      2020-11-14 19:10:19.860+0000 [id=28]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#virtualMachineExists: AzureVMManagementServiceDelegate: virtualMachineExists: check for vs2019-b0ccdf0
      2020-11-14 19:10:19.930+0000 [id=28]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#virtualMachineExists: AzureVMManagementServiceDelegate: virtualMachineExists: vs2019-b0ccdf0 exists
      2020-11-14 19:10:19.932+0000 [id=28]    INFO    c.m.azure.vmagent.AzureVMCloud#provision: AzureVMCloud: provision: asynchronous provision finished, returning 1 planned node(s)
      2020-11-14 19:10:20.168+0000 [id=28]    INFO    o.j.p.cloudstats.CloudStatistics#logTypeNotSupported: No support for cloud-stats plugin by class hudson.slaves.NodeProvisioner$PlannedNode
      2020-11-14 19:10:28.605+0000 [id=37]    WARNING hudson.slaves.NodeProvisioner#lambda$update$6: Unexpected exception encountered while provisioning agent vs2019-b0ccdf0
      java.lang.NullPointerException
              at com.microsoft.azure.vmagent.AzureVMCloud.getLockForAgent(AzureVMCloud.java:1017)
              at com.microsoft.azure.vmagent.AzureVMCloud.lambda$provision$1(AzureVMCloud.java:682)
              at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
              at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      2020-11-14 19:10:28.607+0000 [id=37]    INFO    c.a.j.e.NoDelayProvisionStrategy#apply: currentDemand 3 availableCapacity 0 (availableExecutors 0 connectingExecutors 0 plannedCapacitySnapshot 0 additionalPlannedCapacity 0)
      2020-11-14 19:10:28.618+0000 [id=37]    INFO    c.m.azure.vmagent.AzureVMCloud#provision: AzureVMCloud: provision: start for label windows-vs2019 workLoad 3
      2020-11-14 19:10:28.619+0000 [id=37]    INFO    c.m.azure.vmagent.AzureVMCloud#provision: AzureVMCloud: provision: checking for node reuse options
      2020-11-14 19:10:28.620+0000 [id=37]    INFO    c.m.azure.vmagent.AzureVMCloud#provision: AzureVMCloud: provision: agent computer eligible for reuse vs2019-b0ccdf0
      2020-11-14 19:10:28.620+0000 [id=37]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#virtualMachineExists: AzureVMManagementServiceDelegate: virtualMachineExists: check for vs2019-b0ccdf0
      2020-11-14 19:10:28.690+0000 [id=37]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#virtualMachineExists: AzureVMManagementServiceDelegate: virtualMachineExists: vs2019-b0ccdf0 exists
      2020-11-14 19:10:28.690+0000 [id=37]    INFO    c.m.azure.vmagent.AzureVMCloud#provision: AzureVMCloud: provision: agent computer eligible for reuse vs2019-b5ac440
      2020-11-14 19:10:28.690+0000 [id=37]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#virtualMachineExists: AzureVMManagementServiceDelegate: virtualMachineExists: check for vs2019-b5ac440
      2020-11-14 19:10:28.758+0000 [id=37]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#virtualMachineExists: AzureVMManagementServiceDelegate: virtualMachineExists: vs2019-b5ac440 exists
      2020-11-14 19:10:28.759+0000 [id=37]    INFO    c.m.azure.vmagent.AzureVMCloud#provision: AzureVMCloud: provision: agent computer eligible for reuse vs2019-bdba010
      2020-11-14 19:10:28.759+0000 [id=37]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#virtualMachineExists: AzureVMManagementServiceDelegate: virtualMachineExists: check for vs2019-bdba010
      2020-11-14 19:10:28.828+0000 [id=37]    INFO    c.m.a.v.AzureVMManagementServiceDelegate#virtualMachineExists: AzureVMManagementServiceDelegate: virtualMachineExists: vs2019-bdba010 exists
      2020-11-14 19:10:28.828+0000 [id=37]    INFO    c.m.azure.vmagent.AzureVMCloud#provision: AzureVMCloud: provision: asynchronous provision finished, returning 3 planned node(s)
      2020-11-14 19:10:38.604+0000 [id=35]    WARNING hudson.slaves.NodeProvisioner#lambda$update$6: Unexpected exception encountered while provisioning agent vs2019-b0ccdf0
      java.lang.NullPointerException
              at com.microsoft.azure.vmagent.AzureVMCloud.getLockForAgent(AzureVMCloud.java:1017)
              at com.microsoft.azure.vmagent.AzureVMCloud.lambda$provision$1(AzureVMCloud.java:682)
              at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
              at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      2020-11-14 19:10:38.605+0000 [id=35]    WARNING hudson.slaves.NodeProvisioner#lambda$update$6: Unexpected exception encountered while provisioning agent vs2019-b5ac440
      java.lang.NullPointerException
              at com.microsoft.azure.vmagent.AzureVMCloud.getLockForAgent(AzureVMCloud.java:1017)
              at com.microsoft.azure.vmagent.AzureVMCloud.lambda$provision$1(AzureVMCloud.java:682)
              at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
              at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      2020-11-14 19:10:38.606+0000 [id=35]    WARNING hudson.slaves.NodeProvisioner#lambda$update$6: Unexpected exception encountered while provisioning agent vs2019-bdba010
      java.lang.NullPointerException
              at com.microsoft.azure.vmagent.AzureVMCloud.getLockForAgent(AzureVMCloud.java:1017)
              at com.microsoft.azure.vmagent.AzureVMCloud.lambda$provision$1(AzureVMCloud.java:682)
              at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
              at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748

        Attachments

          Activity

          Hide
          timja Tim Jacomb added a comment -

          All issues have been transferred to GitHub.

          See https://github.com/jenkinsci/azure-vm-agents-plugin/issues

          Search the issue title to find it.

          (This is a bulk comment and can't link to the specific issue)

          Show
          timja Tim Jacomb added a comment - All issues have been transferred to GitHub. See https://github.com/jenkinsci/azure-vm-agents-plugin/issues Search the issue title to find it. (This is a bulk comment and can't link to the specific issue)

            People

            Assignee:
            azure_devops Azure DevOps
            Reporter:
            siigna Steve Bouché
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: