Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-56824

VMs not cleaned up per retention policy when they are in a failed provisioning state.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Component/s: _unsorted
    • Labels:
      None
    • Environment:
      Azure VM Agents 0.9.0, Jenkins 2.164.1
    • Similar Issues:

      Description

      This has happened a couple of times on ci.j.io. For whatever reason, we're not entirely sure, the init script for our Windows VMs sometimes fails.

      As a result, the agents are provisioned in Azure, but Jenkins fails to launch the JNLP agent on the machines. Unfortunately, Jenkins doesn't clean up these resources properly using our 20 minute idle retention policy.

      I discovered this morning that we had 50 VMs online for quite a few days, which never executed a single workload because their init script failed to run, and they were not cleaned up properly by the plugin.

        Attachments

          Activity

          Hide
          jieshe Jie Shen added a comment -

          Thanks R. Tyler Croy, this is should be a bug. I will find out how to fix it.

          Show
          jieshe Jie Shen added a comment - Thanks R. Tyler Croy , this is should be a bug. I will find out how to fix it.
          Hide
          rtyler R. Tyler Croy added a comment -

          Unfortunately I don't have a strong reproduction case, just behavior observed in production

          Show
          rtyler R. Tyler Croy added a comment - Unfortunately I don't have a strong reproduction case, just behavior observed in production
          Hide
          jieshe Jie Shen added a comment -

          Some feedback here.

          Idle retention policy will not delete offline VMs to avoid deleting temporarily disconnected one. So the issue should be related with why not the VMs being deleted immediately if the init scripts failed.

          Show
          jieshe Jie Shen added a comment - Some feedback here. Idle retention policy will not delete offline VMs to avoid deleting temporarily disconnected one. So the issue should be related with why not the VMs being deleted immediately if the init scripts failed.
          Hide
          timja Tim Jacomb added a comment -

          All issues have been transferred to GitHub.

          See https://github.com/jenkinsci/azure-vm-agents-plugin/issues

          Search the issue title to find it.

          (This is a bulk comment and can't link to the specific issue)

          Show
          timja Tim Jacomb added a comment - All issues have been transferred to GitHub. See https://github.com/jenkinsci/azure-vm-agents-plugin/issues Search the issue title to find it. (This is a bulk comment and can't link to the specific issue)

            People

            Assignee:
            jieshe Jie Shen
            Reporter:
            rtyler R. Tyler Croy
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: