Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61018

Cannot use plugin with existing storage account and secure transfer policy

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Duplicate
    • Component/s: _unsorted
    • Labels:
      None
    • Environment:
      Jenkins ver. 2.204.2
      Azure VM Agents 1.4.0
    • Similar Issues:

      Description

      Hi!

      We are trying to use the plugin with an existing resource group and storage account but this is not working with one of our subscription because of a policy that force "secure transfer" enabled.

      The storage account is already configured with secure transfer, but it seems that the plugin is trying to "change" the configuration of the storage account. Which fail with following error.

      Same issue if trying to use a new storage account

      This prevent us using the plugin

      Is it possible for example to have an option on the config to force the use of secure transfer and avoid issue with policies ?

      AzureVMManagementServiceDelegate: deployment: Unable to deploy
      com.microsoft.azure.CloudException: Status code 400, {"error":{"code":"InvalidTemplateDeployment","message":"The template deployment failed because of policy violation. Please see details for more information.","details":[{"code":"RequestDisallowedByPolicy","target":"*********","message":"Resource '*****devop*****jenkins******' was disallowed by policy. Policy identifiers: '[{\"policyAssignment\":{\"name\":\"Secure transfer to storage accounts should be enabled\",\"id\":\"/subscriptions/*********/providers/Microsoft.Authorization/policyAssignments/***********\"},\"policyDefinition\":{\"name\":\"Secure transfer to storage accounts should be enabled\",\"id\":\"/providers/Microsoft.Authorization/policyDefinitions/************\"}},{\"policyAssignment\":{\"name\":\"Secure transfer to storage accounts should be enabled\",\"id\":\"/providers/Microsoft.Management/managementGroups/********-***-mg/providers/Microsoft.Authorization/policyAssignments/************\"},\"policyDefinition\":{\"name\":\"Secure transfer to storage accounts should be enabled\",\"id\":\"/providers/Microsoft.Authorization/policyDefinitions/*************\"}}]'.","additionalInfo":[{"type":"PolicyViolation","info":{"policyDefinitionDisplayName":"Secure transfer to storage accounts should be enabled","evaluationDetails":{"evaluatedExpressions":[{"result":"True","expressionKind":"Field","expression":"type","path":"type","expressionValue":"Microsoft.Storage/storageAccounts","targetValue":"Microsoft.Storage/storageAccounts","operator":"Equals"},
      
      

      Thanks in advance for the help!

      Valentin

        Attachments

          Issue Links

            Activity

            Hide
            mdlugaj Marek Dlugajczyk added a comment - - edited

            Hi, I'm experiencing the same issue with the plugin and Azure policy for storage accounts. We are unable to create any agent either using builtin Azure images or our own custom images.

            Our company enables this storage account policy as mandatory and for sec. reasons we don't want to disable it. However we did a test, we disabled the policy temporairly on Azure side and then we were able to spin up VM agents. We've tested the plugin to create new storage account for the defined VM template - that worked. And we tested it with already existing storage account - that also worked.

            It seems like something is broken for the secure connection then. I also observed that if the policy is disabled and plugin is creating new storage account it creates V1 of the storage account, but the recommended is V2. (see the screenshot of the storage account)

            What we used for testing:

            • Jenkins - 2.235.2
            • Azure VM Agents Plugin - 1.4.1 and 1.5.0 (tested both versions)

            Error we observe when the Azure policy is enabled:

            com.microsoft.azure.vmagent.exceptions.AzureCloudException: Status code 400, {"error":{"code":"InvalidTemplateDeployment","message":"The template deployment failed because of policy violation. Please see details for more information.","details":[{"code":"RequestDisallowedByPolicy","target":"jnnpdbcr2ypejg6y9lsezheg","message":"Resource 'jnnpdbcr2ypejg6y9lsezheg' was disallowed by policy. Policy identifiers: '[{\"policyAssignment\":{\"name\":\"RA - Storage Accounts - Enforce Secure Transfer (HTTPS)\",\"id\":\"/providers/Microsoft.Management/managementgroups/enterprisecloudonazure/providers/Microsoft.Authorization/policyAssignments/507f6c3197724c78a3138f3b\"},\"policyDefinition\":{\"name\":\"RA - Storage Accounts - Enforce Secure Transfer (HTTPS)\",\"id\":\"/providers/Microsoft.Management/managementgroups/enterprisecloudonazure/providers/Microsoft.Authorization/policyDefinitions/pd-DenyStorageAccountUnencryptedTraffic\"}}]'.","additionalInfo":[{"type":"PolicyViolation","info":{"policyDefinitionDisplayName":"RA - Storage Accounts - Enforce Secure Transfer (HTTPS)","evaluationDetails":{"evaluatedExpressions":[{"result":"True","expressionKind":"Field","expression":"type","path":"type","expressionValue":"Microsoft.Storage/storageAccounts","targetValue":"Microsoft.Storage/storageAccounts","operator":"Equals"},{"result":"False","expressionKind":"Field","expression":"Microsoft.Storage/storageAccounts/supportsHttpsTrafficOnly","path":"properties.supportsHttpsTrafficOnly","targetValue":"true","operator":"Equals"}]},"policyDefinitionId":"/providers/Microsoft.Management/managementgroups/enterprisecloudonazure/providers/Microsoft.Authorization/policyDefinitions/pd-DenyStorageAccountUnencryptedTraffic","policyDefinitionName":"pd-DenyStorageAccountUnencryptedTraffic","policyDefinitionEffect":"deny","policyAssignmentId":"/providers/Microsoft.Management/managementgroups/enterprisecloudonazure/providers/Microsoft.Authorization/policyAssignments/507f6c3197724c78a3138f3b","policyAssignmentName":"507f6c3197724c78a3138f3b","policyAssignmentDisplayName":"RA - Storage Accounts - Enforce Secure Transfer (HTTPS)","policyAssignmentScope":"/providers/Microsoft.Management/managementgroups/enterprisecloudonazure"}}]}]}}
            	at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:51)
            	at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:33)
            	at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.createDeployment(AzureVMManagementServiceDelegate.java:565)
            	at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.createDeployment(AzureVMManagementServiceDelegate.java:174)
            	at com.microsoft.azure.vmagent.AzureVMAgentTemplate.provisionAgents(AzureVMAgentTemplate.java:1188)
            	at com.microsoft.azure.vmagent.AzureVMCloud$1.call(AzureVMCloud.java:794)
            	at com.microsoft.azure.vmagent.AzureVMCloud$1.call(AzureVMCloud.java:790)
            Caused: java.util.concurrent.ExecutionException
            	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
            	at java.util.concurrent.FutureTask.get(FutureTask.java:192)
            	at com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:829)
            Caused: com.microsoft.azure.vmagent.exceptions.AzureCloudException
            	at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:54)
            	at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:33)
            	at com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:831)
            	at com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:818)
            	at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
            	at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            	at java.lang.Thread.run(Thread.java:748)
            

             

            Show
            mdlugaj Marek Dlugajczyk added a comment - - edited Hi, I'm experiencing the same issue with the plugin and Azure policy for storage accounts. We are unable to create any agent either using builtin Azure images or our own custom images. Our company enables this storage account policy as mandatory and for sec. reasons we don't want to disable it. However we did a test, we disabled the policy temporairly on Azure side and then we were able to spin up VM agents. We've tested the plugin to create new storage account for the defined VM template - that worked. And we tested it with already existing storage account - that also worked. It seems like something is broken for the secure connection then. I also observed that if the policy is disabled and plugin is creating new storage account it creates V1 of the storage account, but the recommended is V2. (see the screenshot of the storage account) What we used for testing: Jenkins - 2.235.2 Azure VM Agents Plugin - 1.4.1 and 1.5.0 (tested both versions) Error we observe when the Azure policy is enabled: com.microsoft.azure.vmagent.exceptions.AzureCloudException: Status code 400, { "error" :{ "code" : "InvalidTemplateDeployment" , "message" : "The template deployment failed because of policy violation. Please see details for more information." , "details" :[{ "code" : "RequestDisallowedByPolicy" , "target" : "jnnpdbcr2ypejg6y9lsezheg" , "message" : "Resource 'jnnpdbcr2ypejg6y9lsezheg' was disallowed by policy. Policy identifiers: '[{\" policyAssignment\ ":{\" name\ ":\" RA - Storage Accounts - Enforce Secure Transfer (HTTPS)\ ",\" id\ ":\" /providers/Microsoft.Management/managementgroups/enterprisecloudonazure/providers/Microsoft.Authorization/policyAssignments/507f6c3197724c78a3138f3b\ "},\" policyDefinition\ ":{\" name\ ":\" RA - Storage Accounts - Enforce Secure Transfer (HTTPS)\ ",\" id\ ":\" /providers/Microsoft.Management/managementgroups/enterprisecloudonazure/providers/Microsoft.Authorization/policyDefinitions/pd-DenyStorageAccountUnencryptedTraffic\ "}}]' ." , "additionalInfo" :[{ "type" : "PolicyViolation" , "info" :{ "policyDefinitionDisplayName" : "RA - Storage Accounts - Enforce Secure Transfer (HTTPS)" , "evaluationDetails" :{ "evaluatedExpressions" :[{ "result" : "True" , "expressionKind" : "Field" , "expression" : "type" , "path" : "type" , "expressionValue" : "Microsoft.Storage/storageAccounts" , "targetValue" : "Microsoft.Storage/storageAccounts" , " operator " : "Equals" },{ "result" : "False" , "expressionKind" : "Field" , "expression" : "Microsoft.Storage/storageAccounts/supportsHttpsTrafficOnly" , "path" : "properties.supportsHttpsTrafficOnly" , "targetValue" : " true " , " operator " : "Equals" }]}, "policyDefinitionId" : "/providers/Microsoft.Management/managementgroups/enterprisecloudonazure/providers/Microsoft.Authorization/policyDefinitions/pd-DenyStorageAccountUnencryptedTraffic" , "policyDefinitionName" : "pd-DenyStorageAccountUnencryptedTraffic" , "policyDefinitionEffect" : "deny" , "policyAssignmentId" : "/providers/Microsoft.Management/managementgroups/enterprisecloudonazure/providers/Microsoft.Authorization/policyAssignments/507f6c3197724c78a3138f3b" , "policyAssignmentName" : "507f6c3197724c78a3138f3b" , "policyAssignmentDisplayName" : "RA - Storage Accounts - Enforce Secure Transfer (HTTPS)" , "policyAssignmentScope" : "/providers/Microsoft.Management/managementgroups/enterprisecloudonazure" }}]}]}} at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:51) at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:33) at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.createDeployment(AzureVMManagementServiceDelegate.java:565) at com.microsoft.azure.vmagent.AzureVMManagementServiceDelegate.createDeployment(AzureVMManagementServiceDelegate.java:174) at com.microsoft.azure.vmagent.AzureVMAgentTemplate.provisionAgents(AzureVMAgentTemplate.java:1188) at com.microsoft.azure.vmagent.AzureVMCloud$1.call(AzureVMCloud.java:794) at com.microsoft.azure.vmagent.AzureVMCloud$1.call(AzureVMCloud.java:790) Caused: java.util.concurrent.ExecutionException at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:829) Caused: com.microsoft.azure.vmagent.exceptions.AzureCloudException at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:54) at com.microsoft.azure.vmagent.exceptions.AzureCloudException.create(AzureCloudException.java:33) at com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:831) at com.microsoft.azure.vmagent.AzureVMCloud$2.call(AzureVMCloud.java:818) at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46) at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang. Thread .run( Thread .java:748)  
            Hide
            rafa_nab Rafa Pizzi added a comment - - edited

            Hi,

            I'm having the same issue here, and I believe this will be a big thing for most major corporations with policies enabled.

            Seems this issue is linked to a "resolved" issue that hasn't really been resolved: JENKINS-52967 

            We were able to make a copy of the code, and change some parameters to make it work locally.

            Basically, add the 

            ```

            "properties":

            { "supportsHttpsTrafficOnly" = true }

            ```

            https://docs.microsoft.com/en-us/azure/templates/microsoft.storage/2019-04-01/storageaccounts#storageaccountpropertiescreateparameters-object

            at the templates.json here
            https://github.com/jenkinsci/azure-vm-agents-plugin/blob/master/src/main/resources/

             

            Show
            rafa_nab Rafa Pizzi added a comment - - edited Hi, I'm having the same issue here, and I believe this will be a big thing for most major corporations with policies enabled. Seems this issue is linked to a "resolved" issue that hasn't really been resolved:  JENKINS-52967   We were able to make a copy of the code, and change some parameters to make it work locally. Basically, add the  ``` "properties": { "supportsHttpsTrafficOnly" = true } ``` https://docs.microsoft.com/en-us/azure/templates/microsoft.storage/2019-04-01/storageaccounts#storageaccountpropertiescreateparameters-object at the templates.json here https://github.com/jenkinsci/azure-vm-agents-plugin/blob/master/src/main/resources/  
            Hide
            georges474 Georges Zwingelstein added a comment -

            To make it work with an existing storage account, in addition to the changes from Rafa Pizzi, the templates.json must also use https instead of http.

            Show
            georges474 Georges Zwingelstein added a comment - To make it work with an existing storage account, in addition to the changes from Rafa Pizzi , the templates.json must also use https instead of http .
            Hide
            krische Brian Krische added a comment -

            I created a pull request with the necessary changes for this issue: https://github.com/jenkinsci/azure-vm-agents-plugin/pull/188

            However, as I mentioned in the PR, I'm not sure if that would break users that don't have the secure transfer policy enabled.

            But at least for people affected here, you could just build the plugin locally with the changes in my PR to unblock yourself.

            Show
            krische Brian Krische added a comment - I created a pull request with the necessary changes for this issue: https://github.com/jenkinsci/azure-vm-agents-plugin/pull/188 However, as I mentioned in the PR, I'm not sure if that would break users that don't have the secure transfer policy enabled. But at least for people affected here, you could just build the plugin locally with the changes in my PR to unblock yourself.
            Hide
            hentis Henti Smith added a comment -

            While looking at this issue with some colleagues we noted that using API version `2019-04-01` or later of `Microsoft.Storage/storageAccounts` will solve this.

             

            Is there a compelling reason that this plug-in is still using `2016-01-01`

             

            Updating all the API versions in:

              src/main/resources/referenceImageIDTemplateWithManagedDisk.json
              src/main/resources/referenceImageIDTemplateWithScriptAndManagedDisk.json
              src/main/resources/referenceImageTemplate.json
              src/main/resources/referenceImageTemplateWithManagedDisk.json
              src/main/resources/referenceImageTemplateWithScript.json
              src/main/resources/referenceImageTemplateWithScriptAndManagedDisk.json

             

            and rebuilding the plugin worked for us.

            Show
            hentis Henti Smith added a comment - While looking at this issue with some colleagues we noted that using API version `2019-04-01` or later of `Microsoft.Storage/storageAccounts` will solve this.   Is there a compelling reason that this plug-in is still using `2016-01-01`   Updating all the API versions in:   src/main/resources/referenceImageIDTemplateWithManagedDisk.json   src/main/resources/referenceImageIDTemplateWithScriptAndManagedDisk.json   src/main/resources/referenceImageTemplate.json   src/main/resources/referenceImageTemplateWithManagedDisk.json   src/main/resources/referenceImageTemplateWithScript.json   src/main/resources/referenceImageTemplateWithScriptAndManagedDisk.json   and rebuilding the plugin worked for us.
            Hide
            timja Tim Jacomb added a comment -

            All issues have been transferred to GitHub.

            See https://github.com/jenkinsci/azure-vm-agents-plugin/issues

            Search the issue title to find it.

            (This is a bulk comment and can't link to the specific issue)

            Show
            timja Tim Jacomb added a comment - All issues have been transferred to GitHub. See https://github.com/jenkinsci/azure-vm-agents-plugin/issues Search the issue title to find it. (This is a bulk comment and can't link to the specific issue)

              People

              Assignee:
              azure_devops Azure DevOps
              Reporter:
              jonesbusy Valentin Delaye
              Votes:
              10 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: