Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-72520

Http Request plugin Multipart form does not set boundaries for multipart

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • http-request-plugin
    • None
    • Jenkins 2.393

      I'm trying to POST an XML file (OWASP ZAP Report) to an external service (OWASP Defect Dojo), using a shared Jenkins library. A curl-command (see below) for POST-ing the XML file works fine. But I have a hard time translating that into Jenkins' Http Request plugin, so we can include it in a reusable Jenkins shared library.

      This results in the following response:

       

      Response Code: HTTP/1.1 400 Bad Request
      Response: 
      {"detail":"Multipart form parse error - Invalid boundary in multipart: None"} 

      Is this a bug or am I missing something?

       

       

      curl command:

       

      curl -X 'POST' \
        'http:<server>:8080/api/v2/import-scan/' \
        -H 'accept: application/json' \
        -H 'Content-Type: multipart/form-data' \
        -H 'X-CSRFToken: <token>' \
        -H 'Authorization: Token <token>' \
        -F 'scan_date=2023-11-20' \
        -F 'minimum_severity=Info' \
        -F 'active=true' \
        -F 'verified=true' \
        -F 'scan_type=ZAP Scan' \
        -F 'file=@zapreport.xml;type=text/xml' \
        -F 'product_name=Test project' \
        -F 'engagement_name=Test engagement' \
        -F 'close_old_findings=false' \
        -F 'close_old_findings_product_scope=false' \
        -F 'push_to_jira=false' \
        -F 'create_finding_groups_for_all_findings=true' 

       

       

      Jenkins Groovy code:

              ResponseContentSupplier response = steps.httpRequest url: "${env.DEFECT_DOJO_HOST_URL}${api}",
                      acceptType: 'APPLICATION_JSON',
                      contentType: 'APPLICATION_FORM_DATA',
                      httpMode: 'POST',
                      uploadFile: 'zapreport.xml',
                      multipartName: 'zapreport.xml',
                      customHeaders: customHeadersDummy(authHeader, headers), // Includes auth-token
                      formData: [[body: '''{
          "product_name": "Dummy project",
          "engagement_name": "Dummy Engagement",
          "scan_date": "2023-11-20",
          "minimum_severity": "Info",
          "active": "true",
          "verified": "true",
          "scan_type": "ZAP Scan",
          "close_old_findings": "false",
          "close_old_findings_product_scope": "false",
          "push_to_jira": "false",
          "create_finding_groups_for_all_findings": "true"
      }''',
                       contentType: 'text/xml',
                       fileName: 'zapreport.xml',
                       name: 'zapreport.xml',
                       uploadFile: 'zapreport.xml']],
                       consoleLogResponseBody: true,
                      responseHandle: 'NONE',
                      timeout: null
                      wrapAsMultipart: false 

       

          [JENKINS-72520] Http Request plugin Multipart form does not set boundaries for multipart

          Mark Waite added a comment -

          Since you already have it working as a curl command, why not just use the curl command in your Pipeline? Curl is available for multiple platforms and is known to work well.

          Mark Waite added a comment - Since you already have it working as a curl command, why not just use the curl command in your Pipeline? Curl is available for multiple platforms and is known to work well.

          Bram added a comment -

          Well.... is that even possible from a Groovy shared library? And can I catch the response?

          But aside from that: I'm using the httpRequest plugin for integrating / calling the Defect Dojo REST API. So for consistency I'ld like to do that for the import-scan call as well.

          Why are we making shared libraries for this? Because we're tailoring reusable (share) libraries for our development teams to, well.... reuse. Encapsulating the complex logic into easy to use function-calls (facade pattern?) is what we aim for instead of instructions to let them copy-paste a curl command. So much issues with that. What if Defect Dojo moves to another server? Where would they get auth-tokens from? How should they inject them correctly into the pipeline without exposing them? etc.

          2nd aside: switching to curl / sh scripts might result in command injection issues, which I'ld like to prevent.

          Bram added a comment - Well.... is that even possible from a Groovy shared library? And can I catch the response? But aside from that: I'm using the httpRequest plugin for integrating / calling the Defect Dojo REST API. So for consistency I'ld like to do that for the import-scan call as well. Why are we making shared libraries for this? Because we're tailoring reusable (share) libraries for our development teams to, well.... reuse. Encapsulating the complex logic into easy to use function-calls (facade pattern?) is what we aim for instead of instructions to let them copy-paste a curl command. So much issues with that. What if Defect Dojo moves to another server? Where would they get auth-tokens from? How should they inject them correctly into the pipeline without exposing them? etc. 2nd aside: switching to curl / sh scripts might result in command injection issues, which I'ld like to prevent.

          Mark Waite added a comment -

          Well.... is that even possible from a Groovy shared library?

          Yes, it is possible to call sh and bat and powershell from a Pipeline shared library.

          And can I catch the response?

          Yes, it is possible to catch the return code or stdout or stderr from sh and bat and powershell in a Pipeline shared library..

          Because we're tailoring reusable (share) libraries for our development teams to, well.... reuse.

          I think that's a very good thing. Placing the curl call inside the Pipeline shared library will give you the same reuse benefit without requiring that you wait for someone to implement this request in the http-request plugin.

          If you're submitting the improvement as an http-request plugin pull request with tests that confirm the code changes are working well, then the maintainers of the http-request plugin (including me) may be willing to review the pull request, evaluate it, merge it, and release it. If you're volunteering to become one of the maintainers of the http-request plugin, then you could submit the pull request, evaluate it, merge it, and release it yourself. If you're not submitting the improvement, then you are at the mercy of the plugin maintainers. I suggested that you use curl because I assumed that you weren't ready to submit the pull request that implements the feature.

          Mark Waite added a comment - Well.... is that even possible from a Groovy shared library? Yes, it is possible to call sh and bat and powershell from a Pipeline shared library. And can I catch the response? Yes, it is possible to catch the return code or stdout or stderr from sh and bat and powershell in a Pipeline shared library.. Because we're tailoring reusable (share) libraries for our development teams to, well.... reuse. I think that's a very good thing. Placing the curl call inside the Pipeline shared library will give you the same reuse benefit without requiring that you wait for someone to implement this request in the http-request plugin. If you're submitting the improvement as an http-request plugin pull request with tests that confirm the code changes are working well, then the maintainers of the http-request plugin (including me) may be willing to review the pull request, evaluate it, merge it, and release it. If you're volunteering to become one of the maintainers of the http-request plugin, then you could submit the pull request, evaluate it, merge it, and release it yourself. If you're not submitting the improvement, then you are at the mercy of the plugin maintainers. I suggested that you use curl because I assumed that you weren't ready to submit the pull request that implements the feature.

          Bram added a comment -

          Thank you for your response.

          I'm a bit confused. What do you mean with "Implement this feature"? Is it currently not possible to do a POST with form-data and file-upload, similar to my pasted curl command? Or is it not possible to POST to Defect Dojo? Since the Http Request plugin is a generic tool, I'ld  like to assume that any Defect Dojo specific feature doesn't make sense. But I expected it to support any form of Http Requests. Is that incorrect?

          Bram added a comment - Thank you for your response. I'm a bit confused. What do you mean with "Implement this feature"? Is it currently not possible to do a POST with form-data and file-upload, similar to my pasted curl command? Or is it not possible to POST to Defect Dojo? Since the Http Request plugin is a generic tool, I'ld  like to assume that any Defect Dojo specific feature doesn't make sense. But I expected it to support any form of Http Requests. Is that incorrect?

          Mark Waite added a comment -

          What do you mean with "Implement this feature"?

          It would have been clearer if I'd used the phrase "fix this bug" instead of "implement this feature". Jerry Weinberg's definition of a bug is "anything that bugs someone". This bugs you, so it is a bug.

          I mean that you are requesting a change in the behavior of the http-request plugin. You've defined this change in behavior of the http-request plugin as a blocker. I've offered a workaround so that your efforts won't be blocked.

          I don't plan to work on this issue. I suspect that the other maintainers of the http-request plugin are unlikely to work on this issue either. If you decide to submit a pull request to fix the bug and if you include tests that verify the fix, then I'll put the review of that fix into my review queue.

          But I expected it to support any form of Http Requests. Is that incorrect?

          I agree that the http-request plugin should support any form of http request that makes sense in the context of a Jenkins controller or Jenkins agent. I don't know if the http-request plugin support multi-part form submission of files. I assume from your very thorough description of the issue that it includes some support for multi-part form submission of files, but that support is incorrect or insufficient.

          Mark Waite added a comment - What do you mean with "Implement this feature"? It would have been clearer if I'd used the phrase "fix this bug" instead of "implement this feature". Jerry Weinberg's definition of a bug is "anything that bugs someone". This bugs you, so it is a bug. I mean that you are requesting a change in the behavior of the http-request plugin. You've defined this change in behavior of the http-request plugin as a blocker. I've offered a workaround so that your efforts won't be blocked. I don't plan to work on this issue. I suspect that the other maintainers of the http-request plugin are unlikely to work on this issue either. If you decide to submit a pull request to fix the bug and if you include tests that verify the fix, then I'll put the review of that fix into my review queue. But I expected it to support any form of Http Requests. Is that incorrect? I agree that the http-request plugin should support any form of http request that makes sense in the context of a Jenkins controller or Jenkins agent. I don't know if the http-request plugin support multi-part form submission of files. I assume from your very thorough description of the issue that it includes some support for multi-part form submission of files, but that support is incorrect or insufficient.

          Bram added a comment -

          Thank you for clearing this up. This does save me a lot of time trying to trial and error my issue using the current http-request plugin.

          I don't think I'm able to fix this issue in the source of http-request plugin, due to time constraints. So I will first try to implement the workaround in a Groovy shared library. If I can spend the time I will see if and how I can implement the bug-fix eventually, but I do realise that this might be a niche use-case. Although, it is something that's essential to integrating OWASP Defect Dojo's import scan API-calls.

          Bram added a comment - Thank you for clearing this up. This does save me a lot of time trying to trial and error my issue using the current http-request plugin. I don't think I'm able to fix this issue in the source of http-request plugin, due to time constraints. So I will first try to implement the workaround in a Groovy shared library. If I can spend the time I will see if and how I can implement the bug-fix eventually, but I do realise that this might be a niche use-case. Although, it is something that's essential to integrating OWASP Defect Dojo's import scan API-calls.

            janario Janario Oliveira
            brampat Bram
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: