Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33925

Test framework for Jenkinsfile

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      It would be desirable to have a standard mechanism for testing Pipeline scripts without running them on a production server. There are two competing suggestions:

      Mock framework

      Inspired by Job DSL (example).

      We could set up a GroovyShell in which step functions and global variables were predefined as mocks (in a fashion similar to Powermock, but easier in Groovy given its dynamic nature), and stub out the expected return value / exception for each, with some standard predefinitions such as for currentBuild.

      Ideally the shell would be CPS-transformed, with the program state serialized and then reloaded between every continuation (though this might involve a lot of code duplication with workflow-cps).

      Should be easy to pass it through the Groovy sandbox (if requested), though the live Whitelist.all from Jenkins would be unavailable, so we would be limited to known static whitelists, the @Whitelisted annotation, and perhaps some custom additions.

      Quick and flexible, but low fidelity to real behavior.

      JenkinsRule-style

      Use an embedded Jenkins server, as per JenkinsRule in jenkins-test-harness, and actually create a WorkflowJob with the specified definition. Can use for example mock-slave to create nodes.

      Need to have a "dry-run" flag so that attempts to do things like deploy artifacts or send email do not really take action. This could perhaps be a general API in Jenkins core, as it would be useful also for test instances (shadows of production servers), acceptance-test-harness, etc.

      Slower to run (seconds per test case rather than milliseconds), and trickier to set up, but much more realistic coverage. The tests for Pipeline (and Pipeline steps) themselves use this technique.

        Attachments

          Issue Links

            Activity

            Hide
            jglick Jesse Glick added a comment -

            I am not maintaining this code so I cannot comment.

            Show
            jglick Jesse Glick added a comment - I am not maintaining this code so I cannot comment.
            Hide
            sparshev Sergei Parshev added a comment -

            Ok, just added issue and PR with a sample realization to groovy-cps. Hopefully we will find some way to simplify the execution of JenkinsRule-based unit tests for the shared libraries.

            Show
            sparshev Sergei Parshev added a comment - Ok, just added issue and PR with a sample realization to groovy-cps. Hopefully we will find some way to simplify the execution of JenkinsRule-based unit tests for the shared libraries.
            Hide
            sparshev Sergei Parshev added a comment -

            Hi Jesse Glick, so I bumped in the hardcode in cps groovy shell here: https://github.com/cloudbees/groovy-cps/pull/107#issuecomment-601904207 - so it's impossible to use GroovyInterceptor dynamically. Maybe there is some another way?

            Show
            sparshev Sergei Parshev added a comment - Hi Jesse Glick , so I bumped in the hardcode in cps groovy shell here: https://github.com/cloudbees/groovy-cps/pull/107#issuecomment-601904207 - so it's impossible to use GroovyInterceptor dynamically. Maybe there is some another way?
            Hide
            mlasevich Michael Lasevich added a comment - - edited

            FWIW, it seems there are at least two distinct problems here as far as testing is concerned - one is mocking of the pipeline steps. There are at least two frameworks attempting that already (JenkinsPipelineUnit and Jenkins-Spock). But more importantly there is a need to test code, both Jenkinsfile and more importantly Pipeline Library code, under CPS as what is legitimate Groovy code may fail or behave differently under CPS.  While there are limited attempts at it in the above mentioned frameworks, neither can properly execute CPS code - as a result, the code that tested fine under unit testing may fail spectacularly in real world use.

            With that in mind, we created a small utility class that bypasses the greatest limitation of the CPS - ability to invoke CPS code from non-CPS (i.e. unit test) context. We paired this with test-time compilation of code with CPS transform enabled as well as ability to compile a string as a CPS Transformed script. This small utility class solves the second issue of reliable unit-testing of pipeline code under CPS. 

            I would like to share this utility with the community, but what I am wondering is - given the tiny size of the utility (and the fact that it was heavily inspired by groovy-cps's own unit tests, it seems silly to package this separately - does it make sense to submit it to become a part of groovy-cps package? There was a statement early own that maintainers of the groovy-cps may not wish to add testing support to the package - so I am hesitant to submit a PR... On the other side, I would love for someone who actually understands CPS code to review/fix what we cobbled together  

             

            Show
            mlasevich Michael Lasevich added a comment - - edited FWIW, it seems there are at least two distinct problems here as far as testing is concerned - one is mocking of the pipeline steps. There are at least two frameworks attempting that already ( JenkinsPipelineUnit and Jenkins-Spock ). But more importantly there is a need to test code, both Jenkinsfile and more importantly Pipeline Library code, under CPS as what is legitimate Groovy code may fail or behave differently under CPS.  While there are limited attempts at it in the above mentioned frameworks, neither can properly execute CPS code - as a result, the code that tested fine under unit testing may fail spectacularly in real world use. With that in mind, we created a small utility class that bypasses the greatest limitation of the CPS - ability to invoke CPS code from non-CPS (i.e. unit test) context. We paired this with test-time compilation of code with CPS transform enabled as well as ability to compile a string as a CPS Transformed script. This small utility class solves the second issue of reliable unit-testing of pipeline code under CPS.  I would like to share this utility with the community, but what I am wondering is - given the tiny size of the utility (and the fact that it was heavily inspired by groovy-cps 's own unit tests, it seems silly to package this separately - does it make sense to submit it to become a part of groovy-cps package? There was a statement early own that maintainers of the groovy-cps may not wish to add testing support to the package - so I am hesitant to submit a PR... On the other side, I would love for someone who actually understands CPS code to review/fix what we cobbled together    
            Hide
            sparshev Sergei Parshev added a comment -

            Hi Michael Lasevich, that will be great to see the implementation, because my one ( https://github.com/cloudbees/groovy-cps/pull/107 ) is not great... So will be glad to help with testing.

            Show
            sparshev Sergei Parshev added a comment - Hi Michael Lasevich , that will be great to see the implementation, because my one ( https://github.com/cloudbees/groovy-cps/pull/107 ) is not great... So will be glad to help with testing.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              jglick Jesse Glick
              Votes:
              112 Vote for this issue
              Watchers:
              133 Start watching this issue

                Dates

                Created:
                Updated: