It would be desirable to have a standard mechanism for testing Pipeline scripts without running them on a production server. There are two competing suggestions:

      Mock framework

      Inspired by Job DSL (example).

      We could set up a GroovyShell in which step functions and global variables were predefined as mocks (in a fashion similar to Powermock, but easier in Groovy given its dynamic nature), and stub out the expected return value / exception for each, with some standard predefinitions such as for currentBuild.

      Ideally the shell would be CPS-transformed, with the program state serialized and then reloaded between every continuation (though this might involve a lot of code duplication with workflow-cps).

      Should be easy to pass it through the Groovy sandbox (if requested), though the live Whitelist.all from Jenkins would be unavailable, so we would be limited to known static whitelists, the @Whitelisted annotation, and perhaps some custom additions.

      Quick and flexible, but low fidelity to real behavior.

      JenkinsRule-style

      Use an embedded Jenkins server, as per JenkinsRule in jenkins-test-harness, and actually create a WorkflowJob with the specified definition. Can use for example mock-slave to create nodes.

      Need to have a "dry-run" flag so that attempts to do things like deploy artifacts or send email do not really take action. This could perhaps be a general API in Jenkins core, as it would be useful also for test instances (shadows of production servers), acceptance-test-harness, etc.

      Slower to run (seconds per test case rather than milliseconds), and trickier to set up, but much more realistic coverage. The tests for Pipeline (and Pipeline steps) themselves use this technique.

          [JENKINS-33925] Test framework for Jenkinsfile

          Jesse Glick added a comment -

          I am not maintaining this code so I cannot comment.

          Jesse Glick added a comment - I am not maintaining this code so I cannot comment.

          Ok, just added issue and PR with a sample realization to groovy-cps. Hopefully we will find some way to simplify the execution of JenkinsRule-based unit tests for the shared libraries.

          Sergei Parshev added a comment - Ok, just added issue and PR with a sample realization to groovy-cps. Hopefully we will find some way to simplify the execution of JenkinsRule-based unit tests for the shared libraries.

          Hi jglick, so I bumped in the hardcode in cps groovy shell here: https://github.com/cloudbees/groovy-cps/pull/107#issuecomment-601904207 - so it's impossible to use GroovyInterceptor dynamically. Maybe there is some another way?

          Sergei Parshev added a comment - Hi jglick , so I bumped in the hardcode in cps groovy shell here: https://github.com/cloudbees/groovy-cps/pull/107#issuecomment-601904207 - so it's impossible to use GroovyInterceptor dynamically. Maybe there is some another way?

          Michael Lasevich added a comment - - edited

          FWIW, it seems there are at least two distinct problems here as far as testing is concerned - one is mocking of the pipeline steps. There are at least two frameworks attempting that already (JenkinsPipelineUnit and Jenkins-Spock). But more importantly there is a need to test code, both Jenkinsfile and more importantly Pipeline Library code, under CPS as what is legitimate Groovy code may fail or behave differently under CPS.  While there are limited attempts at it in the above mentioned frameworks, neither can properly execute CPS code - as a result, the code that tested fine under unit testing may fail spectacularly in real world use.

          With that in mind, we created a small utility class that bypasses the greatest limitation of the CPS - ability to invoke CPS code from non-CPS (i.e. unit test) context. We paired this with test-time compilation of code with CPS transform enabled as well as ability to compile a string as a CPS Transformed script. This small utility class solves the second issue of reliable unit-testing of pipeline code under CPS. 

          I would like to share this utility with the community, but what I am wondering is - given the tiny size of the utility (and the fact that it was heavily inspired by groovy-cps's own unit tests, it seems silly to package this separately - does it make sense to submit it to become a part of groovy-cps package? There was a statement early own that maintainers of the groovy-cps may not wish to add testing support to the package - so I am hesitant to submit a PR... On the other side, I would love for someone who actually understands CPS code to review/fix what we cobbled together  

           

          Michael Lasevich added a comment - - edited FWIW, it seems there are at least two distinct problems here as far as testing is concerned - one is mocking of the pipeline steps. There are at least two frameworks attempting that already ( JenkinsPipelineUnit and Jenkins-Spock ). But more importantly there is a need to test code, both Jenkinsfile and more importantly Pipeline Library code, under CPS as what is legitimate Groovy code may fail or behave differently under CPS.  While there are limited attempts at it in the above mentioned frameworks, neither can properly execute CPS code - as a result, the code that tested fine under unit testing may fail spectacularly in real world use. With that in mind, we created a small utility class that bypasses the greatest limitation of the CPS - ability to invoke CPS code from non-CPS (i.e. unit test) context. We paired this with test-time compilation of code with CPS transform enabled as well as ability to compile a string as a CPS Transformed script. This small utility class solves the second issue of reliable unit-testing of pipeline code under CPS.  I would like to share this utility with the community, but what I am wondering is - given the tiny size of the utility (and the fact that it was heavily inspired by groovy-cps 's own unit tests, it seems silly to package this separately - does it make sense to submit it to become a part of groovy-cps package? There was a statement early own that maintainers of the groovy-cps may not wish to add testing support to the package - so I am hesitant to submit a PR... On the other side, I would love for someone who actually understands CPS code to review/fix what we cobbled together    

          Hi mlasevich, that will be great to see the implementation, because my one ( https://github.com/cloudbees/groovy-cps/pull/107 ) is not great... So will be glad to help with testing.

          Sergei Parshev added a comment - Hi mlasevich , that will be great to see the implementation, because my one ( https://github.com/cloudbees/groovy-cps/pull/107 ) is not great... So will be glad to help with testing.

          sparshev For the moment I just pushed a (slightly) cleaned up version into github here:

          https://github.com/mlasevich/jenkins-pipeline-library-reference/blob/main/test/src/support/cps/CPSUtils.groovy

          it boils down to two key static methods (heavily inspired by groovy-cps's own unit tests

          • CPSUtils.invokeCpsMethod(Object object, String methodName, Object...args) - this invokes a CPS Method on an object by name
          • CPSUtils.asCPSScript(String script) - which takes a script as a string and runs it (with optional bindings)

          Both are primarily intended for use in Unit Tests. Would be nice to have this be part of standard library to avoid the constant cut-n-paste (and it is a bit tiny to be its own library

          I also had a wrapper that can wrap any object and run invokeCPSMethod transparently - but it proved to be a bit more complicated and unstable, so I removed it for time being

          Michael Lasevich added a comment - sparshev For the moment I just pushed a (slightly) cleaned up version into github here: https://github.com/mlasevich/jenkins-pipeline-library-reference/blob/main/test/src/support/cps/CPSUtils.groovy it boils down to two key static methods (heavily inspired by groovy-cps 's own unit tests CPSUtils.invokeCpsMethod(Object object, String methodName, Object...args) - this invokes a CPS Method on an object by name CPSUtils. asCPSScript(String script) - which takes a script as a string and runs it (with optional bindings) Both are primarily intended for use in Unit Tests. Would be nice to have this be part of standard library to avoid the constant cut-n-paste (and it is a bit tiny to be its own library I also had a wrapper that can wrap any object and run invokeCPSMethod transparently - but it proved to be a bit more complicated and unstable, so I removed it for time being

          Sergei Parshev added a comment - - edited

          mlasevich, thank you for the code. I checked it quickly - so what is the difference between your CPSUtil and JenkinsPipelineUnit https://github.com/jenkinsci/JenkinsPipelineUnit/tree/master/src/main/groovy/com/lesfurets/jenkins/unit/cps ? As far I know it's already using CPS to run the tests... But, for example, when it comes to groovy specifics (like the one I found there: https://github.com/griddynamics/mpl/pull/49#issuecomment-551287490 ) - CPS & Jenkins pipeline starts to behave differently.

          I think it's quite critical to actually use Jenkins & plugins (security, steps...) for testing, that's why, for example I use the jenkins class FunctionCallEnv override and jenkins harness to be able to intercept the calls for mocking.

          So could you please describe how your script is closer to Jenkins than the JenkinsPipelineUnit's one, maybe I got it wrong?

          Sergei Parshev added a comment - - edited mlasevich , thank you for the code. I checked it quickly - so what is the difference between your CPSUtil and JenkinsPipelineUnit https://github.com/jenkinsci/JenkinsPipelineUnit/tree/master/src/main/groovy/com/lesfurets/jenkins/unit/cps ? As far I know it's already using CPS to run the tests... But, for example, when it comes to groovy specifics (like the one I found there: https://github.com/griddynamics/mpl/pull/49#issuecomment-551287490 ) - CPS & Jenkins pipeline starts to behave differently. I think it's quite critical to actually use Jenkins & plugins (security, steps...) for testing, that's why, for example I use the jenkins class FunctionCallEnv override and jenkins harness to be able to intercept the calls for mocking. So could you please describe how your script is closer to Jenkins than the JenkinsPipelineUnit's one, maybe I got it wrong?

          sparshev - My code is far simpler than what JenkinsPipelineUnit does. It simply allows you to execute CPS code - that is all - remainder is up to the testing framework. 

          JenkinsPipelineUnit (and JenkinsSpock) tries to do a lot of other things, like mocking the steps and what not. Those are great things, but I had hard time getting either to play nice with CPS Transformed code. At best they allowed loading scripts directly from groovy files, but that seems to bypass the classloaders and makes it very difficult to actually unit test anything - at best you can execute the entire `vars` code as a script, but not isolate and call specific methods in specific classes. We have a great deal of actual classes and proper code in out libraries, and I found no easy way to create an instance of a class and test it in either framework. That does not mean there isn't a way, I just have not found it.

          My approach is far more straight forward/simplistic. I enable CPS Transform on compile for all code, so all my classes are already transformed, and then just run my unit tests like I would with any other code. If I need to test code that is CPS transformed, I have to call it via CPSUtils but that is it. I am not running any of the actual steps or any Jenkins code beyond groovy-cps transforms - I can stub out the calls and assume that the actual steps are unit tested in the plugin code that provides them. It is not an end-to-end or integration testing - it is pure unit testing at that point.

          Ideally I would love to integrate this functionality with JenkinsPipelineUnit functionality - but for now I am just using pure Spock tests and just mock any steps I need

          Michael Lasevich added a comment - sparshev  - My code is far simpler than what JenkinsPipelineUnit does. It simply allows you to execute CPS code - that is all - remainder is up to the testing framework.  JenkinsPipelineUnit (and JenkinsSpock) tries to do a lot of other things, like mocking the steps and what not. Those are great things, but I had hard time getting either to play nice with CPS Transformed code. At best they allowed loading scripts directly from groovy files, but that seems to bypass the classloaders and makes it very difficult to actually unit test anything - at best you can execute the entire `vars` code as a script, but not isolate and call specific methods in specific classes. We have a great deal of actual classes and proper code in out libraries, and I found no easy way to create an instance of a class and test it in either framework. That does not mean there isn't a way, I just have not found it. My approach is far more straight forward/simplistic. I enable CPS Transform on compile for all code, so all my classes are already transformed, and then just run my unit tests like I would with any other code. If I need to test code that is CPS transformed, I have to call it via CPSUtils but that is it. I am not running any of the actual steps or any Jenkins code beyond groovy-cps transforms - I can stub out the calls and assume that the actual steps are unit tested in the plugin code that provides them. It is not an end-to-end or integration testing - it is pure unit testing at that point. Ideally I would love to integrate this functionality with JenkinsPipelineUnit functionality - but for now I am just using pure Spock tests and just mock any steps I need

          Ok, got it... Unfortunately it is not enough to complete the blackbox pipeline testing due to the reasons I described before: Jenkins pipeline is far more complex than CPS. It introduces restrictions that is not covered via pure CPS, but if this implementation is working for your case - that's great!)

          Sergei Parshev added a comment - Ok, got it... Unfortunately it is not enough to complete the blackbox pipeline testing due to the reasons I described before: Jenkins pipeline is far more complex than CPS. It introduces restrictions that is not covered via pure CPS, but if this implementation is working for your case - that's great!)

          Yeah, this is for pure unit testing and only of the actual code we wrote - i.e. testing that each bit of our code does exactly what we intended it to do  - but it is not intended to test if what we intended to do was the right thing, or if the code we did not write(i.e. Jenkins and plugins) does the right thing. We would need more for that sort of testing, but so far, vast majority of the issues have been purely in our own code - and our primary goal is to make sure that any change we do to that code does not break builds for everyone using this shared code.

          That said, regardless of how you test your code, if you are not testing running it under CPS Transform, your tests are mostly worthless. There are many cases where CPS transformed code just does not behave the same way as non-transformed code. :-/

          Michael Lasevich added a comment - Yeah, this is for pure unit testing and only of the actual code we wrote - i.e. testing that each bit of our code does exactly what we intended it to do  - but it is not intended to test if what we intended to do was the right thing, or if the code we did not write(i.e. Jenkins and plugins) does the right thing. We would need more for that sort of testing, but so far, vast majority of the issues have been purely in our own code - and our primary goal is to make sure that any change we do to that code does not break builds for everyone using this shared code. That said, regardless of how you test your code, if you are not testing running it under CPS Transform, your tests are mostly worthless. There are many cases where CPS transformed code just does not behave the same way as non-transformed code. :-/

            Unassigned Unassigned
            jglick Jesse Glick
            Votes:
            120 Vote for this issue
            Watchers:
            136 Start watching this issue

              Created:
              Updated: