Details
-
New Feature
-
Status: Open (View Workflow)
-
Major
-
Resolution: Unresolved
Description
It would be desirable to have a standard mechanism for testing Pipeline scripts without running them on a production server. There are two competing suggestions:
Mock framework
Inspired by Job DSL (example).
We could set up a GroovyShell in which step functions and global variables were predefined as mocks (in a fashion similar to Powermock, but easier in Groovy given its dynamic nature), and stub out the expected return value / exception for each, with some standard predefinitions such as for currentBuild.
Ideally the shell would be CPS-transformed, with the program state serialized and then reloaded between every continuation (though this might involve a lot of code duplication with workflow-cps).
Should be easy to pass it through the Groovy sandbox (if requested), though the live Whitelist.all from Jenkins would be unavailable, so we would be limited to known static whitelists, the @Whitelisted annotation, and perhaps some custom additions.
Quick and flexible, but low fidelity to real behavior.
JenkinsRule-style
Use an embedded Jenkins server, as per JenkinsRule in jenkins-test-harness, and actually create a WorkflowJob with the specified definition. Can use for example mock-slave to create nodes.
Need to have a "dry-run" flag so that attempts to do things like deploy artifacts or send email do not really take action. This could perhaps be a general API in Jenkins core, as it would be useful also for test instances (shadows of production servers), acceptance-test-harness, etc.
Slower to run (seconds per test case rather than milliseconds), and trickier to set up, but much more realistic coverage. The tests for Pipeline (and Pipeline steps) themselves use this technique.
Attachments
Issue Links
- relates to
-
JENKINS-40285 global libraries test harness
-
- Resolved
-
- links to
Activity
mlasevich, thank you for the code. I checked it quickly - so what is the difference between your CPSUtil and JenkinsPipelineUnit https://github.com/jenkinsci/JenkinsPipelineUnit/tree/master/src/main/groovy/com/lesfurets/jenkins/unit/cps ? As far I know it's already using CPS to run the tests... But, for example, when it comes to groovy specifics (like the one I found there: https://github.com/griddynamics/mpl/pull/49#issuecomment-551287490 ) - CPS & Jenkins pipeline starts to behave differently.
I think it's quite critical to actually use Jenkins & plugins (security, steps...) for testing, that's why, for example I use the jenkins class FunctionCallEnv override and jenkins harness to be able to intercept the calls for mocking.
So could you please describe how your script is closer to Jenkins than the JenkinsPipelineUnit's one, maybe I got it wrong?
Michael Lasevich
added a comment - sparshev - My code is far simpler than what JenkinsPipelineUnit does. It simply allows you to execute CPS code - that is all - remainder is up to the testing framework.
JenkinsPipelineUnit (and JenkinsSpock) tries to do a lot of other things, like mocking the steps and what not. Those are great things, but I had hard time getting either to play nice with CPS Transformed code. At best they allowed loading scripts directly from groovy files, but that seems to bypass the classloaders and makes it very difficult to actually unit test anything - at best you can execute the entire `vars` code as a script, but not isolate and call specific methods in specific classes. We have a great deal of actual classes and proper code in out libraries, and I found no easy way to create an instance of a class and test it in either framework. That does not mean there isn't a way, I just have not found it.
My approach is far more straight forward/simplistic. I enable CPS Transform on compile for all code, so all my classes are already transformed, and then just run my unit tests like I would with any other code. If I need to test code that is CPS transformed, I have to call it via CPSUtils but that is it. I am not running any of the actual steps or any Jenkins code beyond groovy-cps transforms - I can stub out the calls and assume that the actual steps are unit tested in the plugin code that provides them. It is not an end-to-end or integration testing - it is pure unit testing at that point.
Ideally I would love to integrate this functionality with JenkinsPipelineUnit functionality - but for now I am just using pure Spock tests and just mock any steps I need
Michael Lasevich
added a comment - sparshev - My code is far simpler than what JenkinsPipelineUnit does. It simply allows you to execute CPS code - that is all - remainder is up to the testing framework.
JenkinsPipelineUnit (and JenkinsSpock) tries to do a lot of other things, like mocking the steps and what not. Those are great things, but I had hard time getting either to play nice with CPS Transformed code. At best they allowed loading scripts directly from groovy files, but that seems to bypass the classloaders and makes it very difficult to actually unit test anything - at best you can execute the entire `vars` code as a script, but not isolate and call specific methods in specific classes. We have a great deal of actual classes and proper code in out libraries, and I found no easy way to create an instance of a class and test it in either framework. That does not mean there isn't a way, I just have not found it.
My approach is far more straight forward/simplistic. I enable CPS Transform on compile for all code, so all my classes are already transformed, and then just run my unit tests like I would with any other code. If I need to test code that is CPS transformed, I have to call it via CPSUtils but that is it. I am not running any of the actual steps or any Jenkins code beyond groovy-cps transforms - I can stub out the calls and assume that the actual steps are unit tested in the plugin code that provides them. It is not an end-to-end or integration testing - it is pure unit testing at that point.
Ideally I would love to integrate this functionality with JenkinsPipelineUnit functionality - but for now I am just using pure Spock tests and just mock any steps I need Ok, got it... Unfortunately it is not enough to complete the blackbox pipeline testing due to the reasons I described before: Jenkins pipeline is far more complex than CPS. It introduces restrictions that is not covered via pure CPS, but if this implementation is working for your case - that's great!)
Michael Lasevich
added a comment - Yeah, this is for pure unit testing and only of the actual code we wrote - i.e. testing that each bit of our code does exactly what we intended it to do - but it is not intended to test if what we intended to do was the right thing, or if the code we did not write(i.e. Jenkins and plugins) does the right thing. We would need more for that sort of testing, but so far, vast majority of the issues have been purely in our own code - and our primary goal is to make sure that any change we do to that code does not break builds for everyone using this shared code.
That said, regardless of how you test your code, if you are not testing running it under CPS Transform, your tests are mostly worthless. There are many cases where CPS transformed code just does not behave the same way as non-transformed code. :-/
Michael Lasevich
added a comment - Yeah, this is for pure unit testing and only of the actual code we wrote - i.e. testing that each bit of our code does exactly what we intended it to do - but it is not intended to test if what we intended to do was the right thing, or if the code we did not write(i.e. Jenkins and plugins) does the right thing. We would need more for that sort of testing, but so far, vast majority of the issues have been purely in our own code - and our primary goal is to make sure that any change we do to that code does not break builds for everyone using this shared code.
That said, regardless of how you test your code, if you are not testing running it under CPS Transform, your tests are mostly worthless. There are many cases where CPS transformed code just does not behave the same way as non-transformed code. :-/
sparshev For the moment I just pushed a (slightly) cleaned up version into github here:
https://github.com/mlasevich/jenkins-pipeline-library-reference/blob/main/test/src/support/cps/CPSUtils.groovy
it boils down to two key static methods (heavily inspired by groovy-cps's own unit tests
Both are primarily intended for use in Unit Tests. Would be nice to have this be part of standard library to avoid the constant cut-n-paste (and it is a bit tiny to be its own library
I also had a wrapper that can wrap any object and run invokeCPSMethod transparently - but it proved to be a bit more complicated and unstable, so I removed it for time being