Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-33925

Test framework for Jenkinsfile

    XMLWordPrintable

Details

    Description

      It would be desirable to have a standard mechanism for testing Pipeline scripts without running them on a production server. There are two competing suggestions:

      Mock framework

      Inspired by Job DSL (example).

      We could set up a GroovyShell in which step functions and global variables were predefined as mocks (in a fashion similar to Powermock, but easier in Groovy given its dynamic nature), and stub out the expected return value / exception for each, with some standard predefinitions such as for currentBuild.

      Ideally the shell would be CPS-transformed, with the program state serialized and then reloaded between every continuation (though this might involve a lot of code duplication with workflow-cps).

      Should be easy to pass it through the Groovy sandbox (if requested), though the live Whitelist.all from Jenkins would be unavailable, so we would be limited to known static whitelists, the @Whitelisted annotation, and perhaps some custom additions.

      Quick and flexible, but low fidelity to real behavior.

      JenkinsRule-style

      Use an embedded Jenkins server, as per JenkinsRule in jenkins-test-harness, and actually create a WorkflowJob with the specified definition. Can use for example mock-slave to create nodes.

      Need to have a "dry-run" flag so that attempts to do things like deploy artifacts or send email do not really take action. This could perhaps be a general API in Jenkins core, as it would be useful also for test instances (shadows of production servers), acceptance-test-harness, etc.

      Slower to run (seconds per test case rather than milliseconds), and trickier to set up, but much more realistic coverage. The tests for Pipeline (and Pipeline steps) themselves use this technique.

      Attachments

        Issue Links

          Activity

            jglick Jesse Glick created issue -
            jglick Jesse Glick made changes -
            Field Original Value New Value
            Description It would be desirable to have a standard mechanism for testing Pipeline scripts without running them on a production server. There are two competing suggestions:

            h3. Mock framework

            Inspired by Job DSL ([example|https://github.com/sheehan/job-dsl-gradle-example/blob/e240056da6691bf0a2fdc99e5aab33bc49e42b2f/src/test/groovy/com/dslexample/GrailsCiJobBuilderSpec.groovy#L34-L49]).

            We could set up a {{GroovyShell}} in which step functions and global variables were predefined as mocks (in a fashion similar to Powermock, but easier in Groovy given its dynamic nature), and stub out the expected return value / exception for each, with some standard predefinitions such as for {{currentBuild}}.

            Ideally the shell would be CPS-transformed, with the program state serialized and then reloaded between every continuation (though this might involve a lot of code duplication with {{workflow-cps}}).

            Should be easy to pass it through the Groovy sandbox (if requested), though the live {{Whitelist.all}} from Jenkins would be unavailable, so we would be limited to known static whitelists, the {{@Whitelisted}} annotation, and perhaps some custom additions.

            Quick and flexible, but low fidelity to real behavior.

            h3. {{JenkinsRule}}-style

            Use an embedded Jenkins server, as per {{JenkinsRule}} in {{jenkins-test-harness}}, and actually create a {{WorkflowJob}} with the specified definition. Can use for example {{mock-slave}} to create nodes.

            Need to have a "dry-run" flag so that attempts to do things like deploy artifacts or send email do not really take action. This could perhaps be a general API in Jenkins core, as it would be useful also for test instances (shadows of production servers), {{acceptance-test-harness}}, etc.

            Slower to run (seconds per test case rather than milliseconds), and trickier to set up, but much more realistic coverage.
            It would be desirable to have a standard mechanism for testing Pipeline scripts without running them on a production server. There are two competing suggestions:

            h3. Mock framework

            Inspired by Job DSL ([example|https://github.com/sheehan/job-dsl-gradle-example/blob/e240056da6691bf0a2fdc99e5aab33bc49e42b2f/src/test/groovy/com/dslexample/GrailsCiJobBuilderSpec.groovy#L34-L49]).

            We could set up a {{GroovyShell}} in which step functions and global variables were predefined as mocks (in a fashion similar to Powermock, but easier in Groovy given its dynamic nature), and stub out the expected return value / exception for each, with some standard predefinitions such as for {{currentBuild}}.

            Ideally the shell would be CPS-transformed, with the program state serialized and then reloaded between every continuation (though this might involve a lot of code duplication with {{workflow-cps}}).

            Should be easy to pass it through the Groovy sandbox (if requested), though the live {{Whitelist.all}} from Jenkins would be unavailable, so we would be limited to known static whitelists, the {{@Whitelisted}} annotation, and perhaps some custom additions.

            Quick and flexible, but low fidelity to real behavior.

            h3. {{JenkinsRule}}-style

            Use an embedded Jenkins server, as per {{JenkinsRule}} in {{jenkins-test-harness}}, and actually create a {{WorkflowJob}} with the specified definition. Can use for example {{mock-slave}} to create nodes.

            Need to have a "dry-run" flag so that attempts to do things like deploy artifacts or send email do not really take action. This could perhaps be a general API in Jenkins core, as it would be useful also for test instances (shadows of production servers), {{acceptance-test-harness}}, etc.

            Slower to run (seconds per test case rather than milliseconds), and trickier to set up, but much more realistic coverage. The tests for Pipeline (and Pipeline steps) themselves use this technique.
            hrmpw Patrick Wolf made changes -
            Labels testing 3.0 testing
            hrmpw Patrick Wolf made changes -
            Labels 3.0 testing followup testing
            jglick Jesse Glick made changes -
            Epic Link JENKINS-35396 [ 171189 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 169936 ] JNJira + In-Review [ 183698 ]
            abayer Andrew Bayer made changes -
            Component/s pipeline-general [ 21692 ]
            abayer Andrew Bayer made changes -
            Component/s workflow-plugin [ 18820 ]
            jglick Jesse Glick made changes -
            Component/s workflow-cps-plugin [ 21713 ]
            Component/s pipeline [ 21692 ]
            zhelyan Zhelyan Panchev made changes -
            Link This issue relates to JENKINS-40285 [ JENKINS-40285 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "JenkinsPipelineUnit (Web Link)" [ 15606 ]
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            cloudbees CloudBees Inc. made changes -
            Remote Link This issue links to "CloudBees Internal CD-284 (Web Link)" [ 19070 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "Jenkinsfile Runner (Web Link)" [ 20202 ]
            jequals5 Marky Jackson made changes -
            Epic Link JENKINS-35396 [ 171189 ]

            People

              Unassigned Unassigned
              jglick Jesse Glick
              Votes:
              116 Vote for this issue
              Watchers:
              134 Start watching this issue

              Dates

                Created:
                Updated: