Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64846

Pipeline with Matrix doesn't see variables outside pipeline block

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Environment:
    • Similar Issues:

      Description

      This is continuation of problem described in comment JENKINS-37984#comment.

      Example of pipeline:

      #!/usr/bin/env groovy
      
      //library("jenkins_shared_library@1.0.0")
      
      //@groovy.transform.Field
      String resourcePrefix = new Date().getTime().toString()
      
      //@groovy.transform.Field
      Map dockerParameters = [
          registry: "docker.example.com",
          registryType: "internal",
          images: [
              image1: [image: "image1", dockerfile: "Dockerfile1"],
              image2: [image: "image2", dockerfile: "Dockerfile2"]
          ]
      ]
      
      pipeline {
        agent any
        options { skipDefaultCheckout true }
        parameters {
          booleanParam defaultValue: true, description: 'Build & Push image1', name: 'image1'
          booleanParam defaultValue: true, description: 'Build & Push image2', name: 'image2'
        }
      
        stages {
          stage("Prepare") {
            options { skipDefaultCheckout true }
            failFast true
            parallel {
              stage('Test1') {
                steps {
                  // All variables available in simple stages and parallel blocks
                  echo "resourcePrefix: ${resourcePrefix}"
                  echo "dockerParameters: ${dockerParameters}"
                }
              }
              stage('Test2') {
                steps {
                  echo "resourcePrefix: ${resourcePrefix}"
                  echo "dockerParameters: ${dockerParameters}"
                }
              }
            }
          }
      
      
          stage("Docker") {
            options { skipDefaultCheckout true }
            matrix {
              axes {
                axis {
                  name 'COMPONENT'
                  // Note: these values are the same as described in dockerParameters and params
                  values 'image1', 'image2'
                }
              }
              stages {
                stage("Build") {
                  when {
                    beforeAgent true
                    expression { params[COMPONENT] == true }
                  }
                  // agent { kubernetes(k8sAgent(name: 'dind')) }
                  steps {
                    // Failing on resourcePrefix/dockerParameters, as it doesn't have Field annotation
                    // Question is: why variables are not available inside matrix?
      
                    echo "resourcePrefix: ${resourcePrefix}"
                    echo "dockerParameters: ${dockerParameters}"
      
                    // Here is one step as example:
                    //dockerBuild(
                    //    image: dockerParameters.images[COMPONENT].image,
                    //    dockerfile: dockerParameters.images[COMPONENT].dockerfile
                    //)
                  }
                }
              }
            }
          }
      
        }
      }
      

      The result is following (build #2 on screenshot):

      stage `Prepare` goes fine anyway - as expected.

      stage `Docker` fails (on each matrix stage) with the message:

      groovy.lang.MissingPropertyException: No such property: resourcePrefix for class: groovy.lang.Binding
      

      Until I do not add annotation: `@groovy.transform.Field` (build #3 on screenshot).

      The same with `dockerParameters`, where I have map of different values, which are similar and have some common values.

      Note: this is just example, there is parameters, which we use in different stages, and copy-pasting all of them to each stage is not appropriate solution - defining them as common/global outside of `pipeline` block is the only way to do it, isn't it?

      Any splitting params (described in PR #405) or experimental features was never enabled.

      Parameters like: `SCRIPT_SPLITTING_TRANSFORMATION` & `SCRIPT_SPLITTING_ALLOW_LOCAL_VARIABLES` were not defined at all, as experimental feature supposed to be disabled by default.

      Expected result:

      • Users are not forced to fix each pipeline (and branch) variable with `@groovy.transform.Field` annotation
      • Pipelines with matrix (and latest `pipeline-model-definition` plugin) is continue support of variables, defined outside of `pipeline`
      • Experimental features without breaking changes

        Attachments

          Issue Links

            Activity

            moskovych Oleh Moskovych created issue -
            moskovych Oleh Moskovych made changes -
            Field Original Value New Value
            Description This is continuation of problem described in comment[JENKINS-37984#comment|https://issues.jenkins.io/browse/JENKINS-37984?focusedCommentId=404501&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-404501].

            Example of pipeline:

            {code:groovy}
            #!/usr/bin/env groovy

            //library("jenkins_shared_library@1.0.0")

            //@groovy.transform.Field
            String resourcePrefix = new Date().getTime().toString()

            //@groovy.transform.Field
            Map dockerParameters = [
                registry: "docker.example.com",
                registryType: "internal",
                images: [
                    image1: [image: "image1", dockerfile: "Dockerfile1"],
                    image2: [image: "image2", dockerfile: "Dockerfile2"]
                ]
            ]

            pipeline {
              agent any
              options { skipDefaultCheckout true }
              parameters {
                booleanParam defaultValue: true, description: 'Build & Push image1', name: 'image1'
                booleanParam defaultValue: true, description: 'Build & Push image2', name: 'image2'
              }

              stages {
                stage("Prepare") {
                  options { skipDefaultCheckout true }
                  failFast true
                  parallel {
                    stage('Test1') {
                      steps {
                        // All variables available in simple stages and parallel blocks
                        echo "resourcePrefix: ${resourcePrefix}"
                        echo "dockerParameters: ${dockerParameters}"
                      }
                    }
                    stage('Test2') {
                      steps {
                        echo "resourcePrefix: ${resourcePrefix}"
                        echo "dockerParameters: ${dockerParameters}"
                      }
                    }
                  }
                }


                stage("Docker") {
                  options { skipDefaultCheckout true }
                  matrix {
                    axes {
                      axis {
                        name 'COMPONENT'
                        // Note: these values are the same as described in dockerParameters and params
                        values 'image1', 'image2'
                      }
                    }
                    stages {
                      stage("Build") {
                        when {
                          beforeAgent true
                          expression { params[COMPONENT] == true }
                        }
                        // agent { kubernetes(k8sAgent(name: 'dind')) }
                        steps {
                          // Failing on resourcePrefix/dockerParameters, as it doesn't have Field annotation
                          // Question is: why variables are not available inside matrix?

                          echo "resourcePrefix: ${resourcePrefix}"
                          echo "dockerParameters: ${dockerParameters}"

                          // Here is one step as example:
                          //dockerBuild(
                          // image: dockerParameters.images[COMPONENT].image,
                          // dockerfile: dockerParameters.images[COMPONENT].dockerfile
                          //)
                        }
                      }
                    }
                  }
                }

              }
            }
            {code}


            The result is following (build #2 on screenshot):

            stage `Prepare` goes fine anyway - as expected.

            stage `Docker` fails (on each matrix stage) with the message:


            {code:groovy}
            groovy.lang.MissingPropertyException: No such property: resourcePrefix for class: groovy.lang.Binding
            {code}

            Until I do not add annotation: `@groovy.transform.Field` (build #3 on screenshot).

            The same with `dockerParameters`, where I have map of different values, which are similar and have some common values.

            Note: this is just example, there is parameters, which we use in different stages, and copy-pasting all of them to each stage is not appropriate solution - defining them as common/global outside of `pipeline` block is the only way to do it, isn't it?

            Any splitting params ([described in PR #405|https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/405]) or experimental features was never enabled.

            Parameters like: `SCRIPT_SPLITTING_TRANSFORMATION` & `SCRIPT_SPLITTING_ALLOW_LOCAL_VARIABLES` were not defined at all, as experimental feature supposed to be disabled by default.


            Expected result:
            - Users are not forced to fix each pipeline (and branch) variable with `@groovy.transform.Field` annotation
            - Pipelines with matrix (and latest `pipeline-model-definition` plugin) is continue support of variables, defined outside of `pipeline`
            - Experimental features without breaking changes
            This is continuation of problem described in comment [JENKINS-37984#comment|https://issues.jenkins.io/browse/JENKINS-37984?focusedCommentId=404501&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-404501].

            Example of pipeline:

            {code:groovy}
            #!/usr/bin/env groovy

            //library("jenkins_shared_library@1.0.0")

            //@groovy.transform.Field
            String resourcePrefix = new Date().getTime().toString()

            //@groovy.transform.Field
            Map dockerParameters = [
                registry: "docker.example.com",
                registryType: "internal",
                images: [
                    image1: [image: "image1", dockerfile: "Dockerfile1"],
                    image2: [image: "image2", dockerfile: "Dockerfile2"]
                ]
            ]

            pipeline {
              agent any
              options { skipDefaultCheckout true }
              parameters {
                booleanParam defaultValue: true, description: 'Build & Push image1', name: 'image1'
                booleanParam defaultValue: true, description: 'Build & Push image2', name: 'image2'
              }

              stages {
                stage("Prepare") {
                  options { skipDefaultCheckout true }
                  failFast true
                  parallel {
                    stage('Test1') {
                      steps {
                        // All variables available in simple stages and parallel blocks
                        echo "resourcePrefix: ${resourcePrefix}"
                        echo "dockerParameters: ${dockerParameters}"
                      }
                    }
                    stage('Test2') {
                      steps {
                        echo "resourcePrefix: ${resourcePrefix}"
                        echo "dockerParameters: ${dockerParameters}"
                      }
                    }
                  }
                }


                stage("Docker") {
                  options { skipDefaultCheckout true }
                  matrix {
                    axes {
                      axis {
                        name 'COMPONENT'
                        // Note: these values are the same as described in dockerParameters and params
                        values 'image1', 'image2'
                      }
                    }
                    stages {
                      stage("Build") {
                        when {
                          beforeAgent true
                          expression { params[COMPONENT] == true }
                        }
                        // agent { kubernetes(k8sAgent(name: 'dind')) }
                        steps {
                          // Failing on resourcePrefix/dockerParameters, as it doesn't have Field annotation
                          // Question is: why variables are not available inside matrix?

                          echo "resourcePrefix: ${resourcePrefix}"
                          echo "dockerParameters: ${dockerParameters}"

                          // Here is one step as example:
                          //dockerBuild(
                          // image: dockerParameters.images[COMPONENT].image,
                          // dockerfile: dockerParameters.images[COMPONENT].dockerfile
                          //)
                        }
                      }
                    }
                  }
                }

              }
            }
            {code}


            The result is following (build #2 on screenshot):

            stage `Prepare` goes fine anyway - as expected.

            stage `Docker` fails (on each matrix stage) with the message:


            {code:groovy}
            groovy.lang.MissingPropertyException: No such property: resourcePrefix for class: groovy.lang.Binding
            {code}

            Until I do not add annotation: `@groovy.transform.Field` (build #3 on screenshot).

            The same with `dockerParameters`, where I have map of different values, which are similar and have some common values.

            Note: this is just example, there is parameters, which we use in different stages, and copy-pasting all of them to each stage is not appropriate solution - defining them as common/global outside of `pipeline` block is the only way to do it, isn't it?

            Any splitting params ([described in PR #405|https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/405]) or experimental features was never enabled.

            Parameters like: `SCRIPT_SPLITTING_TRANSFORMATION` & `SCRIPT_SPLITTING_ALLOW_LOCAL_VARIABLES` were not defined at all, as experimental feature supposed to be disabled by default.


            h3. Expected result:
            - Users are not forced to fix each pipeline (and branch) variable with `@groovy.transform.Field` annotation
            - Pipelines with matrix (and latest `pipeline-model-definition` plugin) is continue support of variables, defined outside of `pipeline`
            - Experimental features without breaking changes
            moskovych Oleh Moskovych made changes -
            Attachment Jenkinsfile [ 53978 ]
            moskovych Oleh Moskovych made changes -
            Link This issue is related to JENKINS-37984 [ JENKINS-37984 ]
            moskovych Oleh Moskovych made changes -
            Priority Minor [ 4 ] Major [ 3 ]
            bitwiseman Liam Newman made changes -
            Assignee Liam Newman [ bitwiseman ]
            bitwiseman Liam Newman made changes -

              People

              Assignee:
              bitwiseman Liam Newman
              Reporter:
              moskovych Oleh Moskovych
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: