Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64846

Pipeline with Matrix doesn't see variables outside pipeline block

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major

      This is continuation of problem described in comment JENKINS-37984#comment.

      Example of pipeline:

      #!/usr/bin/env groovy
      
      //library("jenkins_shared_library@1.0.0")
      
      //@groovy.transform.Field
      String resourcePrefix = new Date().getTime().toString()
      
      //@groovy.transform.Field
      Map dockerParameters = [
          registry: "docker.example.com",
          registryType: "internal",
          images: [
              image1: [image: "image1", dockerfile: "Dockerfile1"],
              image2: [image: "image2", dockerfile: "Dockerfile2"]
          ]
      ]
      
      pipeline {
        agent any
        options { skipDefaultCheckout true }
        parameters {
          booleanParam defaultValue: true, description: 'Build & Push image1', name: 'image1'
          booleanParam defaultValue: true, description: 'Build & Push image2', name: 'image2'
        }
      
        stages {
          stage("Prepare") {
            options { skipDefaultCheckout true }
            failFast true
            parallel {
              stage('Test1') {
                steps {
                  // All variables available in simple stages and parallel blocks
                  echo "resourcePrefix: ${resourcePrefix}"
                  echo "dockerParameters: ${dockerParameters}"
                }
              }
              stage('Test2') {
                steps {
                  echo "resourcePrefix: ${resourcePrefix}"
                  echo "dockerParameters: ${dockerParameters}"
                }
              }
            }
          }
      
      
          stage("Docker") {
            options { skipDefaultCheckout true }
            matrix {
              axes {
                axis {
                  name 'COMPONENT'
                  // Note: these values are the same as described in dockerParameters and params
                  values 'image1', 'image2'
                }
              }
              stages {
                stage("Build") {
                  when {
                    beforeAgent true
                    expression { params[COMPONENT] == true }
                  }
                  // agent { kubernetes(k8sAgent(name: 'dind')) }
                  steps {
                    // Failing on resourcePrefix/dockerParameters, as it doesn't have Field annotation
                    // Question is: why variables are not available inside matrix?
      
                    echo "resourcePrefix: ${resourcePrefix}"
                    echo "dockerParameters: ${dockerParameters}"
      
                    // Here is one step as example:
                    //dockerBuild(
                    //    image: dockerParameters.images[COMPONENT].image,
                    //    dockerfile: dockerParameters.images[COMPONENT].dockerfile
                    //)
                  }
                }
              }
            }
          }
      
        }
      }
      

      The result is following (build #2 on screenshot):

      stage `Prepare` goes fine anyway - as expected.

      stage `Docker` fails (on each matrix stage) with the message:

      groovy.lang.MissingPropertyException: No such property: resourcePrefix for class: groovy.lang.Binding
      

      Until I do not add annotation: `@groovy.transform.Field` (build #3 on screenshot).

      The same with `dockerParameters`, where I have map of different values, which are similar and have some common values.

      Note: this is just example, there is parameters, which we use in different stages, and copy-pasting all of them to each stage is not appropriate solution - defining them as common/global outside of `pipeline` block is the only way to do it, isn't it?

      Any splitting params (described in PR #405) or experimental features was never enabled.

      Parameters like: `SCRIPT_SPLITTING_TRANSFORMATION` & `SCRIPT_SPLITTING_ALLOW_LOCAL_VARIABLES` were not defined at all, as experimental feature supposed to be disabled by default.

      Expected result:

      • Users are not forced to fix each pipeline (and branch) variable with `@groovy.transform.Field` annotation
      • Pipelines with matrix (and latest `pipeline-model-definition` plugin) is continue support of variables, defined outside of `pipeline`
      • Experimental features without breaking changes

          [JENKINS-64846] Pipeline with Matrix doesn't see variables outside pipeline block

          Liam Newman added a comment -

          moskovych
          Good catch. Working on fix.

          Liam Newman added a comment - moskovych Good catch. Working on fix.

          Liam Newman added a comment -

          The fix has been published. It included tests to prevent this from happen in the future.

          Liam Newman added a comment - The fix has been published. It included tests to prevent this from happen in the future.

          bitwiseman, thanks for so quick fix. Looks like it works fine.

          Oleh Moskovych added a comment - bitwiseman , thanks for so quick fix. Looks like it works fine.

          Joe Athman added a comment -

          bitwiseman we seem to be hitting this same issue but are using a declarative pipeline, do you think this fix should work for those as well? We are running the 1.8.4 version of the plugin and still see the problem. We have a multi-branch job, which uses a shared pipeline coming from a global library. The shared pipeline uses local variables within the call method. In the pipeline we have a stage which is a matrix stage, the options and agent settings seem to cause a problem for us and we get the missing property exception. Here's a relatively simplified look at what our pipeline looks like:

          def call(Map<String, Object> pipelineParams) {
              def cronString = pipelineParams.getOrDefault('cronString', "")
              def testTimeoutInMinutes = pipelineParams.getOrDefault('testTimeoutInMinutes', 30)
              // add 2 hours to testTimeoutInMinutes to provide a buffer for getting an executor
              def expandedBuildAndTestTimeoutMinutes = testTimeoutInMinutes + 120
              debugEcho("expandedBuildAndTestTimeoutMinutes=$expandedBuildAndTestTimeoutMinutes")
          
              def office365Options = getTeamsOptions(pipelineParams)
              def jenkinsAgentLabel = pipelineParams.getOrDefault('jenkinsAgentLabel', 'linux_docker')
              def dockerImageName = 'customimagename'
              def dockerMaxMemory = pipelineParams.getOrDefault('dockerMaxMemory', "2g")
              def dockerRunArgs = pipelineParams.getOrDefault('dockerRunArgs', "-m $dockerMaxMemory".toString())
          
              pipeline {
                  agent none
                  options {
                      office365ConnectorWebhooks([office365Options])
                      timestamps()
                      ansiColor('xterm')
                  }
                  triggers {
                      cron(cronString)
                  }
                  stages {
                      stage('Windows Tests') {
                          matrix {
                              axes {
                                  axis {
                                      name 'PLATFORM'
                                      values 'WINDOWS_10'
                                  }
                                  axis {
                                      name 'BROWSER'
                                      values 'CHROME', 'FIREFOX', 'EDGE'
                                  }
                              }
                              stages {
                                  stage('Execute Tests') {
                                      agent {
                                          docker {
                                              label jenkinsAgentLabel
                                              image dockerImageName
                                              args dockerRunArgs
                                          }
                                      }
                                      options { timeout(time: expandedBuildAndTestTimeoutMinutes, unit: 'MINUTES') }
                                      steps {
                                          echo 'here'
                                      }
                                  }
                              }
                          }
                      }
                  }
              }
          }
          

           

          Joe Athman added a comment - bitwiseman  we seem to be hitting this same issue but are using a declarative pipeline, do you think this fix should work for those as well? We are running the 1.8.4 version of the plugin and still see the problem. We have a multi-branch job, which uses a shared pipeline coming from a global library. The shared pipeline uses local variables within the call method. In the pipeline we have a stage which is a matrix stage, the options and agent settings seem to cause a problem for us and we get the missing property exception. Here's a relatively simplified look at what our pipeline looks like: def call(Map< String , Object > pipelineParams) { def cronString = pipelineParams.getOrDefault( 'cronString' , "") def testTimeoutInMinutes = pipelineParams.getOrDefault( 'testTimeoutInMinutes' , 30) // add 2 hours to testTimeoutInMinutes to provide a buffer for getting an executor def expandedBuildAndTestTimeoutMinutes = testTimeoutInMinutes + 120 debugEcho( "expandedBuildAndTestTimeoutMinutes=$expandedBuildAndTestTimeoutMinutes" ) def office365Options = getTeamsOptions(pipelineParams) def jenkinsAgentLabel = pipelineParams.getOrDefault( 'jenkinsAgentLabel' , 'linux_docker' ) def dockerImageName = 'customimagename' def dockerMaxMemory = pipelineParams.getOrDefault( 'dockerMaxMemory' , "2g" ) def dockerRunArgs = pipelineParams.getOrDefault( 'dockerRunArgs' , "-m $dockerMaxMemory" .toString()) pipeline { agent none options { office365ConnectorWebhooks([office365Options]) timestamps() ansiColor( 'xterm' ) } triggers { cron(cronString) } stages { stage( 'Windows Tests' ) { matrix { axes { axis { name 'PLATFORM' values 'WINDOWS_10' } axis { name 'BROWSER' values 'CHROME' , 'FIREFOX' , 'EDGE' } } stages { stage( 'Execute Tests' ) { agent { docker { label jenkinsAgentLabel image dockerImageName args dockerRunArgs } } options { timeout(time: expandedBuildAndTestTimeoutMinutes, unit: 'MINUTES' ) } steps { echo 'here' } } } } } } } }  

          Liam Newman added a comment -

          jjathman
          from https://issues.jenkins.io/browse/JENKINS-37984:
          "This workaround generally does NOT work if the pipeline directive inside a shared library method. If this is a scenario you want, please come join the pipeline authoring SIG and we can discuss."

          Considering this feature is experimental still it might be early to file a bug on it, but then again tracking this limitation is probably a good idea. At very least, we probably want it to not break this scenario and/or give meaningful feedback.

          Liam Newman added a comment - jjathman from https://issues.jenkins.io/browse/JENKINS-37984: "This workaround generally does NOT work if the pipeline directive inside a shared library method. If this is a scenario you want, please come join the pipeline authoring SIG and we can discuss." Considering this feature is experimental still it might be early to file a bug on it, but then again tracking this limitation is probably a good idea. At very least, we probably want it to not break this scenario and/or give meaningful feedback.

            bitwiseman Liam Newman
            moskovych Oleh Moskovych
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: