Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64846

Pipeline with Matrix doesn't see variables outside pipeline block

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed

    Description

      This is continuation of problem described in comment JENKINS-37984#comment.

      Example of pipeline:

      #!/usr/bin/env groovy
      
      //library("jenkins_shared_library@1.0.0")
      
      //@groovy.transform.Field
      String resourcePrefix = new Date().getTime().toString()
      
      //@groovy.transform.Field
      Map dockerParameters = [
          registry: "docker.example.com",
          registryType: "internal",
          images: [
              image1: [image: "image1", dockerfile: "Dockerfile1"],
              image2: [image: "image2", dockerfile: "Dockerfile2"]
          ]
      ]
      
      pipeline {
        agent any
        options { skipDefaultCheckout true }
        parameters {
          booleanParam defaultValue: true, description: 'Build & Push image1', name: 'image1'
          booleanParam defaultValue: true, description: 'Build & Push image2', name: 'image2'
        }
      
        stages {
          stage("Prepare") {
            options { skipDefaultCheckout true }
            failFast true
            parallel {
              stage('Test1') {
                steps {
                  // All variables available in simple stages and parallel blocks
                  echo "resourcePrefix: ${resourcePrefix}"
                  echo "dockerParameters: ${dockerParameters}"
                }
              }
              stage('Test2') {
                steps {
                  echo "resourcePrefix: ${resourcePrefix}"
                  echo "dockerParameters: ${dockerParameters}"
                }
              }
            }
          }
      
      
          stage("Docker") {
            options { skipDefaultCheckout true }
            matrix {
              axes {
                axis {
                  name 'COMPONENT'
                  // Note: these values are the same as described in dockerParameters and params
                  values 'image1', 'image2'
                }
              }
              stages {
                stage("Build") {
                  when {
                    beforeAgent true
                    expression { params[COMPONENT] == true }
                  }
                  // agent { kubernetes(k8sAgent(name: 'dind')) }
                  steps {
                    // Failing on resourcePrefix/dockerParameters, as it doesn't have Field annotation
                    // Question is: why variables are not available inside matrix?
      
                    echo "resourcePrefix: ${resourcePrefix}"
                    echo "dockerParameters: ${dockerParameters}"
      
                    // Here is one step as example:
                    //dockerBuild(
                    //    image: dockerParameters.images[COMPONENT].image,
                    //    dockerfile: dockerParameters.images[COMPONENT].dockerfile
                    //)
                  }
                }
              }
            }
          }
      
        }
      }
      

      The result is following (build #2 on screenshot):

      stage `Prepare` goes fine anyway - as expected.

      stage `Docker` fails (on each matrix stage) with the message:

      groovy.lang.MissingPropertyException: No such property: resourcePrefix for class: groovy.lang.Binding
      

      Until I do not add annotation: `@groovy.transform.Field` (build #3 on screenshot).

      The same with `dockerParameters`, where I have map of different values, which are similar and have some common values.

      Note: this is just example, there is parameters, which we use in different stages, and copy-pasting all of them to each stage is not appropriate solution - defining them as common/global outside of `pipeline` block is the only way to do it, isn't it?

      Any splitting params (described in PR #405) or experimental features was never enabled.

      Parameters like: `SCRIPT_SPLITTING_TRANSFORMATION` & `SCRIPT_SPLITTING_ALLOW_LOCAL_VARIABLES` were not defined at all, as experimental feature supposed to be disabled by default.

      Expected result:

      • Users are not forced to fix each pipeline (and branch) variable with `@groovy.transform.Field` annotation
      • Pipelines with matrix (and latest `pipeline-model-definition` plugin) is continue support of variables, defined outside of `pipeline`
      • Experimental features without breaking changes

      Attachments

        Issue Links

          Activity

            bitwiseman Liam Newman added a comment -

            moskovych
            Good catch. Working on fix.

            bitwiseman Liam Newman added a comment - moskovych Good catch. Working on fix.
            bitwiseman Liam Newman added a comment -

            The fix has been published. It included tests to prevent this from happen in the future.

            bitwiseman Liam Newman added a comment - The fix has been published. It included tests to prevent this from happen in the future.

            bitwiseman, thanks for so quick fix. Looks like it works fine.

            moskovych Oleh Moskovych added a comment - bitwiseman , thanks for so quick fix. Looks like it works fine.
            jjathman Joe Athman added a comment -

            bitwiseman we seem to be hitting this same issue but are using a declarative pipeline, do you think this fix should work for those as well? We are running the 1.8.4 version of the plugin and still see the problem. We have a multi-branch job, which uses a shared pipeline coming from a global library. The shared pipeline uses local variables within the call method. In the pipeline we have a stage which is a matrix stage, the options and agent settings seem to cause a problem for us and we get the missing property exception. Here's a relatively simplified look at what our pipeline looks like:

            def call(Map<String, Object> pipelineParams) {
                def cronString = pipelineParams.getOrDefault('cronString', "")
                def testTimeoutInMinutes = pipelineParams.getOrDefault('testTimeoutInMinutes', 30)
                // add 2 hours to testTimeoutInMinutes to provide a buffer for getting an executor
                def expandedBuildAndTestTimeoutMinutes = testTimeoutInMinutes + 120
                debugEcho("expandedBuildAndTestTimeoutMinutes=$expandedBuildAndTestTimeoutMinutes")
            
                def office365Options = getTeamsOptions(pipelineParams)
                def jenkinsAgentLabel = pipelineParams.getOrDefault('jenkinsAgentLabel', 'linux_docker')
                def dockerImageName = 'customimagename'
                def dockerMaxMemory = pipelineParams.getOrDefault('dockerMaxMemory', "2g")
                def dockerRunArgs = pipelineParams.getOrDefault('dockerRunArgs', "-m $dockerMaxMemory".toString())
            
                pipeline {
                    agent none
                    options {
                        office365ConnectorWebhooks([office365Options])
                        timestamps()
                        ansiColor('xterm')
                    }
                    triggers {
                        cron(cronString)
                    }
                    stages {
                        stage('Windows Tests') {
                            matrix {
                                axes {
                                    axis {
                                        name 'PLATFORM'
                                        values 'WINDOWS_10'
                                    }
                                    axis {
                                        name 'BROWSER'
                                        values 'CHROME', 'FIREFOX', 'EDGE'
                                    }
                                }
                                stages {
                                    stage('Execute Tests') {
                                        agent {
                                            docker {
                                                label jenkinsAgentLabel
                                                image dockerImageName
                                                args dockerRunArgs
                                            }
                                        }
                                        options { timeout(time: expandedBuildAndTestTimeoutMinutes, unit: 'MINUTES') }
                                        steps {
                                            echo 'here'
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
            

             

            jjathman Joe Athman added a comment - bitwiseman  we seem to be hitting this same issue but are using a declarative pipeline, do you think this fix should work for those as well? We are running the 1.8.4 version of the plugin and still see the problem. We have a multi-branch job, which uses a shared pipeline coming from a global library. The shared pipeline uses local variables within the call method. In the pipeline we have a stage which is a matrix stage, the options and agent settings seem to cause a problem for us and we get the missing property exception. Here's a relatively simplified look at what our pipeline looks like: def call(Map< String , Object > pipelineParams) { def cronString = pipelineParams.getOrDefault( 'cronString' , "") def testTimeoutInMinutes = pipelineParams.getOrDefault( 'testTimeoutInMinutes' , 30) // add 2 hours to testTimeoutInMinutes to provide a buffer for getting an executor def expandedBuildAndTestTimeoutMinutes = testTimeoutInMinutes + 120 debugEcho( "expandedBuildAndTestTimeoutMinutes=$expandedBuildAndTestTimeoutMinutes" ) def office365Options = getTeamsOptions(pipelineParams) def jenkinsAgentLabel = pipelineParams.getOrDefault( 'jenkinsAgentLabel' , 'linux_docker' ) def dockerImageName = 'customimagename' def dockerMaxMemory = pipelineParams.getOrDefault( 'dockerMaxMemory' , "2g" ) def dockerRunArgs = pipelineParams.getOrDefault( 'dockerRunArgs' , "-m $dockerMaxMemory" .toString()) pipeline { agent none options { office365ConnectorWebhooks([office365Options]) timestamps() ansiColor( 'xterm' ) } triggers { cron(cronString) } stages { stage( 'Windows Tests' ) { matrix { axes { axis { name 'PLATFORM' values 'WINDOWS_10' } axis { name 'BROWSER' values 'CHROME' , 'FIREFOX' , 'EDGE' } } stages { stage( 'Execute Tests' ) { agent { docker { label jenkinsAgentLabel image dockerImageName args dockerRunArgs } } options { timeout(time: expandedBuildAndTestTimeoutMinutes, unit: 'MINUTES' ) } steps { echo 'here' } } } } } } } }  
            bitwiseman Liam Newman added a comment -

            jjathman
            from https://issues.jenkins.io/browse/JENKINS-37984:
            "This workaround generally does NOT work if the pipeline directive inside a shared library method. If this is a scenario you want, please come join the pipeline authoring SIG and we can discuss."

            Considering this feature is experimental still it might be early to file a bug on it, but then again tracking this limitation is probably a good idea. At very least, we probably want it to not break this scenario and/or give meaningful feedback.

            bitwiseman Liam Newman added a comment - jjathman from https://issues.jenkins.io/browse/JENKINS-37984: "This workaround generally does NOT work if the pipeline directive inside a shared library method. If this is a scenario you want, please come join the pipeline authoring SIG and we can discuss." Considering this feature is experimental still it might be early to file a bug on it, but then again tracking this limitation is probably a good idea. At very least, we probably want it to not break this scenario and/or give meaningful feedback.

            People

              bitwiseman Liam Newman
              moskovych Oleh Moskovych
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: