Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64846

Pipeline with Matrix doesn't see variables outside pipeline block

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Major
    • Resolution: Fixed
    • Environment:
    • Similar Issues:

      Description

      This is continuation of problem described in comment JENKINS-37984#comment.

      Example of pipeline:

      #!/usr/bin/env groovy
      
      //library("jenkins_shared_library@1.0.0")
      
      //@groovy.transform.Field
      String resourcePrefix = new Date().getTime().toString()
      
      //@groovy.transform.Field
      Map dockerParameters = [
          registry: "docker.example.com",
          registryType: "internal",
          images: [
              image1: [image: "image1", dockerfile: "Dockerfile1"],
              image2: [image: "image2", dockerfile: "Dockerfile2"]
          ]
      ]
      
      pipeline {
        agent any
        options { skipDefaultCheckout true }
        parameters {
          booleanParam defaultValue: true, description: 'Build & Push image1', name: 'image1'
          booleanParam defaultValue: true, description: 'Build & Push image2', name: 'image2'
        }
      
        stages {
          stage("Prepare") {
            options { skipDefaultCheckout true }
            failFast true
            parallel {
              stage('Test1') {
                steps {
                  // All variables available in simple stages and parallel blocks
                  echo "resourcePrefix: ${resourcePrefix}"
                  echo "dockerParameters: ${dockerParameters}"
                }
              }
              stage('Test2') {
                steps {
                  echo "resourcePrefix: ${resourcePrefix}"
                  echo "dockerParameters: ${dockerParameters}"
                }
              }
            }
          }
      
      
          stage("Docker") {
            options { skipDefaultCheckout true }
            matrix {
              axes {
                axis {
                  name 'COMPONENT'
                  // Note: these values are the same as described in dockerParameters and params
                  values 'image1', 'image2'
                }
              }
              stages {
                stage("Build") {
                  when {
                    beforeAgent true
                    expression { params[COMPONENT] == true }
                  }
                  // agent { kubernetes(k8sAgent(name: 'dind')) }
                  steps {
                    // Failing on resourcePrefix/dockerParameters, as it doesn't have Field annotation
                    // Question is: why variables are not available inside matrix?
      
                    echo "resourcePrefix: ${resourcePrefix}"
                    echo "dockerParameters: ${dockerParameters}"
      
                    // Here is one step as example:
                    //dockerBuild(
                    //    image: dockerParameters.images[COMPONENT].image,
                    //    dockerfile: dockerParameters.images[COMPONENT].dockerfile
                    //)
                  }
                }
              }
            }
          }
      
        }
      }
      

      The result is following (build #2 on screenshot):

      stage `Prepare` goes fine anyway - as expected.

      stage `Docker` fails (on each matrix stage) with the message:

      groovy.lang.MissingPropertyException: No such property: resourcePrefix for class: groovy.lang.Binding
      

      Until I do not add annotation: `@groovy.transform.Field` (build #3 on screenshot).

      The same with `dockerParameters`, where I have map of different values, which are similar and have some common values.

      Note: this is just example, there is parameters, which we use in different stages, and copy-pasting all of them to each stage is not appropriate solution - defining them as common/global outside of `pipeline` block is the only way to do it, isn't it?

      Any splitting params (described in PR #405) or experimental features was never enabled.

      Parameters like: `SCRIPT_SPLITTING_TRANSFORMATION` & `SCRIPT_SPLITTING_ALLOW_LOCAL_VARIABLES` were not defined at all, as experimental feature supposed to be disabled by default.

      Expected result:

      • Users are not forced to fix each pipeline (and branch) variable with `@groovy.transform.Field` annotation
      • Pipelines with matrix (and latest `pipeline-model-definition` plugin) is continue support of variables, defined outside of `pipeline`
      • Experimental features without breaking changes

        Attachments

          Issue Links

            Activity

            Hide
            bitwiseman Liam Newman added a comment -

            Oleh Moskovych
            Good catch. Working on fix.

            Show
            bitwiseman Liam Newman added a comment - Oleh Moskovych Good catch. Working on fix.
            Hide
            bitwiseman Liam Newman added a comment -

            The fix has been published. It included tests to prevent this from happen in the future.

            Show
            bitwiseman Liam Newman added a comment - The fix has been published. It included tests to prevent this from happen in the future.
            Hide
            moskovych Oleh Moskovych added a comment -

            Liam Newman, thanks for so quick fix. Looks like it works fine.

            Show
            moskovych Oleh Moskovych added a comment - Liam Newman , thanks for so quick fix. Looks like it works fine.
            Hide
            jjathman Joe Athman added a comment -

            Liam Newman we seem to be hitting this same issue but are using a declarative pipeline, do you think this fix should work for those as well? We are running the 1.8.4 version of the plugin and still see the problem. We have a multi-branch job, which uses a shared pipeline coming from a global library. The shared pipeline uses local variables within the call method. In the pipeline we have a stage which is a matrix stage, the options and agent settings seem to cause a problem for us and we get the missing property exception. Here's a relatively simplified look at what our pipeline looks like:

            def call(Map<String, Object> pipelineParams) {
                def cronString = pipelineParams.getOrDefault('cronString', "")
                def testTimeoutInMinutes = pipelineParams.getOrDefault('testTimeoutInMinutes', 30)
                // add 2 hours to testTimeoutInMinutes to provide a buffer for getting an executor
                def expandedBuildAndTestTimeoutMinutes = testTimeoutInMinutes + 120
                debugEcho("expandedBuildAndTestTimeoutMinutes=$expandedBuildAndTestTimeoutMinutes")
            
                def office365Options = getTeamsOptions(pipelineParams)
                def jenkinsAgentLabel = pipelineParams.getOrDefault('jenkinsAgentLabel', 'linux_docker')
                def dockerImageName = 'customimagename'
                def dockerMaxMemory = pipelineParams.getOrDefault('dockerMaxMemory', "2g")
                def dockerRunArgs = pipelineParams.getOrDefault('dockerRunArgs', "-m $dockerMaxMemory".toString())
            
                pipeline {
                    agent none
                    options {
                        office365ConnectorWebhooks([office365Options])
                        timestamps()
                        ansiColor('xterm')
                    }
                    triggers {
                        cron(cronString)
                    }
                    stages {
                        stage('Windows Tests') {
                            matrix {
                                axes {
                                    axis {
                                        name 'PLATFORM'
                                        values 'WINDOWS_10'
                                    }
                                    axis {
                                        name 'BROWSER'
                                        values 'CHROME', 'FIREFOX', 'EDGE'
                                    }
                                }
                                stages {
                                    stage('Execute Tests') {
                                        agent {
                                            docker {
                                                label jenkinsAgentLabel
                                                image dockerImageName
                                                args dockerRunArgs
                                            }
                                        }
                                        options { timeout(time: expandedBuildAndTestTimeoutMinutes, unit: 'MINUTES') }
                                        steps {
                                            echo 'here'
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
            

             

            Show
            jjathman Joe Athman added a comment - Liam Newman  we seem to be hitting this same issue but are using a declarative pipeline, do you think this fix should work for those as well? We are running the 1.8.4 version of the plugin and still see the problem. We have a multi-branch job, which uses a shared pipeline coming from a global library. The shared pipeline uses local variables within the call method. In the pipeline we have a stage which is a matrix stage, the options and agent settings seem to cause a problem for us and we get the missing property exception. Here's a relatively simplified look at what our pipeline looks like: def call(Map< String , Object > pipelineParams) { def cronString = pipelineParams.getOrDefault( 'cronString' , "") def testTimeoutInMinutes = pipelineParams.getOrDefault( 'testTimeoutInMinutes' , 30) // add 2 hours to testTimeoutInMinutes to provide a buffer for getting an executor def expandedBuildAndTestTimeoutMinutes = testTimeoutInMinutes + 120 debugEcho( "expandedBuildAndTestTimeoutMinutes=$expandedBuildAndTestTimeoutMinutes" ) def office365Options = getTeamsOptions(pipelineParams) def jenkinsAgentLabel = pipelineParams.getOrDefault( 'jenkinsAgentLabel' , 'linux_docker' ) def dockerImageName = 'customimagename' def dockerMaxMemory = pipelineParams.getOrDefault( 'dockerMaxMemory' , "2g" ) def dockerRunArgs = pipelineParams.getOrDefault( 'dockerRunArgs' , "-m $dockerMaxMemory" .toString()) pipeline { agent none options { office365ConnectorWebhooks([office365Options]) timestamps() ansiColor( 'xterm' ) } triggers { cron(cronString) } stages { stage( 'Windows Tests' ) { matrix { axes { axis { name 'PLATFORM' values 'WINDOWS_10' } axis { name 'BROWSER' values 'CHROME' , 'FIREFOX' , 'EDGE' } } stages { stage( 'Execute Tests' ) { agent { docker { label jenkinsAgentLabel image dockerImageName args dockerRunArgs } } options { timeout(time: expandedBuildAndTestTimeoutMinutes, unit: 'MINUTES' ) } steps { echo 'here' } } } } } } } }  
            Hide
            bitwiseman Liam Newman added a comment -

            Joe Athman
            from https://issues.jenkins.io/browse/JENKINS-37984:
            "This workaround generally does NOT work if the pipeline directive inside a shared library method. If this is a scenario you want, please come join the pipeline authoring SIG and we can discuss."

            Considering this feature is experimental still it might be early to file a bug on it, but then again tracking this limitation is probably a good idea. At very least, we probably want it to not break this scenario and/or give meaningful feedback.

            Show
            bitwiseman Liam Newman added a comment - Joe Athman from https://issues.jenkins.io/browse/JENKINS-37984: "This workaround generally does NOT work if the pipeline directive inside a shared library method. If this is a scenario you want, please come join the pipeline authoring SIG and we can discuss." Considering this feature is experimental still it might be early to file a bug on it, but then again tracking this limitation is probably a good idea. At very least, we probably want it to not break this scenario and/or give meaningful feedback.

              People

              Assignee:
              bitwiseman Liam Newman
              Reporter:
              moskovych Oleh Moskovych
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: