Details
-
Bug
-
Status: Resolved (View Workflow)
-
Major
-
Resolution: Fixed
-
Jenkins version: 2.235.3
Java: OpenJDK 1.8.0_275
Main plugins:
pipeline-build-step 2.13
pipeline-graph-analysis 1.10
pipeline-input-step 2.12
pipeline-milestone-step 1.3.2
pipeline-model-api 1.8.2
pipeline-model-definition 1.8.2
pipeline-model-extensions 1.8.2
pipeline-rest-api 2.19
pipeline-stage-step 2.5
pipeline-stage-tags-metadata 1.8.2
pipeline-stage-view 2.19
pipeline-utility-steps 2.6.1
workflow-aggregator 2.6
workflow-api 2.41
workflow-basic-steps 2.21
workflow-cps 2.87
workflow-cps-global-lib 2.17
workflow-durable-task-step 2.35
workflow-job 2.40
workflow-multibranch 2.22
workflow-scm-step 2.11
workflow-step-api 2.23
workflow-support 3.7
and others...Jenkins version: 2.235.3 Java: OpenJDK 1.8.0_275 Main plugins: pipeline-build-step 2.13 pipeline-graph-analysis 1.10 pipeline-input-step 2.12 pipeline-milestone-step 1.3.2 pipeline-model-api 1.8.2 pipeline-model-definition 1.8.2 pipeline-model-extensions 1.8.2 pipeline-rest-api 2.19 pipeline-stage-step 2.5 pipeline-stage-tags-metadata 1.8.2 pipeline-stage-view 2.19 pipeline-utility-steps 2.6.1 workflow-aggregator 2.6 workflow-api 2.41 workflow-basic-steps 2.21 workflow-cps 2.87 workflow-cps-global-lib 2.17 workflow-durable-task-step 2.35 workflow-job 2.40 workflow-multibranch 2.22 workflow-scm-step 2.11 workflow-step-api 2.23 workflow-support 3.7 and others...
Description
This is continuation of problem described in comment JENKINS-37984#comment.
Example of pipeline:
#!/usr/bin/env groovy //library("jenkins_shared_library@1.0.0") //@groovy.transform.Field String resourcePrefix = new Date().getTime().toString() //@groovy.transform.Field Map dockerParameters = [ registry: "docker.example.com", registryType: "internal", images: [ image1: [image: "image1", dockerfile: "Dockerfile1"], image2: [image: "image2", dockerfile: "Dockerfile2"] ] ] pipeline { agent any options { skipDefaultCheckout true } parameters { booleanParam defaultValue: true, description: 'Build & Push image1', name: 'image1' booleanParam defaultValue: true, description: 'Build & Push image2', name: 'image2' } stages { stage("Prepare") { options { skipDefaultCheckout true } failFast true parallel { stage('Test1') { steps { // All variables available in simple stages and parallel blocks echo "resourcePrefix: ${resourcePrefix}" echo "dockerParameters: ${dockerParameters}" } } stage('Test2') { steps { echo "resourcePrefix: ${resourcePrefix}" echo "dockerParameters: ${dockerParameters}" } } } } stage("Docker") { options { skipDefaultCheckout true } matrix { axes { axis { name 'COMPONENT' // Note: these values are the same as described in dockerParameters and params values 'image1', 'image2' } } stages { stage("Build") { when { beforeAgent true expression { params[COMPONENT] == true } } // agent { kubernetes(k8sAgent(name: 'dind')) } steps { // Failing on resourcePrefix/dockerParameters, as it doesn't have Field annotation // Question is: why variables are not available inside matrix? echo "resourcePrefix: ${resourcePrefix}" echo "dockerParameters: ${dockerParameters}" // Here is one step as example: //dockerBuild( // image: dockerParameters.images[COMPONENT].image, // dockerfile: dockerParameters.images[COMPONENT].dockerfile //) } } } } } } }
The result is following (build #2 on screenshot):
stage `Prepare` goes fine anyway - as expected.
stage `Docker` fails (on each matrix stage) with the message:
groovy.lang.MissingPropertyException: No such property: resourcePrefix for class: groovy.lang.Binding
Until I do not add annotation: `@groovy.transform.Field` (build #3 on screenshot).
The same with `dockerParameters`, where I have map of different values, which are similar and have some common values.
Note: this is just example, there is parameters, which we use in different stages, and copy-pasting all of them to each stage is not appropriate solution - defining them as common/global outside of `pipeline` block is the only way to do it, isn't it?
Any splitting params (described in PR #405) or experimental features was never enabled.
Parameters like: `SCRIPT_SPLITTING_TRANSFORMATION` & `SCRIPT_SPLITTING_ALLOW_LOCAL_VARIABLES` were not defined at all, as experimental feature supposed to be disabled by default.
Expected result:
- Users are not forced to fix each pipeline (and branch) variable with `@groovy.transform.Field` annotation
- Pipelines with matrix (and latest `pipeline-model-definition` plugin) is continue support of variables, defined outside of `pipeline`
- Experimental features without breaking changes
Attachments
Issue Links
- is related to
-
-
- Open
-
Activity
The fix has been published. It included tests to prevent this from happen in the future.
Joe Athman
added a comment - bitwiseman we seem to be hitting this same issue but are using a declarative pipeline, do you think this fix should work for those as well? We are running the 1.8.4 version of the plugin and still see the problem. We have a multi-branch job, which uses a shared pipeline coming from a global library. The shared pipeline uses local variables within the call method. In the pipeline we have a stage which is a matrix stage, the options and agent settings seem to cause a problem for us and we get the missing property exception. Here's a relatively simplified look at what our pipeline looks like:
def call(Map<String, Object> pipelineParams) { def cronString = pipelineParams.getOrDefault('cronString', "") def testTimeoutInMinutes = pipelineParams.getOrDefault('testTimeoutInMinutes', 30) // add 2 hours to testTimeoutInMinutes to provide a buffer for getting an executor def expandedBuildAndTestTimeoutMinutes = testTimeoutInMinutes + 120 debugEcho("expandedBuildAndTestTimeoutMinutes=$expandedBuildAndTestTimeoutMinutes") def office365Options = getTeamsOptions(pipelineParams) def jenkinsAgentLabel = pipelineParams.getOrDefault('jenkinsAgentLabel', 'linux_docker') def dockerImageName = 'customimagename' def dockerMaxMemory = pipelineParams.getOrDefault('dockerMaxMemory', "2g") def dockerRunArgs = pipelineParams.getOrDefault('dockerRunArgs', "-m $dockerMaxMemory".toString()) pipeline { agent none options { office365ConnectorWebhooks([office365Options]) timestamps() ansiColor('xterm') } triggers { cron(cronString) } stages { stage('Windows Tests') { matrix { axes { axis { name 'PLATFORM' values 'WINDOWS_10' } axis { name 'BROWSER' values 'CHROME', 'FIREFOX', 'EDGE' } } stages { stage('Execute Tests') { agent { docker { label jenkinsAgentLabel image dockerImageName args dockerRunArgs } } options { timeout(time: expandedBuildAndTestTimeoutMinutes, unit: 'MINUTES') } steps { echo 'here' } } } } } } } }
Joe Athman
added a comment - bitwiseman we seem to be hitting this same issue but are using a declarative pipeline, do you think this fix should work for those as well? We are running the 1.8.4 version of the plugin and still see the problem. We have a multi-branch job, which uses a shared pipeline coming from a global library. The shared pipeline uses local variables within the call method. In the pipeline we have a stage which is a matrix stage, the options and agent settings seem to cause a problem for us and we get the missing property exception. Here's a relatively simplified look at what our pipeline looks like:
def call(Map< String , Object > pipelineParams) {
def cronString = pipelineParams.getOrDefault( 'cronString' , "")
def testTimeoutInMinutes = pipelineParams.getOrDefault( 'testTimeoutInMinutes' , 30)
// add 2 hours to testTimeoutInMinutes to provide a buffer for getting an executor
def expandedBuildAndTestTimeoutMinutes = testTimeoutInMinutes + 120
debugEcho( "expandedBuildAndTestTimeoutMinutes=$expandedBuildAndTestTimeoutMinutes" )
def office365Options = getTeamsOptions(pipelineParams)
def jenkinsAgentLabel = pipelineParams.getOrDefault( 'jenkinsAgentLabel' , 'linux_docker' )
def dockerImageName = 'customimagename'
def dockerMaxMemory = pipelineParams.getOrDefault( 'dockerMaxMemory' , "2g" )
def dockerRunArgs = pipelineParams.getOrDefault( 'dockerRunArgs' , "-m $dockerMaxMemory" .toString())
pipeline {
agent none
options {
office365ConnectorWebhooks([office365Options])
timestamps()
ansiColor( 'xterm' )
}
triggers {
cron(cronString)
}
stages {
stage( 'Windows Tests' ) {
matrix {
axes {
axis {
name 'PLATFORM'
values 'WINDOWS_10'
}
axis {
name 'BROWSER'
values 'CHROME' , 'FIREFOX' , 'EDGE'
}
}
stages {
stage( 'Execute Tests' ) {
agent {
docker {
label jenkinsAgentLabel
image dockerImageName
args dockerRunArgs
}
}
options { timeout(time: expandedBuildAndTestTimeoutMinutes, unit: 'MINUTES' ) }
steps {
echo 'here'
}
}
}
}
}
}
}
}
jjathman
from https://issues.jenkins.io/browse/JENKINS-37984:
"This workaround generally does NOT work if the pipeline directive inside a shared library method. If this is a scenario you want, please come join the pipeline authoring SIG and we can discuss."
Considering this feature is experimental still it might be early to file a bug on it, but then again tracking this limitation is probably a good idea. At very least, we probably want it to not break this scenario and/or give meaningful feedback.
moskovych
Good catch. Working on fix.