-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
Jenkins: 2.164.1
Pipeline plugins:
* Build Step: 2.8
* Job: 2.32
We have a pipeline which in Test stage executes a lot of new jobs by using build step (we cannot move the jobs logic into this pipeline due to JENKINS-52391 - we don't want to rebuild 120 jobs because 1 has randomly failed):
stage('Test') { steps { script { def parallelTests = [:] def recipes = env.TESTED_RECIPES.split(',') def testSplit = env.TEST_SPLIT != null ? env.TEST_SPLIT.toInteger() : 1 def testWebSplit = env.WEBTEST_SPLIT != null ? env.WEBTEST_SPLIT.toInteger() : 1 for (def recipe in recipes) { for (def index = 1; index <= testSplit; ++index) { parallelTests["Test JUnit $recipe $index/$testSplit"] = createTestExecutionStage('generic-test-junit', [ string(name: 'PIPELINE_ID', value: params.PIPELINE_ID), string(name: 'RECIPE', value: recipe), string(name: 'TEST_SPLIT_IDX', value: "${index}"), string(name: 'TEST_SPLIT_MAX', value: "${testSplit}"), string(name: 'PRIORITY', value: params.PRIORITY) ]) } for (def index = 1; index <= testWebSplit; ++index) { parallelTests["Test Web JUnit $recipe $index/$testWebSplit"] = createTestExecutionStage('generic-test-junit', [ string(name: 'PIPELINE_ID', value: params.PIPELINE_ID), string(name: 'RECIPE', value: recipe), booleanParam(name: 'WEB_TESTS', value: true), string(name: 'TEST_SPLIT_IDX', value: "${index}"), string(name: 'TEST_SPLIT_MAX', value: "${testWebSplit}"), string(name: 'PRIORITY', value: params.PRIORITY) ], false) } parallelTests["Test Initialization $recipe"] = createTestExecutionStage('generic-test-initialization', [ string(name: 'PIPELINE_ID', value: params.PIPELINE_ID), string(name: 'RECIPE', value: recipe), string(name: 'PRIORITY', value: params.PRIORITY) ]) parallelTests["Test Server $recipe"] = createTestExecutionStage('generic-test-server', [ string(name: 'PIPELINE_ID', value: params.PIPELINE_ID), string(name: 'RECIPE', value: recipe), string(name: 'PRIORITY', value: params.PRIORITY) ]) } parallel parallelTests } } }
def createTestExecutionStage(name, parameters, propagate = true) { return { build(job: name, parameters: parameters, propagate: propagate) } }
Unfortunately, very rarely we see the following entries in logs:
Apr 04, 2019 4:30:04 AM WARNING hudson.model.ParametersAction filter Skipped parameter `PIPELINE_ID` as it is undefined on `generic-test-server`. Set `-Dhudson.model.ParametersAction.keepUndefinedParameters=true` to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach or `-Dhudson.model.ParametersAction.keepUndefinedParameters=false` to no longer show this message. Apr 04, 2019 4:30:04 AM WARNING hudson.model.ParametersAction filter Skipped parameter `RECIPE` as it is undefined on `generic-test-server`. Set `-Dhudson.model.ParametersAction.keepUndefinedParameters=true` to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach or `-Dhudson.model.ParametersAction.keepUndefinedParameters=false` to no longer show this message. Apr 04, 2019 4:30:04 AM WARNING hudson.model.ParametersAction filter Skipped parameter `PRIORITY` as it is undefined on `generic-test-server`. Set `-Dhudson.model.ParametersAction.keepUndefinedParameters=true` to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach or `-Dhudson.model.ParametersAction.keepUndefinedParameters=false` to no longer show this message.
It of course breaks the build because parameters are missing. This is how generic-test-junit looks after I clicked Build with Parameters:
In most cases it looks correct:
But sometimes it is broken:
The generic-junit-test job is generated by Job DSL plugin:
pipelineJob('generic-test-junit') { displayName('Generic JUnit Test') description('''Multi-line description is here''') logRotator { daysToKeep(90) } parameters { stringParam('PIPELINE_ID', '', 'Identifier of the pipeline which will be built.') stringParam('RECIPE', '', 'Recipe which will be tested.') booleanParam('WEB_TESTS', false, 'Defines whether web tests should be executed instead of the non-web tests.') stringParam('TEST_SPLIT_IDX', '', 'Index of the test package to be performed.') stringParam('TEST_SPLIT_MAX', '', 'Number of the test packages.') booleanParam('DEBUG', false, 'Defines whether test target should be executed in debug mode.') stringParam('PRIORITY', '', 'Pipeline priority (number from 1 to 5, where 1 is the highest).') } definition { cps { script(readFileFromWorkspace('jobs/genericTestJunit.jenkinsfile')) sandbox() } } }
This is the definition of generic-junit-test job (jobs/genericTestJunit.jenkinsfile):
pipeline { agent { label 'swarm' } options { ansiColor('xterm') } stages { stage('Test') { steps { cleanWs() sh """#!/bin/bash rsync -az jkmaster:build-scripts . rsync -az jkmaster:resources/${params.PIPELINE_ID}/pipeline.properties . """ script { def environmentVariables = "" readProperties(file: 'pipeline.properties') .findAll { !params.containsKey(it.key) } .each { k, v -> environmentVariables = "${environmentVariables}env.${k}='${v}'\n" } writeFile file: 'env.groovy', text: environmentVariables } load 'env.groovy' script { currentBuild.displayName = "#${env.BUILD_NUMBER}-${env.PRODUCT_NAME}-${params.PIPELINE_ID}-${params.RECIPE}-${params.WEB_TESTS ? 'WEB-' : ''}${params.TEST_SPLIT_IDX}-${params.TEST_SPLIT_MAX}" } timestamps { lock(resource: null, label: 'rsync', quantity: 1) { sh './build-scripts/synchronize-with-jkmaster.sh' } } wrap([$class: 'LogfilesizecheckerWrapper', failBuild: true, maxLogSize: 0, setOwn: false]) { wrap([$class: 'Xvfb', displayNameOffset: 2]) { timestamps { timeout(time: 6, unit: 'HOURS') { timeout(time: 15, activity: true) { sh "./build-scripts/jobs/test-junit.sh ${params.WEB_TESTS ? 'web' : ''}" } } } } } junit 'junit/**/*.xml' archiveArtifacts artifacts: 'junit/**/*.xml' } } } post { always { cleanWs() } } }
(I know the code is not very pretty (rsync etc.) - we are migrating very old Jenkins to newer technologies ).
At this moment we are forced to use this flag -Dhudson.model.ParametersAction.keepUndefinedParameters=true.