-
New Feature
-
Resolution: Fixed
-
Major
-
None
-
jenkins 2.76 and 2.60.3, pipeline model definition plugin 1.1.9
I have project https://github.com/patope/jenkins-pipeline-test having two branches: master and direct. On 'direct' pipeline is defined in Jenkinsfile file and on master Jenkinsfile uses shared pipeline.
Shared pipeline is defined in https://github.com/patope/jenkins-pipeline-test-shared
On branch 'direct' stage 'Build 2' is skipped correctly.
On branch 'master' (using shared pipeline) stage 'Build 2' is executed.
pipeline {
agent any
stages {
stage('Build 1') {
when { expression { true } }
steps {
echo('1')
}
}
stage('Build 2') {
when { expression { false } }
steps {
echo('2')
}
}
stage('Build 3') {
when { expression { true } }
steps {
echo('3')
}
}
}
}
- is blocking
-
JENKINS-42730 declarative-linter don't work with shared library
-
- Open
-
- is duplicated by
-
-
- Closed
-
- relates to
-
-
- Open
-
-
JENKINS-42224 Need ability to create reusable chunks of Declarative Pipeline
-
- Open
-
- links to
[JENKINS-46547] Add support for defining Declarative pipelines in shared libraries
Very preliminary PR up at https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/193
Reinhold Füreder
added a comment - abayer I am already very happily using pipeline blocks in shared library code in classes section (i.e. in 'src' folder, and NOT in 'vars' folder). According to the first bullet of the first comment (link to comment) you aim at dis-allowing that behaviour. Is that true, and/or why and/or can I somehow stop you from doing that?
Reinhold Füreder
added a comment - abayer I am already very happily using pipeline blocks in shared library code in classes section (i.e. in 'src' folder, and NOT in 'vars' folder). According to the first bullet of the first comment ( link to comment ) you aim at dis-allowing that behaviour. Is that true, and/or why and/or can I somehow stop you from doing that? reinholdfuereder - so, defining Declarative Pipelines in shared libraries has never actually been supported. It may have worked in some scenarios pre-1.2, but was never guaranteed to work or to keep working in the future. In 1.2, with the new parser logic, it won't work at all, in fact. When this change lands (presumably in 1.3, probably in late October), defining Decalarative Pipelines in files under vars in shared libraries will actually work, be supported, and be guaranteed to keep working going forward.
The reasoning for only supporting in vars is that the Declarative parser needs to be able to do some significant transformations of the code at compile time to work, and we don't want to be doing those transformations everywhere, just in Jenkinsfiles and in vars. Classes under src don't have the same automatic access to things like parameters, steps, etc, so the transformation wouldn't work properly there.
Reinhold Füreder
added a comment - Thanks for the explanation.
When you write "In 1.2, with the new parser logic, it won't work at all, in fact." you are presumably referring to plugin "pipeline-model-definition" version (currently being 1.1.9)? (Not to be mistaken with Blue Ocean which is currently version 1.2.1 already?)
Reinhold Füreder
added a comment - Thanks for the explanation.
When you write " In 1.2, with the new parser logic, it won't work at all, in fact. " you are presumably referring to plugin "pipeline-model-definition" version (currently being 1.1.9)? (Not to be mistaken with Blue Ocean which is currently version 1.2.1 already?) reinholdfuereder Correct, Declarative 1.2. And I am debating holding up the 1.2 release to include https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/193.
Code changed in jenkins
User: Andrew Bayer
Path:
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/GroovyShellDecoratorImpl.java
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java
pipeline-model-definition/src/test/resources/libForPipelineDefinedInLibrary.groovy
pipeline-model-definition/src/test/resources/pipelineDefinedInLibrary.groovy
http://jenkins-ci.org/commit/pipeline-model-definition-plugin/cbdb67a58a79dfb9c5af8d2d716d4685933f1456
Log:
JENKINS-46547 Allow loading pipeline {} blocks from shared libraries
This is a work-in-progress - more is needed to be sure that we don't
parse/validate/transform things we shouldn't, etc
Code changed in jenkins
User: Andrew Bayer
Path:
pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStages.java
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Root.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ASTParserUtils.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/PipelineStepFinder.groovy
pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy
pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/actions/ExecutionModelAction.java
pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/GroovyShellDecoratorImpl.java
pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java
pipeline-model-definition/src/test/resources/fromEvaluate.groovy
pipeline-model-definition/src/test/resources/libForMultiplePipelinesDefinedInLibrary.groovy
pipeline-model-definition/src/test/resources/libForMultiplePipelinesExecutedInLibrary.groovy
pipeline-model-definition/src/test/resources/libForPipelineDefinedInLibrary.groovy
pipeline-model-definition/src/test/resources/multiplePipelinesDefinedInLibraryFirst.groovy
pipeline-model-definition/src/test/resources/multiplePipelinesDefinedInLibrarySecond.groovy
pipeline-model-definition/src/test/resources/pipelineDefinedInLibrary.groovy
pipeline-model-definition/src/test/resources/postStage/globalAndLocalAlways.groovy
pipeline-model-definition/src/test/resources/postStage/localAll.groovy
pipeline-model-definition/src/test/resources/postStage/localAlways.groovy
http://jenkins-ci.org/commit/pipeline-model-definition-plugin/6dba039fe3d5f1539db2cab7ddbf68e4a8a40789
Log:
Merge pull request #193 from abayer/jenkins-46547
JENKINS-46547 Allow loading pipeline {} blocks from shared libraries
Compare: https://github.com/jenkinsci/pipeline-model-definition-plugin/compare/e9d0feee71db...6dba039fe3d5
Ok, this is in 1.2-beta-5 (which I'm cutting now) and will be in 1.2 when it's released next week.
Code changed in jenkins
User: Andrew Bayer
Path:
content/doc/book/pipeline/shared-libraries.adoc
http://jenkins-ci.org/commit/jenkins.io/7009b1682960bdae12d5bfb54540dcff68529944
Log:
Preliminary doc update for JENKINS-46547
Reinhold Füreder
added a comment - abayer While the preliminary doc update is consistent with your comments (from 2017-09-14) above, I think the recent blog post is not (please mind the diffierence "src" vs. "vars" folder) – or do I misunderstand that?
This is part of the blog post of https://jenkins.io/blog/2017/09/25/declarative-1
Reinhold Füreder
added a comment - abayer While the preliminary doc update is consistent with your comments (from 2017-09-14) above, I think the recent blog post is not (please mind the diffierence "src" vs. "vars" folder) – or do I misunderstand that?
This is part of the blog post of https://jenkins.io/blog/2017/09/25/declarative-1
Ben Middleton
added a comment - Oh dear. I was one of those who was happily using a pipeline in a shared library (under vars) without any problems. This new release has introduced loads of problems, e.g.:
/opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 119: Expected to find someKey "someValue" @ line 119, column 18. node { ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 133: Not a valid section definition: "cronTrigger && triggers { cron(cronTrigger) }". Some extra configuration is required. @ line 133, column 9. cronTrigger && triggers { ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 140: Method calls on objects not allowed outside "script" blocks. @ line 140, column 21. util.info "Building ${project}/${repository}/${jobName} on ${agentLabel}" ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 142: Method calls on objects not allowed outside "script" blocks. @ line 142, column 21. util.hipChatMessage type: 'started', repository: repository, commit: GIT_COMMIT, ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 145: Expected a step @ line 145, column 21. mavenBuild && util.mvn(mavenArguments) ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 147: Expected a step @ line 147, column 21. gradleBuild && util.gradle(gradleArguments) ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 138: Not a valid stage section definition: "gradleBuild && post { always { // junit allowEmptyResults: true, testResults: '**/target/surefire-reports/TEST-*.xml' (gradleArtifacts != '') && archiveArtifacts(artifacts: gradleArtifacts, onlyIfSuccessful: true) } }". Some extra configuration is required. @ line 138, column 13. stage('Build') { ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 161: Expected a step @ line 161, column 21. checkstyleReport && checkstyle() ^
Here's the shared pipeline (note that there are a number of boolean flags here used to toggle various features, depending on the configuration from the calling Jenkinsfile:
pipeline {
agent {
node {
label agentLabel
windowsBuild && customWorkspace(workspaceFolder)
}
}
options {
ansiColor('xterm')
buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: "${keptArtifacts}",
daysToKeepStr: '', numToKeepStr: '10'))
timestamps()
}
cronTrigger && triggers {
cron(cronTrigger)
}
stages {
stage('Build') {
steps {
util.info "Building ${project}/${repository}/${jobName} on ${agentLabel}"
util.hipChatMessage type: 'started', repository: repository, commit: GIT_COMMIT,
bitbucketUrl: bitbucketUrl, rooms: hipChatRooms
mavenBuild && util.mvn(mavenArguments)
gradleBuild && util.gradle(gradleArguments)
}
gradleBuild && post {
always {
(gradleArtifacts != '') && archiveArtifacts(artifacts: gradleArtifacts, onlyIfSuccessful: true)
}
}
}
stage('Quality') {
steps {
checkstyleReport && checkstyle()
jacocoReport && jacoco()
sonarQube && withSonarQubeEnv('SonarQube') {
if (mavenBuild) {
sonarUrl = sonarUrl + "/dashboard?id=${POM_GROUPID}:${POM_ARTIFACTID}:${jobName}"
util.sonar env.BRANCH_NAME
}
}
}
}
}
Ben Middleton
added a comment - Oh dear. I was one of those who was happily using a pipeline in a shared library (under vars) without any problems. This new release has introduced loads of problems, e.g.:
/opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 119: Expected to find someKey "someValue" @ line 119, column 18.
node {
^
/opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 133: Not a valid section definition: "cronTrigger && triggers {
cron(cronTrigger)
}". Some extra configuration is required. @ line 133, column 9.
cronTrigger && triggers {
^
/opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 140: Method calls on objects not allowed outside "script" blocks. @ line 140, column 21.
util.info "Building ${project}/${repository}/${jobName} on ${agentLabel}"
^
/opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 142: Method calls on objects not allowed outside "script" blocks. @ line 142, column 21.
util.hipChatMessage type: 'started' , repository: repository, commit: GIT_COMMIT,
^
/opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 145: Expected a step @ line 145, column 21.
mavenBuild && util.mvn(mavenArguments)
^
/opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 147: Expected a step @ line 147, column 21.
gradleBuild && util.gradle(gradleArguments)
^
/opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 138: Not a valid stage section definition: "gradleBuild && post {
always {
// junit allowEmptyResults: true , testResults: '**/target/surefire-reports/TEST-*.xml'
(gradleArtifacts != '') && archiveArtifacts(artifacts: gradleArtifacts, onlyIfSuccessful: true )
}
}". Some extra configuration is required. @ line 138, column 13.
stage( 'Build' ) {
^
/opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 161: Expected a step @ line 161, column 21.
checkstyleReport && checkstyle()
^
Here's the shared pipeline (note that there are a number of boolean flags here used to toggle various features, depending on the configuration from the calling Jenkinsfile:
pipeline {
agent {
node {
label agentLabel
windowsBuild && customWorkspace(workspaceFolder)
}
}
options {
ansiColor( 'xterm' )
buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: "${keptArtifacts}" ,
daysToKeepStr: '', numToKeepStr: ' 10'))
timestamps()
}
cronTrigger && triggers {
cron(cronTrigger)
}
stages {
stage( 'Build' ) {
steps {
util.info "Building ${project}/${repository}/${jobName} on ${agentLabel}"
util.hipChatMessage type: 'started' , repository: repository, commit: GIT_COMMIT,
bitbucketUrl: bitbucketUrl, rooms: hipChatRooms
mavenBuild && util.mvn(mavenArguments)
gradleBuild && util.gradle(gradleArguments)
}
gradleBuild && post {
always {
(gradleArtifacts != '') && archiveArtifacts(artifacts: gradleArtifacts, onlyIfSuccessful: true )
}
}
}
stage( 'Quality' ) {
steps {
checkstyleReport && checkstyle()
jacocoReport && jacoco()
sonarQube && withSonarQubeEnv( 'SonarQube' ) {
if (mavenBuild) {
sonarUrl = sonarUrl + "/dashboard?id=${POM_GROUPID}:${POM_ARTIFACTID}:${jobName}"
util.sonar env.BRANCH_NAME
}
}
}
}
}
Ben Middleton
added a comment - OK - after wrapping everything in script steps, I can resolve most of these issues. However, the following two are still a challenge:
/opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/29/libs/delivery/vars/commonBuild.groovy: 119: Expected to find someKey "someValue" @ line 119, column 18.
node {
^
/opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/30/libs/delivery/vars/commonBuild.groovy: 134: Expected a trigger @ line 134, column 13.
cronTrigger && cron(cronTrigger)
^
The first one just looks like a bug.
For the second, how can I write this so that the cron trigger is only activated if the cronTrigger is defined? I don't want to have if blocks wrapping the entire pipeline definitions just for the trigger section. Maybe time to revert back to scripted pipelines.
Ben Middleton
added a comment - OK - after wrapping everything in script steps, I can resolve most of these issues. However, the following two are still a challenge:
/opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/29/libs/delivery/vars/commonBuild.groovy: 119: Expected to find someKey "someValue" @ line 119, column 18.
node {
^
/opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/30/libs/delivery/vars/commonBuild.groovy: 134: Expected a trigger @ line 134, column 13.
cronTrigger && cron(cronTrigger)
^
The first one just looks like a bug.
For the second, how can I write this so that the cron trigger is only activated if the cronTrigger is defined? I don't want to have if blocks wrapping the entire pipeline definitions just for the trigger section. Maybe time to revert back to scripted pipelines. 
So as of now, this isn't expected to work (but isn't well documented that it won't) - pipeline blocks can only be defined in the main Pipeline itself, not in a shared library.
But...I'm starting some experiments here. Here's what I'm aiming at right now: