Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46547

Add support for defining Declarative pipelines in shared libraries

    • Icon: New Feature New Feature
    • Resolution: Fixed
    • Icon: Major Major
    • None
    • jenkins 2.76 and 2.60.3, pipeline model definition plugin 1.1.9

      I have project https://github.com/patope/jenkins-pipeline-test having two branches: master and direct. On 'direct' pipeline is defined in Jenkinsfile file and on master Jenkinsfile uses shared pipeline.

      Shared pipeline is defined in https://github.com/patope/jenkins-pipeline-test-shared

      On branch 'direct' stage 'Build 2' is skipped correctly.

      On branch 'master' (using shared pipeline) stage 'Build 2' is executed. 

       

      pipeline {
        agent any
        stages {
          stage('Build 1') {
            when { expression { true } }
            steps {
              echo('1')
            }
          }
          stage('Build 2') {
            when { expression { false } }
            steps {
              echo('2')
            }
          }
          stage('Build 3') {
            when { expression { true } }
            steps {
              echo('3')
            }
          }
        }
      }
      

       

          [JENKINS-46547] Add support for defining Declarative pipelines in shared libraries

          Thanks for the explanation.

          When you write "In 1.2, with the new parser logic, it won't work at all, in fact." you are presumably referring to plugin "pipeline-model-definition" version (currently being 1.1.9)? (Not to be mistaken with Blue Ocean which is currently version 1.2.1 already?)

          Reinhold Füreder added a comment - Thanks for the explanation. When you write " In 1.2, with the new parser logic, it won't work at all, in fact. " you are presumably referring to plugin "pipeline-model-definition" version (currently being 1.1.9)? (Not to be mistaken with Blue Ocean which is currently version 1.2.1 already?)

          Andrew Bayer added a comment -

          reinholdfuereder Correct, Declarative 1.2. And I am debating holding up the 1.2 release to include https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/193.

          Andrew Bayer added a comment - reinholdfuereder Correct, Declarative 1.2. And I am debating holding up the 1.2 release to include https://github.com/jenkinsci/pipeline-model-definition-plugin/pull/193 .

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/GroovyShellDecoratorImpl.java
          pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java
          pipeline-model-definition/src/test/resources/libForPipelineDefinedInLibrary.groovy
          pipeline-model-definition/src/test/resources/pipelineDefinedInLibrary.groovy
          http://jenkins-ci.org/commit/pipeline-model-definition-plugin/cbdb67a58a79dfb9c5af8d2d716d4685933f1456
          Log:
          JENKINS-46547 Allow loading pipeline {} blocks from shared libraries

          This is a work-in-progress - more is needed to be sure that we don't
          parse/validate/transform things we shouldn't, etc

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/GroovyShellDecoratorImpl.java pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java pipeline-model-definition/src/test/resources/libForPipelineDefinedInLibrary.groovy pipeline-model-definition/src/test/resources/pipelineDefinedInLibrary.groovy http://jenkins-ci.org/commit/pipeline-model-definition-plugin/cbdb67a58a79dfb9c5af8d2d716d4685933f1456 Log: JENKINS-46547 Allow loading pipeline {} blocks from shared libraries This is a work-in-progress - more is needed to be sure that we don't parse/validate/transform things we shouldn't, etc

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStages.java
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Root.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ASTParserUtils.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/PipelineStepFinder.groovy
          pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/actions/ExecutionModelAction.java
          pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/GroovyShellDecoratorImpl.java
          pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy
          pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java
          pipeline-model-definition/src/test/resources/fromEvaluate.groovy
          pipeline-model-definition/src/test/resources/libForMultiplePipelinesDefinedInLibrary.groovy
          pipeline-model-definition/src/test/resources/libForMultiplePipelinesExecutedInLibrary.groovy
          pipeline-model-definition/src/test/resources/libForPipelineDefinedInLibrary.groovy
          pipeline-model-definition/src/test/resources/multiplePipelinesDefinedInLibraryFirst.groovy
          pipeline-model-definition/src/test/resources/multiplePipelinesDefinedInLibrarySecond.groovy
          pipeline-model-definition/src/test/resources/pipelineDefinedInLibrary.groovy
          pipeline-model-definition/src/test/resources/postStage/globalAndLocalAlways.groovy
          pipeline-model-definition/src/test/resources/postStage/localAll.groovy
          pipeline-model-definition/src/test/resources/postStage/localAlways.groovy
          http://jenkins-ci.org/commit/pipeline-model-definition-plugin/6dba039fe3d5f1539db2cab7ddbf68e4a8a40789
          Log:
          Merge pull request #193 from abayer/jenkins-46547

          JENKINS-46547 Allow loading pipeline {} blocks from shared libraries

          Compare: https://github.com/jenkinsci/pipeline-model-definition-plugin/compare/e9d0feee71db...6dba039fe3d5

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: pipeline-model-api/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/ast/ModelASTStages.java pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/Utils.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/model/Root.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ASTParserUtils.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/ModelParser.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/PipelineStepFinder.groovy pipeline-model-definition/src/main/groovy/org/jenkinsci/plugins/pipeline/modeldefinition/parser/RuntimeASTTransformer.groovy pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/actions/ExecutionModelAction.java pipeline-model-definition/src/main/java/org/jenkinsci/plugins/pipeline/modeldefinition/parser/GroovyShellDecoratorImpl.java pipeline-model-definition/src/main/resources/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy pipeline-model-definition/src/test/java/org/jenkinsci/plugins/pipeline/modeldefinition/BasicModelDefTest.java pipeline-model-definition/src/test/resources/fromEvaluate.groovy pipeline-model-definition/src/test/resources/libForMultiplePipelinesDefinedInLibrary.groovy pipeline-model-definition/src/test/resources/libForMultiplePipelinesExecutedInLibrary.groovy pipeline-model-definition/src/test/resources/libForPipelineDefinedInLibrary.groovy pipeline-model-definition/src/test/resources/multiplePipelinesDefinedInLibraryFirst.groovy pipeline-model-definition/src/test/resources/multiplePipelinesDefinedInLibrarySecond.groovy pipeline-model-definition/src/test/resources/pipelineDefinedInLibrary.groovy pipeline-model-definition/src/test/resources/postStage/globalAndLocalAlways.groovy pipeline-model-definition/src/test/resources/postStage/localAll.groovy pipeline-model-definition/src/test/resources/postStage/localAlways.groovy http://jenkins-ci.org/commit/pipeline-model-definition-plugin/6dba039fe3d5f1539db2cab7ddbf68e4a8a40789 Log: Merge pull request #193 from abayer/jenkins-46547 JENKINS-46547 Allow loading pipeline {} blocks from shared libraries Compare: https://github.com/jenkinsci/pipeline-model-definition-plugin/compare/e9d0feee71db...6dba039fe3d5

          Andrew Bayer added a comment -

          Ok, this is in 1.2-beta-5 (which I'm cutting now) and will be in 1.2 when it's released next week.

          Andrew Bayer added a comment - Ok, this is in 1.2-beta-5 (which I'm cutting now) and will be in 1.2 when it's released next week.

          Code changed in jenkins
          User: Andrew Bayer
          Path:
          content/doc/book/pipeline/shared-libraries.adoc
          http://jenkins-ci.org/commit/jenkins.io/7009b1682960bdae12d5bfb54540dcff68529944
          Log:
          Preliminary doc update for JENKINS-46547

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Andrew Bayer Path: content/doc/book/pipeline/shared-libraries.adoc http://jenkins-ci.org/commit/jenkins.io/7009b1682960bdae12d5bfb54540dcff68529944 Log: Preliminary doc update for JENKINS-46547

          abayer While the preliminary doc update is consistent with your comments (from 2017-09-14) above, I think the recent blog post is not (please mind the diffierence "src" vs. "vars" folder) – or do I misunderstand that?

          This is part of the blog post of https://jenkins.io/blog/2017/09/25/declarative-1

          Reinhold Füreder added a comment - abayer While the preliminary doc update is consistent with your comments (from 2017-09-14) above, I think the recent blog post is not (please mind the diffierence "src" vs. "vars" folder) – or do I misunderstand that? This is part of the blog post of https://jenkins.io/blog/2017/09/25/declarative-1

          Ben Middleton added a comment -

          Oh dear. I was one of those who was happily using a pipeline in a shared library (under vars) without any problems. This new release has introduced loads of problems, e.g.:

          /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 119: Expected to find someKey "someValue" @ line 119, column 18.
                         node {
                              ^
          
          /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 133: Not a valid section definition: "cronTrigger && triggers {
                      cron(cronTrigger)
                  }". Some extra configuration is required. @ line 133, column 9.
                     cronTrigger && triggers {
                     ^
          
          /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 140: Method calls on objects not allowed outside "script" blocks. @ line 140, column 21.
                                 util.info "Building ${project}/${repository}/${jobName} on ${agentLabel}"
                                 ^
          
          /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 142: Method calls on objects not allowed outside "script" blocks. @ line 142, column 21.
                                 util.hipChatMessage type: 'started', repository: repository, commit: GIT_COMMIT,
                                 ^
          
          /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 145: Expected a step @ line 145, column 21.
                                 mavenBuild && util.mvn(mavenArguments)
                                 ^
          
          /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 147: Expected a step @ line 147, column 21.
                                 gradleBuild && util.gradle(gradleArguments)
                                 ^
          
          /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 138: Not a valid stage section definition: "gradleBuild && post {
                              always {
                                  //        junit allowEmptyResults: true, testResults: '**/target/surefire-reports/TEST-*.xml'
                                  (gradleArtifacts != '') && archiveArtifacts(artifacts: gradleArtifacts, onlyIfSuccessful: true)
                              }
                          }". Some extra configuration is required. @ line 138, column 13.
                         stage('Build') {
                         ^
          
          /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 161: Expected a step @ line 161, column 21.
                                 checkstyleReport && checkstyle()
                                 ^
          

          Here's the shared pipeline (note that there are a number of boolean flags here used to toggle various features, depending on the configuration from the calling Jenkinsfile:

          pipeline {
          
                  agent {
                      node {
                          label agentLabel
                          windowsBuild && customWorkspace(workspaceFolder)
                      }
                  }
          
                  options {
                      ansiColor('xterm')
                      buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: "${keptArtifacts}",
                              daysToKeepStr: '', numToKeepStr: '10'))
                      timestamps()
                  }
          
                  cronTrigger && triggers {
                      cron(cronTrigger)
                  }
          
                  stages {
                      stage('Build') {
                          steps {
                              util.info "Building ${project}/${repository}/${jobName} on ${agentLabel}"
          
                              util.hipChatMessage type: 'started', repository: repository, commit: GIT_COMMIT,
                                      bitbucketUrl: bitbucketUrl, rooms: hipChatRooms
          
                              mavenBuild && util.mvn(mavenArguments)
          
                              gradleBuild && util.gradle(gradleArguments)
                          }
          
                          gradleBuild && post {
                              always {
                                  (gradleArtifacts != '') && archiveArtifacts(artifacts: gradleArtifacts, onlyIfSuccessful: true)
                              }
                          }
                      }
          
                      stage('Quality') {
                          steps {
          
                              checkstyleReport && checkstyle()
          
                              jacocoReport && jacoco()
          
                              sonarQube && withSonarQubeEnv('SonarQube') {
                                  if (mavenBuild) {
                                      sonarUrl = sonarUrl + "/dashboard?id=${POM_GROUPID}:${POM_ARTIFACTID}:${jobName}"
                                      util.sonar env.BRANCH_NAME
                                  }
                              }
                          }
                      }
                  }
          

          Ben Middleton added a comment - Oh dear. I was one of those who was happily using a pipeline in a shared library (under vars) without any problems. This new release has introduced loads of problems, e.g.: /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 119: Expected to find someKey "someValue" @ line 119, column 18. node { ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 133: Not a valid section definition: "cronTrigger && triggers { cron(cronTrigger) }". Some extra configuration is required. @ line 133, column 9. cronTrigger && triggers { ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 140: Method calls on objects not allowed outside "script" blocks. @ line 140, column 21. util.info "Building ${project}/${repository}/${jobName} on ${agentLabel}" ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 142: Method calls on objects not allowed outside "script" blocks. @ line 142, column 21. util.hipChatMessage type: 'started' , repository: repository, commit: GIT_COMMIT, ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 145: Expected a step @ line 145, column 21. mavenBuild && util.mvn(mavenArguments) ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 147: Expected a step @ line 147, column 21. gradleBuild && util.gradle(gradleArguments) ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 138: Not a valid stage section definition: "gradleBuild && post { always { // junit allowEmptyResults: true , testResults: '**/target/surefire-reports/TEST-*.xml' (gradleArtifacts != '') && archiveArtifacts(artifacts: gradleArtifacts, onlyIfSuccessful: true ) } }". Some extra configuration is required. @ line 138, column 13. stage( 'Build' ) { ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/24/libs/delivery/vars/commonBuild.groovy: 161: Expected a step @ line 161, column 21. checkstyleReport && checkstyle() ^ Here's the shared pipeline (note that there are a number of boolean flags here used to toggle various features, depending on the configuration from the calling Jenkinsfile: pipeline { agent { node { label agentLabel windowsBuild && customWorkspace(workspaceFolder) } } options { ansiColor( 'xterm' ) buildDiscarder(logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: "${keptArtifacts}" , daysToKeepStr: '', numToKeepStr: ' 10')) timestamps() } cronTrigger && triggers { cron(cronTrigger) } stages { stage( 'Build' ) { steps { util.info "Building ${project}/${repository}/${jobName} on ${agentLabel}" util.hipChatMessage type: 'started' , repository: repository, commit: GIT_COMMIT, bitbucketUrl: bitbucketUrl, rooms: hipChatRooms mavenBuild && util.mvn(mavenArguments) gradleBuild && util.gradle(gradleArguments) } gradleBuild && post { always { (gradleArtifacts != '') && archiveArtifacts(artifacts: gradleArtifacts, onlyIfSuccessful: true ) } } } stage( 'Quality' ) { steps { checkstyleReport && checkstyle() jacocoReport && jacoco() sonarQube && withSonarQubeEnv( 'SonarQube' ) { if (mavenBuild) { sonarUrl = sonarUrl + "/dashboard?id=${POM_GROUPID}:${POM_ARTIFACTID}:${jobName}" util.sonar env.BRANCH_NAME } } } } }

          Ben Middleton added a comment -

          OK - after wrapping everything in script steps, I can resolve most of these issues. However, the following two are still a challenge:

          /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/29/libs/delivery/vars/commonBuild.groovy: 119: Expected to find someKey "someValue" @ line 119, column 18.
                         node {
                              ^
          
          /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/30/libs/delivery/vars/commonBuild.groovy: 134: Expected a trigger @ line 134, column 13.
                         cronTrigger && cron(cronTrigger)
                         ^
          

          The first one just looks like a bug.

          For the second, how can I write this so that the cron trigger is only activated if the cronTrigger is defined? I don't want to have if blocks wrapping the entire pipeline definitions just for the trigger section. Maybe time to revert back to scripted pipelines.

          Ben Middleton added a comment - OK - after wrapping everything in script steps, I can resolve most of these issues. However, the following two are still a challenge: /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/29/libs/delivery/vars/commonBuild.groovy: 119: Expected to find someKey "someValue" @ line 119, column 18. node { ^ /opt/jenkins-data/jobs/XO/jobs/xo-common/branches/develop/builds/30/libs/delivery/vars/commonBuild.groovy: 134: Expected a trigger @ line 134, column 13. cronTrigger && cron(cronTrigger) ^ The first one just looks like a bug. For the second, how can I write this so that the cron trigger is only activated if the cronTrigger is defined? I don't want to have if blocks wrapping the entire pipeline definitions just for the trigger section. Maybe time to revert back to scripted pipelines.

          Liam Newman added a comment -

          Bulk closing resolved issues.

          Liam Newman added a comment - Bulk closing resolved issues.

            abayer Andrew Bayer
            patope Tomi Pakarinen
            Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved: