Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-42079

Allow "local" Shared Library to be loaded from current repo branch

      For context see: https://jenkins.io/blog/2017/02/15/declarative-notifications/
      And the linked bugs, in particular JENKINS-41396.

      I've been informed that this is a horrible idea, but I'm going to describe it here just to make sure.

      There have been a number of issues and discussions going around, dealing with how to define variables and methods in Declarative Pipeline.

      Right now, we can define variables and methods outside of the pipeline section but that is not a supported solution and it is not viable long-term.

      As noted in JENKINS-41396, we have a supported way to define variables and methods that we can use in Declarative Pipeline via
      Shared Libraries. However, that means I have to spin up a separate repository just to make a helper method that might be used in only one
      Pipeline.

      JENKINS-41396 and JENKINS-41335 suggest adding sections to the Declarative Pipeline model to allow these things to defined inside
      the same Jenkinsfile. This is also problematic: there are implementation issues, and it means that the distinction between the overall flow
      (defined in Declarative Pipeline) and the implementation details (defined in methods, variables, and classes) becomes more blurry.

      What I'm suggesting, is we reuse the functionality of "Shared libraries" but support loading from a subfolder in the current branch of the currently checked out repository.

      This would have a number of benefits over other solutions, it would:

      • reuse existing (and tested) functionality, limiting risk
      • not require creating a separate repository
      • keep the script code out of the Declarative Pipeline Jenkinsfile
      • use the same file/folder structure as Shared Library, getting users familiar with it for when they decide to use Shared Libraries

      On the downside, it would:

      • depend on additional folder/files beyond Jenkinsfile (method and define would be in Jenkinsfile)
      • require some changes to workflow-cps-global-lib to support loading from local and from subfolder instead of root folder.
      • make users have to understand library folder/file structure even to write simple helper method.

      We had a IRC chat about this, which I've attached.

      Also attached are images from a Pipeline job that I created as a proof-of-concept.
      The code for this POC is in: https://github.com/bitwiseman/hermann/tree/issue/jenkins-library/example

      What is not shown/done in the POC:

      • loading from a subfolder (not currently supported), subfolder name should default to "jenkins" (maybe?) and be configurable to other values.
      • * Worked around by putting the vars folder in the root - this would not work if the project already had a src or vars folder.
      • loading from the currently checked out branch/commit (not currently supported)
      • * Worked around by hard coding the branch name
      • using a library local directive to tell declarative to load a local library
      • * Worked around by adding the configuration to the parent Multibranch pipeline job and loading implicitly.
      • Using the already checked-out code
      • * Shared library simply clones it's own copy. For this small repo, this is fine, for larger projects it would not be acceptable.

          [JENKINS-42079] Allow "local" Shared Library to be loaded from current repo branch

          Liam Newman added a comment -

          Some have suggested that evaluate or readTrusted would be preferable solutions.
          They are fine for loading individual files, but not for more than one. Also, they behave differently from code in shared libraries, meaning users will have learn different behaviors for each.
          These also have drawbacks in areas such as testing, support, and documentation. If we guide users towards one consistent way to load things (Libraries) we do not have to explain it multiple times and fixes that help Shared Libraries would generally help local libraries as well.

          Liam Newman added a comment - Some have suggested that evaluate or readTrusted would be preferable solutions. They are fine for loading individual files, but not for more than one. Also, they behave differently from code in shared libraries, meaning users will have learn different behaviors for each. These also have drawbacks in areas such as testing, support, and documentation. If we guide users towards one consistent way to load things (Libraries) we do not have to explain it multiple times and fixes that help Shared Libraries would generally help local libraries as well.

          Liam Newman added a comment -

          One other note: even if we implement the method or define sections, I believe we should still consider this option. It provides another helpful interim step between on the way to Shared Libraries.

          Liam Newman added a comment - One other note: even if we implement the method or define sections, I believe we should still consider this option. It provides another helpful interim step between on the way to Shared Libraries.

          Jon Sten added a comment -

          We have a similar needs like this, in our case or needs are:

          • We wan't to extract all the "bulk" parts of our Jenkinsfile into a library
          • We have downstream products which wan't to run part of our tests (but with their latest version), running the tests isn't as simple as a single call and requires several lines of groovy code and thus we don't wan't to duplicate this. So ideally they wan't to call one of our functions in the library
          • Generally it feels natural to co-develop and version the library with the product in the same repo (since you usually require updates to both).

           

          Now, I have found a solution where you can load a library in a sub-folder of the repository, this requires that you load the Jenkinsfile through scm so that you have the scm variable. Note, we use SVN but it should be possible to port to GIT or similar:

          library identifier: 'OurLib@JenkinsLIB', retriever: modernSCM([$class: 'SubversionSCMSource', remoteBase: scm.getLocations()[0].remote])

          This loads the folder JenkinsLIB and treats it as a jenkins library. I wouldn't say that this is a nice solution, another problem is that the library is unaware of the repo url that it is associated with (which is necessary if you wan't to checkout the actual repo).

          Jon Sten added a comment - We have a similar needs like this, in our case or needs are: We wan't to extract all the "bulk" parts of our Jenkinsfile into a library We have downstream products which wan't to run part of our tests (but with their latest version), running the tests isn't as simple as a single call and requires several lines of groovy code and thus we don't wan't to duplicate this. So ideally they wan't to call one of our functions in the library Generally it feels natural to co-develop and version the library with the product in the same repo (since you usually require updates to both).   Now, I have found a solution where you can load a library in a sub-folder of the repository, this requires that you load the Jenkinsfile through scm so that you have the scm variable. Note, we use SVN but it should be possible to port to GIT or similar: library identifier: 'OurLib@JenkinsLIB', retriever: modernSCM([$class: 'SubversionSCMSource', remoteBase: scm.getLocations()[0].remote]) This loads the folder JenkinsLIB and treats it as a jenkins library. I wouldn't say that this is a nice solution, another problem is that the library is unaware of the repo url that it is associated with (which is necessary if you wan't to checkout the actual repo).

          Nick Jones added a comment -

          In my case I handle reusable notification code via a .groovy file committed into the project's source control (managed through a Jenkins-independent packaging mechanism like npm or NuGet, so versioned and updated there), which allows a single "library" to be loaded from the local branch via readTrusted:

          def lib = evaluate readTrusted('build/Library.groovy')
          def slackChannel = '#foo'
          
          pipeline {
            ...
            stages {
              stage('Build') {
                script {
                  lib.notifySlack(slackChannel, 'STARTED')
                }
              }
            }
            ...
            post {
              always {
                script {
                  lib.notifySlack(slackChannel, currentBuild.result)
                }
              }
            }
          }
          

          The script blocks are necessary to avoid Method calls on objects not allowed outside "script" blocks errors (JENKINS-42360, Won't Fix), and to enable the use of a slackChannel variable to avoid hard-coding the channel name in multiple places.

          My fear is that this loophole of using def outside the pipeline block will get closed, in as much as it's ostensibly unsupported behavior. But I don't currently have another way that I know of to have the syntactic goodness of Declarative while still leveraging code reuse. (And I should note that the exact same Library.groovy file in question is used in Scripted Pipeline jobs too.)

          Nick Jones added a comment - In my case I handle reusable notification code via a .groovy file committed into the project's source control (managed through a Jenkins-independent packaging mechanism like npm or NuGet, so versioned and updated there), which allows a single "library" to be loaded from the local branch via readTrusted : def lib = evaluate readTrusted( 'build/Library.groovy' ) def slackChannel = '#foo' pipeline { ... stages { stage( 'Build' ) { script { lib.notifySlack(slackChannel, 'STARTED' ) } } } ... post { always { script { lib.notifySlack(slackChannel, currentBuild.result) } } } } The script blocks are necessary to avoid Method calls on objects not allowed outside "script" blocks errors ( JENKINS-42360 , Won't Fix), and to enable the use of a slackChannel variable to avoid hard-coding the channel name in multiple places. My fear is that this loophole of using def outside the pipeline block will get closed, in as much as it's ostensibly unsupported behavior. But I don't currently have another way that I know of to have the syntactic goodness of Declarative while still leveraging code reuse. (And I should note that the exact same  Library.groovy file in question is used in Scripted Pipeline jobs too.)

          I'm kind of new to this pipeline stuff, but I think I'm doing this now?

          // To bring in the snippets, we need to be running on a real node
          // TODO: There are Jenkins plugins that can fix this.
          node('master') {
            checkout scm
            def import_config = load("releng/jenkins/runtime/groovy/import_config.groovy")
            branch_config = import_config()
            def find_last_success = load("releng/jenkins/runtime/groovy/find_last_success.groovy")
            previous_changesets = find_last_success()
          }
          

          Aaron D. Marasco added a comment - I'm kind of new to this pipeline stuff, but I think I'm doing this now? // To bring in the snippets, we need to be running on a real node // TODO: There are Jenkins plugins that can fix this . node( 'master' ) { checkout scm def import_config = load( "releng/jenkins/runtime/groovy/import_config.groovy" ) branch_config = import_config() def find_last_success = load( "releng/jenkins/runtime/groovy/find_last_success.groovy" ) previous_changesets = find_last_success() }

          Mike Kobit added a comment -

          Mike Kobit added a comment - Is https://github.com/jenkinsci/workflow-cps-global-lib-plugin/pull/37  related to this ?

          Jon Sten added a comment -

          mkobit It sure looks like that, that is great news!

          Jon Sten added a comment - mkobit It sure looks like that, that is great news!

          André Carmo added a comment - - edited

          I would really like this to be implemented. There is a scenario where this makes perfectly sense.

           

          Jenkins Shared Libraries are code, and because of that, me and my team we follow best practices. And of them is to have tests for our code. Because of this we have a Jenkins pipeline for the Jenkins Shared Library. We need to test against the current version of the code without manually managing versions.

          André Carmo added a comment - - edited I would really like this to be implemented. There is a scenario where this makes perfectly sense.   Jenkins Shared Libraries are code, and because of that, me and my team we follow best practices. And of them is to have tests for our code. Because of this we have a Jenkins pipeline for the Jenkins Shared Library. We need to test against the current version of the code without manually managing versions.

            Unassigned Unassigned
            bitwiseman Liam Newman
            Votes:
            16 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated: