Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31155

Workflow shared library improvements

XMLWordPrintable

      In the context of pipeline as code, as an user, I have a lot of Jenkinsfile everywhere in my repositories and branches.

      • The developers that deal with them as not necessarily familar with Jenkins, so those files should be high-level and understandable to those who don't know Jenkins.
      • As an admin of Jenkins, I want to keep those files DRY by defining higher-level custom idioms
      • As an intermediate user of Jenkins, I want to reuse building blocks of workflow that other knowledgeable people in the Jenkins community has developed.

      Workflow plugin has one mechanism that touches (some of) this use case today, but it still need more improvements. Also, conversations with other Jenkins devs indicate that many want additional mechanisms to refer to reusable modules, and some has already developed additional plugins to do that.

      As the Jenkins community, we need to build a small set of mechanisms to achieve 80% of these goals, while enabling other people to write plugins that support more unique use cases.

      This ticket is a part of the 2.0 planning ticket to drive conversations about this and link to relevant activities.

        1. LibraryStep-JENKINS-31155.diff
          13 kB

          [JENKINS-31155] Workflow shared library improvements

          Kohsuke Kawaguchi created issue -
          Kohsuke Kawaguchi made changes -
          Epic Link New: JENKINS-31152 [ 165808 ]

          Brian Ray added a comment - - edited

          I am very slowly piloting a refactor of certain multijob and build DSL job chains into workflow DSL and the shared library mechanism is a bit of a sticking point because 1) the repo is internal to Jenkins and 2) we're going to be an SVN shop for a long time to come.

          The approach implemented in that experimental plugin would be good for us, if it supported SVN.

          Brian Ray added a comment - - edited I am very slowly piloting a refactor of certain multijob and build DSL job chains into workflow DSL and the shared library mechanism is a bit of a sticking point because 1) the repo is internal to Jenkins and 2) we're going to be an SVN shop for a long time to come. The approach implemented in that experimental plugin would be good for us, if it supported SVN.

          Martin d'Anjou added a comment -

          I'd like to point out closed issue JENKINS-26101, its description contains:

          Perhaps allow the workspace to be treated as a sourcepath so libraries can be imported.

          If that were implemented, and if classes could be loaded from that sourcepath, then wouldn't that solve the problem? All the workflow code could reside in a single SCM, the one that the job needs. The Workflow Global Library would not be needed, nor remote loader plugin.

          Martin d'Anjou added a comment - I'd like to point out closed issue JENKINS-26101 , its description contains: Perhaps allow the workspace to be treated as a sourcepath so libraries can be imported. If that were implemented, and if classes could be loaded from that sourcepath, then wouldn't that solve the problem? All the workflow code could reside in a single SCM, the one that the job needs. The Workflow Global Library would not be needed, nor remote loader plugin.

          Jesse Glick added a comment -

          If that were implemented, and if classes could be loaded from that sourcepath, then wouldn't that solve the problem?

          Would be one approach. There are other use cases for wanting things that look like global variables or functions, which workflow-cps-global-lib offers, which that would not cover.

          Also there were doubts about whether that approach could even be implemented reliably; having ClassLoader methods blocking on Jenkins remoting calls seemed rather trouble-prone.

          Jesse Glick added a comment - If that were implemented, and if classes could be loaded from that sourcepath, then wouldn't that solve the problem? Would be one approach. There are other use cases for wanting things that look like global variables or functions, which workflow-cps-global-lib offers, which that would not cover. Also there were doubts about whether that approach could even be implemented reliably; having ClassLoader methods blocking on Jenkins remoting calls seemed rather trouble-prone.

          Martin d'Anjou added a comment -

          Thank you for this info Jesse.

          IMO storing everything a given workflow job needs in a single SCM would be better than having two SCMs (easier to manage because the code would not get out of sync between the job workflow code, and the workflow global lib).

          If jobs configurations could select which branch of the workflow-cps-global-lib they take their classes from, it would be possible to do gradual deployments of new global lib versions. Right now if I push a global lib change, all jobs get the change, which is not great if I want to test before I apply it everywhere. I could have a test instance of Jenkins, but when there are externalities (e.g. a database), a test instance is not easy.

          Gradle has a buildscript construct. Can something similar exist when the workflow script comes from SCM? I guess this goes back to ClassLoader methods. I am not sure why it is trouble-prone, perhaps I don't understand the implications.

          I wish I could help with some code...

          Martin d'Anjou added a comment - Thank you for this info Jesse. IMO storing everything a given workflow job needs in a single SCM would be better than having two SCMs (easier to manage because the code would not get out of sync between the job workflow code, and the workflow global lib). If jobs configurations could select which branch of the workflow-cps-global-lib they take their classes from, it would be possible to do gradual deployments of new global lib versions. Right now if I push a global lib change, all jobs get the change, which is not great if I want to test before I apply it everywhere. I could have a test instance of Jenkins, but when there are externalities (e.g. a database), a test instance is not easy. Gradle has a buildscript construct. Can something similar exist when the workflow script comes from SCM? I guess this goes back to ClassLoader methods. I am not sure why it is trouble-prone, perhaps I don't understand the implications. I wish I could help with some code...

          Jesse Glick added a comment -

          storing everything a given workflow job needs in a single SCM would be better than having two SCMs

          This is already easily accomplished using the load step, or even by having a single Jenkinsfile.

          Right now if I push a global lib change, all jobs get the change, which is not great if I want to test before I apply it everywhere.

          Yes, this is my main criticism of workflow-cps-global-lib.

          Jesse Glick added a comment - storing everything a given workflow job needs in a single SCM would be better than having two SCMs This is already easily accomplished using the load step, or even by having a single Jenkinsfile . Right now if I push a global lib change, all jobs get the change, which is not great if I want to test before I apply it everywhere. Yes, this is my main criticism of workflow-cps-global-lib .

          Oleg Nenashev added a comment -

          https://github.com/jenkinsci/workflow-remote-loader-plugin is another partial solution

          Oleg Nenashev added a comment - https://github.com/jenkinsci/workflow-remote-loader-plugin is another partial solution

          Flávio Augusto Valones added a comment -

          The SCMSource approach proposed in JENKINS-32018, along with the versioning use of workflow-cps-global-lib seems an awesome combination.

          Flávio Augusto Valones added a comment - The SCMSource approach proposed in JENKINS-32018 , along with the versioning use of workflow-cps-global-lib seems an awesome combination.

          Jesse Glick added a comment -

          kohsuke also suggests having a resource file associated with a global library which could be easily copied into the workspace, retrieved as text, etc.

          Jesse Glick added a comment - kohsuke also suggests having a resource file associated with a global library which could be easily copied into the workspace, retrieved as text, etc.

            jglick Jesse Glick
            kohsuke Kohsuke Kawaguchi
            Votes:
            17 Vote for this issue
            Watchers:
            40 Start watching this issue

              Created:
              Updated:
              Resolved: