Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31155

Workflow shared library improvements

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      In the context of pipeline as code, as an user, I have a lot of Jenkinsfile everywhere in my repositories and branches.

      • The developers that deal with them as not necessarily familar with Jenkins, so those files should be high-level and understandable to those who don't know Jenkins.
      • As an admin of Jenkins, I want to keep those files DRY by defining higher-level custom idioms
      • As an intermediate user of Jenkins, I want to reuse building blocks of workflow that other knowledgeable people in the Jenkins community has developed.

      Workflow plugin has one mechanism that touches (some of) this use case today, but it still need more improvements. Also, conversations with other Jenkins devs indicate that many want additional mechanisms to refer to reusable modules, and some has already developed additional plugins to do that.

      As the Jenkins community, we need to build a small set of mechanisms to achieve 80% of these goals, while enabling other people to write plugins that support more unique use cases.

      This ticket is a part of the 2.0 planning ticket to drive conversations about this and link to relevant activities.

        Attachments

          Issue Links

            Activity

            kohsuke Kohsuke Kawaguchi created issue -
            kohsuke Kohsuke Kawaguchi made changes -
            Field Original Value New Value
            Epic Link JENKINS-31152 [ 165808 ]
            Hide
            brianeray Brian Ray added a comment - - edited

            I am very slowly piloting a refactor of certain multijob and build DSL job chains into workflow DSL and the shared library mechanism is a bit of a sticking point because 1) the repo is internal to Jenkins and 2) we're going to be an SVN shop for a long time to come.

            The approach implemented in that experimental plugin would be good for us, if it supported SVN.

            Show
            brianeray Brian Ray added a comment - - edited I am very slowly piloting a refactor of certain multijob and build DSL job chains into workflow DSL and the shared library mechanism is a bit of a sticking point because 1) the repo is internal to Jenkins and 2) we're going to be an SVN shop for a long time to come. The approach implemented in that experimental plugin would be good for us, if it supported SVN.
            Hide
            deepchip Martin d'Anjou added a comment -

            I'd like to point out closed issue JENKINS-26101, its description contains:

            Perhaps allow the workspace to be treated as a sourcepath so libraries can be imported.

            If that were implemented, and if classes could be loaded from that sourcepath, then wouldn't that solve the problem? All the workflow code could reside in a single SCM, the one that the job needs. The Workflow Global Library would not be needed, nor remote loader plugin.

            Show
            deepchip Martin d'Anjou added a comment - I'd like to point out closed issue JENKINS-26101 , its description contains: Perhaps allow the workspace to be treated as a sourcepath so libraries can be imported. If that were implemented, and if classes could be loaded from that sourcepath, then wouldn't that solve the problem? All the workflow code could reside in a single SCM, the one that the job needs. The Workflow Global Library would not be needed, nor remote loader plugin.
            Hide
            jglick Jesse Glick added a comment -

            If that were implemented, and if classes could be loaded from that sourcepath, then wouldn't that solve the problem?

            Would be one approach. There are other use cases for wanting things that look like global variables or functions, which workflow-cps-global-lib offers, which that would not cover.

            Also there were doubts about whether that approach could even be implemented reliably; having ClassLoader methods blocking on Jenkins remoting calls seemed rather trouble-prone.

            Show
            jglick Jesse Glick added a comment - If that were implemented, and if classes could be loaded from that sourcepath, then wouldn't that solve the problem? Would be one approach. There are other use cases for wanting things that look like global variables or functions, which workflow-cps-global-lib offers, which that would not cover. Also there were doubts about whether that approach could even be implemented reliably; having ClassLoader methods blocking on Jenkins remoting calls seemed rather trouble-prone.
            Hide
            deepchip Martin d'Anjou added a comment -

            Thank you for this info Jesse.

            IMO storing everything a given workflow job needs in a single SCM would be better than having two SCMs (easier to manage because the code would not get out of sync between the job workflow code, and the workflow global lib).

            If jobs configurations could select which branch of the workflow-cps-global-lib they take their classes from, it would be possible to do gradual deployments of new global lib versions. Right now if I push a global lib change, all jobs get the change, which is not great if I want to test before I apply it everywhere. I could have a test instance of Jenkins, but when there are externalities (e.g. a database), a test instance is not easy.

            Gradle has a buildscript construct. Can something similar exist when the workflow script comes from SCM? I guess this goes back to ClassLoader methods. I am not sure why it is trouble-prone, perhaps I don't understand the implications.

            I wish I could help with some code...

            Show
            deepchip Martin d'Anjou added a comment - Thank you for this info Jesse. IMO storing everything a given workflow job needs in a single SCM would be better than having two SCMs (easier to manage because the code would not get out of sync between the job workflow code, and the workflow global lib). If jobs configurations could select which branch of the workflow-cps-global-lib they take their classes from, it would be possible to do gradual deployments of new global lib versions. Right now if I push a global lib change, all jobs get the change, which is not great if I want to test before I apply it everywhere. I could have a test instance of Jenkins, but when there are externalities (e.g. a database), a test instance is not easy. Gradle has a buildscript construct. Can something similar exist when the workflow script comes from SCM? I guess this goes back to ClassLoader methods. I am not sure why it is trouble-prone, perhaps I don't understand the implications. I wish I could help with some code...
            Hide
            jglick Jesse Glick added a comment -

            storing everything a given workflow job needs in a single SCM would be better than having two SCMs

            This is already easily accomplished using the load step, or even by having a single Jenkinsfile.

            Right now if I push a global lib change, all jobs get the change, which is not great if I want to test before I apply it everywhere.

            Yes, this is my main criticism of workflow-cps-global-lib.

            Show
            jglick Jesse Glick added a comment - storing everything a given workflow job needs in a single SCM would be better than having two SCMs This is already easily accomplished using the load step, or even by having a single Jenkinsfile . Right now if I push a global lib change, all jobs get the change, which is not great if I want to test before I apply it everywhere. Yes, this is my main criticism of workflow-cps-global-lib .
            Hide
            oleg_nenashev Oleg Nenashev added a comment -
            Show
            oleg_nenashev Oleg Nenashev added a comment - https://github.com/jenkinsci/workflow-remote-loader-plugin is another partial solution
            Hide
            valones Flávio Augusto Valones added a comment -

            The SCMSource approach proposed in JENKINS-32018, along with the versioning use of workflow-cps-global-lib seems an awesome combination.

            Show
            valones Flávio Augusto Valones added a comment - The SCMSource approach proposed in JENKINS-32018 , along with the versioning use of workflow-cps-global-lib seems an awesome combination.
            Hide
            jglick Jesse Glick added a comment -

            Kohsuke Kawaguchi also suggests having a resource file associated with a global library which could be easily copied into the workspace, retrieved as text, etc.

            Show
            jglick Jesse Glick added a comment - Kohsuke Kawaguchi also suggests having a resource file associated with a global library which could be easily copied into the workspace, retrieved as text, etc.
            kohsuke Kohsuke Kawaguchi made changes -
            Labels 2.0 2.0-rejected
            jglick Jesse Glick made changes -
            Epic Link JENKINS-31152 [ 165808 ] JENKINS-35386 [ 171179 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-34476 [ JENKINS-34476 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-26538 [ JENKINS-26538 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-34186 [ JENKINS-34186 ]
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-32400 [ JENKINS-32400 ]
            Hide
            jglick Jesse Glick added a comment -

            JENKINS-26192 requested Grape support. Seems unlikely this would work but deserves investigation.

            Show
            jglick Jesse Glick added a comment - JENKINS-26192 requested Grape support. Seems unlikely this would work but deserves investigation.
            jglick Jesse Glick made changes -
            Link This issue is related to JENKINS-26192 [ JENKINS-26192 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 166335 ] JNJira + In-Review [ 182357 ]
            cprice404 Chris Price made changes -
            Link This issue relates to JENKINS-37125 [ JENKINS-37125 ]
            Hide
            jglick Jesse Glick added a comment -

            Would be desirable to retain the benefit of JENKINS-34650 in some cases: for externally versioned libraries which we can say are “blessed” by a Jenkins administrator somehow, skip sandbox checks. But we also need regular developers to be able to define and use libraries, hence the need for an analogue of JENKINS-26538 (which might itself simply be closed, if this plan works out).

            Show
            jglick Jesse Glick added a comment - Would be desirable to retain the benefit of JENKINS-34650 in some cases: for externally versioned libraries which we can say are “blessed” by a Jenkins administrator somehow, skip sandbox checks. But we also need regular developers to be able to define and use libraries, hence the need for an analogue of JENKINS-26538 (which might itself simply be closed, if this plan works out).
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-34650 [ JENKINS-34650 ]
            Hide
            jglick Jesse Glick added a comment -

            Confirmed that Grape is not going to work for the purpose of bringing in external CPS-transformed Groovy code.

            Show
            jglick Jesse Glick added a comment - Confirmed that Grape is not going to work for the purpose of bringing in external CPS-transformed Groovy code.
            Hide
            jglick Jesse Glick added a comment -

            I take that back—it does seem possible. Needs study, at least as an option.

            Show
            jglick Jesse Glick added a comment - I take that back—it does seem possible. Needs study, at least as an option.
            Hide
            jglick Jesse Glick added a comment -

            Amended: Grape works to pull in CPS-transformed code in Jenkins 2, presumably because of some change in Groovy 2, but not in Jenkins 1.

            Show
            jglick Jesse Glick added a comment - Amended: Grape works to pull in CPS-transformed code in Jenkins 2, presumably because of some change in Groovy 2, but not in Jenkins 1.
            Hide
            jglick Jesse Glick added a comment -

            Anyway adding it only for use in global libraries. Does not work from sandboxed scripts, making it ill suited to the needs of this issue.

            Show
            jglick Jesse Glick added a comment - Anyway adding it only for use in global libraries. Does not work from sandboxed scripts, making it ill suited to the needs of this issue.
            Hide
            jglick Jesse Glick added a comment - - edited

            What I think the mandatory requirements are:

            • Must allow library code to be pulled from an external location on demand, rather than pushed into Jenkins.
            • Must allow each job using a library, or other developer-level configuration, to choose whether to pick the latest version, or a specific version.
            • Must support the “external location” being in any SCM plugin supported by Jenkins, where “latest version” is a branch and “specific version” is a revision or tag. (TBD whether conformance is to the old SCM API, the newer SCMSource with fewer implementations, or either.)
            • Must allow library code developed by, or on behalf of, a Jenkins administrator to run outside the Groovy sandbox.
            • Must require library code developed by others to run inside the Groovy sandbox.
            • Must support both syntax options currently supported by workflow-cps-global-lib: class library, or global variable/function.
            • Accompanying functional tests must prove that behavior properly persists across Jenkins restarts, and (in CloudBees Jenkins Enterprise) after resuming from checkpoint.
            • Build resumption must not make network connections.
            • Must be possible to use a named library in a job without specifying connection details such as URL or credentials.
            • Must be possible to use the Replay feature to temporarily test the effect of changing some code inside a library on a given job.

            Nice to have:

            • Possibility for “external location” to be an artifact repository rather than an SCM, for example via @Grab.
            • Ability to implicitly load a library in all jobs in the system or within a folder (as in workflow-cps-global-lib).
            • Global Variables Reference support for the global variable mode of a library (as in workflow-cps-global-lib).
            • Ability to load non-Groovy support files packaged with the library, probably as String.
            • Ability to load third-party binary libraries, for example via @Grab.
            • Step to dynamically load a specified library into a running build, assuming its types were not already consulted during parsing.
            Show
            jglick Jesse Glick added a comment - - edited What I think the mandatory requirements are: Must allow library code to be pulled from an external location on demand, rather than pushed into Jenkins. Must allow each job using a library, or other developer-level configuration, to choose whether to pick the latest version, or a specific version. Must support the “external location” being in any SCM plugin supported by Jenkins, where “latest version” is a branch and “specific version” is a revision or tag. (TBD whether conformance is to the old SCM API, the newer SCMSource with fewer implementations, or either.) Must allow library code developed by, or on behalf of, a Jenkins administrator to run outside the Groovy sandbox. Must require library code developed by others to run inside the Groovy sandbox. Must support both syntax options currently supported by workflow-cps-global-lib : class library, or global variable/function. Accompanying functional tests must prove that behavior properly persists across Jenkins restarts, and (in CloudBees Jenkins Enterprise) after resuming from checkpoint . Build resumption must not make network connections. Must be possible to use a named library in a job without specifying connection details such as URL or credentials. Must be possible to use the Replay feature to temporarily test the effect of changing some code inside a library on a given job. Nice to have: Possibility for “external location” to be an artifact repository rather than an SCM, for example via @Grab . Ability to implicitly load a library in all jobs in the system or within a folder (as in workflow-cps-global-lib ). Global Variables Reference support for the global variable mode of a library (as in workflow-cps-global-lib ). Ability to load non-Groovy support files packaged with the library, probably as String . Ability to load third-party binary libraries, for example via @Grab . Step to dynamically load a specified library into a running build, assuming its types were not already consulted during parsing.
            Hide
            hrmpw Patrick Wolf added a comment - - edited

            Jesse's list from above with comments

            Must Have:

            • Must allow library code to be pulled from an external location on demand, rather than pushed into Jenkins.
            • Must allow each job using a library, or other developer-level configuration, to choose whether to pick the latest version, or a specific version.
              • Would this also include a default version or would default always be Master/Head unless specified on job? Setting version used at Folder level might be a nice way to allow teams to migrate to newer version at own pace. Then again if we use a folder level library the folder level library could define a default version of the global library to be used.
            • Must support the “external location” being in any SCM plugin supported by Jenkins, where “latest version” is a branch and “specific version” is a revision or tag. (TBD whether conformance is to the old SCM API, the newer SCMSource with fewer implementations, or either.)
            • Must allow library code developed by, or on behalf of, a Jenkins administrator to run outside the Groovy sandbox.
              • for Global library. I assume this won't be true for folder level if that is done.
            • Must require library code developed by others to run inside the Groovy sandbox.
            • Must support both syntax options currently supported by workflow-cps-global-lib: class library, or global variable/function.
            • Accompanying functional tests must prove that behavior properly persists across Jenkins restarts, and (in CloudBees Jenkins Enterprise) after resuming from checkpoint.
            • Build resumption must not make network connections.
            • Must be possible to use a named library in a job without specifying connection details such as URL or credentials.
            • Must be possible to use the Replay feature to temporarily test the effect of changing some code inside a library on a given job.

            Nice to have:

            • Possibility for “external location” to be an artifact repository rather than an SCM, for example via @Grab.
            • Ability to implicitly load a library in all jobs in the system or within a folder (as in workflow-cps-global-lib).
            • Global Variables Reference support for the global variable mode of a library (as in workflow-cps-global-lib).
            • Ability to load non-Groovy support files packaged with the library, probably as String.
            • Ability to load third-party binary libraries, for example via @Grab.
            • Step to dynamically load a specified library into a running build, assuming its types were not already consulted during parsing.
            Show
            hrmpw Patrick Wolf added a comment - - edited Jesse's list from above with comments Must Have: Must allow library code to be pulled from an external location on demand, rather than pushed into Jenkins. Must allow each job using a library, or other developer-level configuration, to choose whether to pick the latest version, or a specific version. Would this also include a default version or would default always be Master/Head unless specified on job? Setting version used at Folder level might be a nice way to allow teams to migrate to newer version at own pace. Then again if we use a folder level library the folder level library could define a default version of the global library to be used. Must support the “external location” being in any SCM plugin supported by Jenkins, where “latest version” is a branch and “specific version” is a revision or tag. (TBD whether conformance is to the old SCM API, the newer SCMSource with fewer implementations, or either.) Must allow library code developed by, or on behalf of, a Jenkins administrator to run outside the Groovy sandbox. for Global library. I assume this won't be true for folder level if that is done. Must require library code developed by others to run inside the Groovy sandbox. Must support both syntax options currently supported by workflow-cps-global-lib: class library, or global variable/function. Accompanying functional tests must prove that behavior properly persists across Jenkins restarts, and (in CloudBees Jenkins Enterprise) after resuming from checkpoint. Build resumption must not make network connections. Must be possible to use a named library in a job without specifying connection details such as URL or credentials. Must be possible to use the Replay feature to temporarily test the effect of changing some code inside a library on a given job. Nice to have: Possibility for “external location” to be an artifact repository rather than an SCM, for example via @Grab. Ability to implicitly load a library in all jobs in the system or within a folder (as in workflow-cps-global-lib). Global Variables Reference support for the global variable mode of a library (as in workflow-cps-global-lib). Ability to load non-Groovy support files packaged with the library, probably as String. Ability to load third-party binary libraries, for example via @Grab. Step to dynamically load a specified library into a running build, assuming its types were not already consulted during parsing.
            Hide
            jglick Jesse Glick added a comment -

            Would this also include a default version or would default always be Master/Head unless specified on job?

            TBD, but probably a global or folder-level setting could specify a version.

            I assume this won't be true for folder level if that is done.

            Right, folder-level settings would not be trusted as far as Overall/RunScripts is concerned.

            Show
            jglick Jesse Glick added a comment - Would this also include a default version or would default always be Master/Head unless specified on job? TBD, but probably a global or folder-level setting could specify a version. I assume this won't be true for folder level if that is done. Right, folder-level settings would not be trusted as far as Overall/RunScripts is concerned.
            Hide
            gitt Slawa Giterman added a comment -

            We currently do not use cps-global-lib and store our shared modules in SCM because of lack of versioning support in cps-global-lib. We then checkout the files from SCM and use load step to load them in Jenkinsfiles in different pipelines.

            My question is: if you implement the feature as described above (seems exactly what many teams need), will it replace cps-global-lib? And do you have any particular timetable for this feature?

            Show
            gitt Slawa Giterman added a comment - We currently do not use cps-global-lib and store our shared modules in SCM because of lack of versioning support in cps-global-lib. We then checkout the files from SCM and use load step to load them in Jenkinsfiles in different pipelines. My question is: if you implement the feature as described above (seems exactly what many teams need), will it replace cps-global-lib? And do you have any particular timetable for this feature?
            Hide
            jglick Jesse Glick added a comment -

            will it replace cps-global-lib

            Probably I would add options to the same plugin. Either way, the existing feature is not going to go anywhere.

            do you have any particular timetable for this feature?

            I am looking into it.

            Rough outline of proposed implementation:

            • a configuration fragment defining a “library”, including SCM details and default version choice (branch/tag/revision); the repository should have the same layout as the current global library repo, plus a resources folder
            • a configuration fragment for a list of libraries with symbolic names, some of which may be loaded by default (Jenkinsfile need do nothing to start using them)
            • a global configuration setting available with RUN_SCRIPTS
            • a folder property
            • an ASTTransformation allowing some annotation to be processed in Jenkinsfile specifying libraries to use, optionally with versions
            • a per-job per-library private workspace on master to check out SCM
            • a per-build JAR of loaded libraries as checked out at the start of the build (including the patternset src/,vars/,resources/)—allows consistent snapshots of branches to be used; prevents network I/O when resuming a build; and allows Replay to see what was used
            • a GroovyShellDecorator adding those JARs to the classpath
            • some step to load resources
            • for trusted libraries, third-party code via @Grab (already implemented in JENKINS-26192 but integration would need to be tested)
            • GlobalVariableList using the last-loaded versions of global libraries for reference documentation (perhaps also for folder-level libraries, with some API hooks)
            Show
            jglick Jesse Glick added a comment - will it replace cps-global-lib Probably I would add options to the same plugin. Either way, the existing feature is not going to go anywhere. do you have any particular timetable for this feature? I am looking into it. Rough outline of proposed implementation: a configuration fragment defining a “library”, including SCM details and default version choice (branch/tag/revision); the repository should have the same layout as the current global library repo, plus a resources folder a configuration fragment for a list of libraries with symbolic names, some of which may be loaded by default ( Jenkinsfile need do nothing to start using them) a global configuration setting available with RUN_SCRIPTS a folder property an ASTTransformation allowing some annotation to be processed in Jenkinsfile specifying libraries to use, optionally with versions a per-job per-library private workspace on master to check out SCM a per-build JAR of loaded libraries as checked out at the start of the build (including the patternset src/,vars/,resources/ )—allows consistent snapshots of branches to be used; prevents network I/O when resuming a build; and allows Replay to see what was used a GroovyShellDecorator adding those JARs to the classpath some step to load resources for trusted libraries, third-party code via @Grab (already implemented in JENKINS-26192 but integration would need to be tested) GlobalVariableList using the last-loaded versions of global libraries for reference documentation (perhaps also for folder-level libraries, with some API hooks)
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-cps-global-lib PR 10 (Web Link)" [ 14716 ]
            Hide
            jglick Jesse Glick added a comment -

            A sticky point is checking out a specified SCM at a specified tag. SCMSource does not really support this; it defines “heads” but these are generally only implemented to be branches. SCM does not offer any API to select even a branch, much less a tag, though typical implementations would allow the critical configuration field to be interpolated with build variables during checkout. I am thinking of a two-pronged approach:

            • Extend SCMSource to allow you to specify a version string and get back an SCMRevision that it resolves to; a Git implementation could accept branch names, tag names, other refs, short or long hashes—whatever Git normally accepts as per gitrevisions(7). Then build would check out the specified revision. This would provide the best experience, for sufficiently up-to-date SCM plugins.
            • As a fallback for older SCMSource implementations, or SCM plugins not implementing SCMSource at all, allow SingleSCMSource to be used, but add an EnvironmentContributor which sets variables with predictable names to the library versions used by the build. Then the global configuration would define, say, a library foo using GitSCM with a BranchSpec like ${lib.foo.version}, and leave it to checkout time to actually resolve that to something.
            Show
            jglick Jesse Glick added a comment - A sticky point is checking out a specified SCM at a specified tag. SCMSource does not really support this; it defines “heads” but these are generally only implemented to be branches. SCM does not offer any API to select even a branch, much less a tag, though typical implementations would allow the critical configuration field to be interpolated with build variables during checkout. I am thinking of a two-pronged approach: Extend SCMSource to allow you to specify a version string and get back an SCMRevision that it resolves to; a Git implementation could accept branch names, tag names, other refs, short or long hashes—whatever Git normally accepts as per gitrevisions(7) . Then build would check out the specified revision. This would provide the best experience, for sufficiently up-to-date SCM plugins. As a fallback for older SCMSource implementations, or SCM plugins not implementing SCMSource at all, allow SingleSCMSource to be used, but add an EnvironmentContributor which sets variables with predictable names to the library versions used by the build. Then the global configuration would define, say, a library foo using GitSCM with a BranchSpec like ${lib.foo.version }, and leave it to checkout time to actually resolve that to something.
            jamesdumay James Dumay made changes -
            Labels 2.0-rejected 2.0-rejected blueocean
            jamesdumay James Dumay made changes -
            Labels 2.0-rejected blueocean 2.0-rejected blueocean x-team
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-32768 [ JENKINS-32768 ]
            Hide
            jglick Jesse Glick added a comment -

            Fallback mode verified to be functional. Have not yet tested a real implementation of retrieve(String, TaskListener).

            Show
            jglick Jesse Glick added a comment - Fallback mode verified to be functional. Have not yet tested a real implementation of retrieve(String, TaskListener) .
            Hide
            jglick Jesse Glick added a comment -

            Should also add an automatic library resolver for GitHub, and think about whether there is a reasonable equivalent for private SCM servers.

            Show
            jglick Jesse Glick added a comment - Should also add an automatic library resolver for GitHub, and think about whether there is a reasonable equivalent for private SCM servers.
            Hide
            jglick Jesse Glick added a comment -

            Should also investigate whether a library step can be used at runtime to load an (untrusted) library into the running program’s classpath. Would be most suitable for variable-style usage, since at this point the compiler has already resolved symbols, but it may even be possible to use class libraries this way (TBD).

            Most useful in combination with automatic resolvers, though could also consider letting the step pick an SCMSource and version.

            In principle a library step could allow libraries to load other libraries. This is getting dangerously close to Grape territory however.

            Show
            jglick Jesse Glick added a comment - Should also investigate whether a library step can be used at runtime to load an (untrusted) library into the running program’s classpath. Would be most suitable for variable-style usage, since at this point the compiler has already resolved symbols, but it may even be possible to use class libraries this way (TBD). Most useful in combination with automatic resolvers, though could also consider letting the step pick an SCMSource and version . In principle a library step could allow libraries to load other libraries. This is getting dangerously close to Grape territory however.
            jglick Jesse Glick made changes -
            Link This issue depends on JENKINS-33273 [ JENKINS-33273 ]
            Hide
            jglick Jesse Glick added a comment - - edited

            Wrote a library step but not having any success using it to load class libraries: even though all the GroovyClassLoader in the chain (the InnerLoader, the untrusted loader, and the trusted loader can load the named class, trying to refer to it by FQN still results in a MissingPropertyException. You can load the class using reflection, but this is kind of pointless as your script then cannot run in the sandbox. Calling GroovyClassLoader.parseClass does not seem to help.

            Show
            jglick Jesse Glick added a comment - - edited Wrote a library step but not having any success using it to load class libraries: even though all the GroovyClassLoader in the chain (the InnerLoader , the untrusted loader, and the trusted loader can load the named class, trying to refer to it by FQN still results in a MissingPropertyException . You can load the class using reflection, but this is kind of pointless as your script then cannot run in the sandbox. Calling GroovyClassLoader.parseClass does not seem to help.
            Hide
            jglick Jesse Glick added a comment -

            Could probably hack around this by returning a dummy object akin to Packages in Rhino which would dynamically resolve FQNs one package segment at a time, terminating in another placeholder from which you could access static fields or methods directly, or call a virtual new method to invoke constructors. But then you would still not be able to declare variables or fields of types defined in the library, for example. So it feels like much of the clarity of importing a class library is gone.

            Show
            jglick Jesse Glick added a comment - Could probably hack around this by returning a dummy object akin to Packages in Rhino which would dynamically resolve FQNs one package segment at a time, terminating in another placeholder from which you could access static fields or methods directly, or call a virtual new method to invoke constructors. But then you would still not be able to declare variables or fields of types defined in the library, for example. So it feels like much of the clarity of importing a class library is gone.
            jglick Jesse Glick made changes -
            Attachment LibraryStep-JENKINS-31155.diff [ 33673 ]
            Hide
            jglick Jesse Glick added a comment -

            Attached the library step implementation as far as I got with it.

            Show
            jglick Jesse Glick added a comment - Attached the library step implementation as far as I got with it.
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            Hide
            jglick Jesse Glick added a comment -

            Implementation ready. Includes all mandatory requirements specified above, and all optional requirements except the dynamic library step (see discussion above), and ability to load libraries from a non-SCM repository.

            Show
            jglick Jesse Glick added a comment - Implementation ready. Includes all mandatory requirements specified above, and all optional requirements except the dynamic library step (see discussion above), and ability to load libraries from a non-SCM repository.
            mkobit Mike Kobit made changes -
            Assignee Jesse Glick [ jglick ] Mike Kobit [ mkobit ]
            mkobit Mike Kobit made changes -
            Assignee Mike Kobit [ mkobit ] Jesse Glick [ jglick ]
            abayer Andrew Bayer made changes -
            Component/s pipeline-general [ 21692 ]
            abayer Andrew Bayer made changes -
            Component/s workflow-plugin [ 18820 ]
            jglick Jesse Glick made changes -
            Component/s workflow-cps-global-lib-plugin [ 21714 ]
            Component/s pipeline [ 21692 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "git PR 433 (Web Link)" [ 14801 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "subversion PR 168 (Web Link)" [ 14802 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "scm-api PR 12 (Web Link)" [ 14803 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "github-branch-source PR 72 (Web Link)" [ 14804 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "github-organization-folder PR 22 (Web Link)" [ 14805 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-cps PR 45 (Web Link)" [ 14806 ]
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-api PR 11 (Web Link)" [ 14807 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScript.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsWhitelist.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/EnvActionImpl.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/GlobalVariable.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/GlobalVariableSet.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/Snippetizer.java
            http://jenkins-ci.org/commit/workflow-cps-plugin/964295fc6e1becebf33a675c13673c7fdf58a842
            Log:
            JENKINS-31155 Extended GlobalVariableSet API to allow a Run context to be supplied.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScript.java src/main/java/org/jenkinsci/plugins/workflow/cps/CpsWhitelist.java src/main/java/org/jenkinsci/plugins/workflow/cps/EnvActionImpl.java src/main/java/org/jenkinsci/plugins/workflow/cps/GlobalVariable.java src/main/java/org/jenkinsci/plugins/workflow/cps/GlobalVariableSet.java src/main/java/org/jenkinsci/plugins/workflow/cps/Snippetizer.java http://jenkins-ci.org/commit/workflow-cps-plugin/964295fc6e1becebf33a675c13673c7fdf58a842 Log: JENKINS-31155 Extended GlobalVariableSet API to allow a Run context to be supplied.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScript.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/CpsWhitelist.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/DSL.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/EnvActionImpl.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/GlobalVariable.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/GlobalVariableSet.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/Snippetizer.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/replay/OriginalLoadedScripts.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/replay/ReplayAction.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/replay/ReplayFlowFactoryAction.java
            src/main/java/org/jenkinsci/plugins/workflow/cps/steps/LoadStepExecution.java
            src/main/resources/org/jenkinsci/plugins/workflow/cps/replay/ReplayAction/index.jelly
            http://jenkins-ci.org/commit/workflow-cps-plugin/1100650f3c9bc92e67b614ea5d06767bea40f296
            Log:
            Merge pull request #45 from jglick/shared-libs-JENKINS-31155

            JENKINS-31155 Infrastructure for shared libraries

            Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/4969ce0d0731...1100650f3c9b

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScript.java src/main/java/org/jenkinsci/plugins/workflow/cps/CpsWhitelist.java src/main/java/org/jenkinsci/plugins/workflow/cps/DSL.java src/main/java/org/jenkinsci/plugins/workflow/cps/EnvActionImpl.java src/main/java/org/jenkinsci/plugins/workflow/cps/GlobalVariable.java src/main/java/org/jenkinsci/plugins/workflow/cps/GlobalVariableSet.java src/main/java/org/jenkinsci/plugins/workflow/cps/Snippetizer.java src/main/java/org/jenkinsci/plugins/workflow/cps/replay/OriginalLoadedScripts.java src/main/java/org/jenkinsci/plugins/workflow/cps/replay/ReplayAction.java src/main/java/org/jenkinsci/plugins/workflow/cps/replay/ReplayFlowFactoryAction.java src/main/java/org/jenkinsci/plugins/workflow/cps/steps/LoadStepExecution.java src/main/resources/org/jenkinsci/plugins/workflow/cps/replay/ReplayAction/index.jelly http://jenkins-ci.org/commit/workflow-cps-plugin/1100650f3c9bc92e67b614ea5d06767bea40f296 Log: Merge pull request #45 from jglick/shared-libs- JENKINS-31155 JENKINS-31155 Infrastructure for shared libraries Compare: https://github.com/jenkinsci/workflow-cps-plugin/compare/4969ce0d0731...1100650f3c9b
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Manuel Recena
            Path:
            pom.xml
            src/main/java/hudson/scm/SubversionSCM.java
            src/main/java/jenkins/scm/impl/subversion/SubversionSCMSource.java
            src/main/resources/jenkins/scm/impl/subversion/SubversionSCMSource/config-detail.jelly
            src/test/java/jenkins/scm/impl/subversion/SubversionSCMSourceIntegrationTest.java
            src/test/java/jenkins/scm/impl/subversion/SubversionSampleRepoRule.java
            http://jenkins-ci.org/commit/subversion-plugin/755053989c9d0b97ef9f11696298a77d8baa875b
            Log:
            Merge pull request #168 from jglick/shared-libs-JENKINS-31155

            JENKINS-31155 Ability to retrieve a single revision

            Compare: https://github.com/jenkinsci/subversion-plugin/compare/e86d736814fe...755053989c9d

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Manuel Recena Path: pom.xml src/main/java/hudson/scm/SubversionSCM.java src/main/java/jenkins/scm/impl/subversion/SubversionSCMSource.java src/main/resources/jenkins/scm/impl/subversion/SubversionSCMSource/config-detail.jelly src/test/java/jenkins/scm/impl/subversion/SubversionSCMSourceIntegrationTest.java src/test/java/jenkins/scm/impl/subversion/SubversionSampleRepoRule.java http://jenkins-ci.org/commit/subversion-plugin/755053989c9d0b97ef9f11696298a77d8baa875b Log: Merge pull request #168 from jglick/shared-libs- JENKINS-31155 JENKINS-31155 Ability to retrieve a single revision Compare: https://github.com/jenkinsci/subversion-plugin/compare/e86d736814fe...755053989c9d
            Hide
            jglick Jesse Glick added a comment -

            Filed a separate WiP PR 14 for a library step.

            Show
            jglick Jesse Glick added a comment - Filed a separate WiP PR 14 for a library step.
            jglick Jesse Glick made changes -
            Remote Link This issue links to "workflow-cps-global-lib PR 14 (Web Link)" [ 14814 ]
            jglick Jesse Glick made changes -
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Resolved [ 5 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            demo/Dockerfile
            demo/JENKINS_HOME/org.jenkinsci.plugins.workflow.libs.GlobalLibraries.xml
            demo/plugins.txt
            demo/standard-build-lib/vars/standardBuild.groovy
            demo/workflow-libs/vars/standardBuild.groovy
            http://jenkins-ci.org/commit/github-branch-source-plugin/0a8cb257abd859bb7c7a8350e48a013ed2f7398c
            Log:
            JENKINS-31155 multibranch-demo working using a global external library.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: demo/Dockerfile demo/JENKINS_HOME/org.jenkinsci.plugins.workflow.libs.GlobalLibraries.xml demo/plugins.txt demo/standard-build-lib/vars/standardBuild.groovy demo/workflow-libs/vars/standardBuild.groovy http://jenkins-ci.org/commit/github-branch-source-plugin/0a8cb257abd859bb7c7a8350e48a013ed2f7398c Log: JENKINS-31155 multibranch-demo working using a global external library.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            demo/Dockerfile
            demo/Makefile
            demo/plugins.txt
            demo/workflow-libs/vars/standardBuild.groovy
            http://jenkins-ci.org/commit/github-branch-source-plugin/c04a9dd24c85ce447db16f66a5d7a710550821b0
            Log:
            Merge pull request #72 from jglick/shared-libs-JENKINS-31155

            JENKINS-31155 Use shared libraries

            Compare: https://github.com/jenkinsci/github-branch-source-plugin/compare/03df07cc04d4...c04a9dd24c85

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: demo/Dockerfile demo/Makefile demo/plugins.txt demo/workflow-libs/vars/standardBuild.groovy http://jenkins-ci.org/commit/github-branch-source-plugin/c04a9dd24c85ce447db16f66a5d7a710550821b0 Log: Merge pull request #72 from jglick/shared-libs- JENKINS-31155 JENKINS-31155 Use shared libraries Compare: https://github.com/jenkinsci/github-branch-source-plugin/compare/03df07cc04d4...c04a9dd24c85
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-38048 [ JENKINS-38048 ]
            djviking Sverre Moe made changes -
            Comment [ Is the change backward compatible? If I upgrade would workflowLibs.git still work or do I need to change it to use library step? ]
            Hide
            hieber Thomas Hieber added a comment -

            Seems like there is a Problem with the new function, that I can't configure which SVN credentials to use. The selection box für credentials stays empty. It doesn't show the predefined credentials, and even if I create new credentials, they won't show up.

            Show
            hieber Thomas Hieber added a comment - Seems like there is a Problem with the new function, that I can't configure which SVN credentials to use. The selection box für credentials stays empty. It doesn't show the predefined credentials, and even if I create new credentials, they won't show up.
            jglick Jesse Glick made changes -
            Link This issue is blocked by JENKINS-38167 [ JENKINS-38167 ]
            Hide
            mkobit Mike Kobit added a comment -

            I see the same problem.

            I can't configure either legacy or modern SCMs from the global configuration page.

            Show
            mkobit Mike Kobit added a comment - I see the same problem. I can't configure either legacy or modern SCMs from the global configuration page.
            Hide
            mkobit Mike Kobit added a comment -

            I actually haven't been able to configure any global libraries using Jenkins' `Global Configuration`

            Show
            mkobit Mike Kobit added a comment - I actually haven't been able to configure any global libraries using Jenkins' `Global Configuration`
            Hide
            valones Flávio Augusto Valones added a comment - - edited

            same problem here, can't configure legacy or modern and the options seem to be inverted.

            Show
            valones Flávio Augusto Valones added a comment - - edited same problem here, can't configure legacy or modern and the options seem to be inverted.
            Hide
            milehimikey Mike Key added a comment -

            +1

            Show
            milehimikey Mike Key added a comment - +1
            Hide
            mkobit Mike Kobit added a comment -

            Is there a separate issue tracking the above behavior (I'm guessing there should be)?

            Show
            mkobit Mike Kobit added a comment - Is there a separate issue tracking the above behavior (I'm guessing there should be)?
            Hide
            zot24 Israel Sotomayor added a comment -

            Same problem here when loading the library from Jenkinsfile I can see:

            Loading library github.com/moltin/jenkins-pipeline-library@master
            Creating git repository in /var/jenkins_home/caches/git-10e1b910805192749e5be85c3935e6d6
             > git init /var/jenkins_home/caches/git-10e1b910805192749e5be85c3935e6d6 # timeout=10
            Setting origin to https://github.com/moltin/jenkins-pipeline-library.git
            

            But when trying to use the Global Configuration the following it's thrown:

            Loading library github.com/moltin/jenkins-pipeline-library@branches/master
            java.lang.NullPointerException
            	at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:74)
            	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:150)
            	at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:131)
            

            So something different it's happening when loading the library from the Jenkinsfile vs Global configuration

            Show
            zot24 Israel Sotomayor added a comment - Same problem here when loading the library from Jenkinsfile I can see: Loading library github.com/moltin/jenkins-pipeline-library@master Creating git repository in / var /jenkins_home/caches/git-10e1b910805192749e5be85c3935e6d6 > git init / var /jenkins_home/caches/git-10e1b910805192749e5be85c3935e6d6 # timeout=10 Setting origin to https: //github.com/moltin/jenkins-pipeline-library.git But when trying to use the Global Configuration the following it's thrown: Loading library github.com/moltin/jenkins-pipeline-library@branches/master java.lang.NullPointerException at org.jenkinsci.plugins.workflow.libs.SCMSourceRetriever.retrieve(SCMSourceRetriever.java:74) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.retrieve(LibraryAdder.java:150) at org.jenkinsci.plugins.workflow.libs.LibraryAdder.add(LibraryAdder.java:131) So something different it's happening when loading the library from the Jenkinsfile vs Global configuration
            Hide
            mkobit Mike Kobit added a comment -

            Filed https://issues.jenkins-ci.org/browse/JENKINS-38424 for the issue I was seeing with not being able to configure global pipeline libraries.

            Show
            mkobit Mike Kobit added a comment - Filed https://issues.jenkins-ci.org/browse/JENKINS-38424 for the issue I was seeing with not being able to configure global pipeline libraries.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Jesse Glick
            Path:
            pom.xml
            src/main/java/jenkins/plugins/git/AbstractGitSCMSource.java
            src/test/java/jenkins/plugins/git/AbstractGitSCMSourceTest.java
            src/test/java/jenkins/plugins/git/GitSampleRepoRule.java
            http://jenkins-ci.org/commit/git-plugin/2b201d825c101b7ff6be71dad11d2d7d59ce1f81
            Log:
            JENKINS-31155 Implement new retrieve(String, TaskListener) overload.

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: pom.xml src/main/java/jenkins/plugins/git/AbstractGitSCMSource.java src/test/java/jenkins/plugins/git/AbstractGitSCMSourceTest.java src/test/java/jenkins/plugins/git/GitSampleRepoRule.java http://jenkins-ci.org/commit/git-plugin/2b201d825c101b7ff6be71dad11d2d7d59ce1f81 Log: JENKINS-31155 Implement new retrieve(String, TaskListener) overload.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            pom.xml
            src/main/java/hudson/plugins/git/UserRemoteConfig.java
            src/main/java/jenkins/plugins/git/AbstractGitSCMSource.java
            src/test/java/jenkins/plugins/git/AbstractGitSCMSourceTest.java
            src/test/java/jenkins/plugins/git/GitSampleRepoRule.java
            http://jenkins-ci.org/commit/git-plugin/0c6d93913a70007c8061a5a37438549d72bede9f
            Log:
            Merge pull request #433 from jglick/retrieve-revision-JENKINS-31155

            JENKINS-31155 Ability to retrieve a single revision

            Compare: https://github.com/jenkinsci/git-plugin/compare/2025127d7e82...0c6d93913a70

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: pom.xml src/main/java/hudson/plugins/git/UserRemoteConfig.java src/main/java/jenkins/plugins/git/AbstractGitSCMSource.java src/test/java/jenkins/plugins/git/AbstractGitSCMSourceTest.java src/test/java/jenkins/plugins/git/GitSampleRepoRule.java http://jenkins-ci.org/commit/git-plugin/0c6d93913a70007c8061a5a37438549d72bede9f Log: Merge pull request #433 from jglick/retrieve-revision- JENKINS-31155 JENKINS-31155 Ability to retrieve a single revision Compare: https://github.com/jenkinsci/git-plugin/compare/2025127d7e82...0c6d93913a70
            Hide
            markewaite Mark Waite added a comment -

            Released in git plugin 3.0.1 18 Nov 2016

            Show
            markewaite Mark Waite added a comment - Released in git plugin 3.0.1 18 Nov 2016
            markewaite Mark Waite made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            jglick Jesse Glick made changes -
            Link This issue is duplicated by JENKINS-37270 [ JENKINS-37270 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-57023 [ JENKINS-57023 ]

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              kohsuke Kohsuke Kawaguchi
              Votes:
              17 Vote for this issue
              Watchers:
              40 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: