Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-40695

Allow lightweight or remote checkout for Groovy libraries

      New feature suggestion.

      Currently, all shared library functions seem to execute on the master.

      Allow defining some sort of mechanism to allow specifying an executor for the code execution.

      Some ideas:

      • by defining this in the library code (something similar to what node(...) definitions do)
      • via a shared library form field that allows defining the agent to run on
      • checkbox on the shared library form that toggles whether code should be run same node as calling Pipeline block

          [JENKINS-40695] Allow lightweight or remote checkout for Groovy libraries

          Ronen Peleg added a comment -

          For me, Libraries = GIT Repository which means temporary files to the Master! without cleanup control system.
           
          To download repository (call it library if you want but it's a fully source repository files) to the master without properly cleanup system it's a recipe for disaster! Also the Master need to stay clean all the time without a jobs/builds temporary files as library... 

          I also like the suggestion of add new option to run all the pipeline project in specific node, as we already have and exist in normal Jenkins job: "Restrict where this project can be run". so simple so smart.

          There must be a way to allow users to control the node to where download the library...

          I don't see any reason that you prevent us from getting this new option.

           

          Ronen Peleg added a comment - For me, Libraries = GIT Repository which means temporary files to the Master! without cleanup control system.   To download repository (call it library if you want but it's a fully source repository files) to the master without properly cleanup system it's a recipe for disaster! Also the Master need to stay clean all the time without a jobs/builds temporary files as library...  I also like the suggestion of add new option to run all the pipeline project in specific node, as we already have and exist in normal Jenkins job: "Restrict where this project can be run". so simple so smart. There must be a way to allow users to control the node to where download the library... I don't see any reason that you prevent us from getting this new option.  

          Amir Barkal added a comment - - edited

          I concur. Here's another use case for this request:

          When we're developing our CI pipeline, we want to make sure we don't introduce regressions to Jenkins shared libraries, so we have a pipeline that's defined as a gate for our shared library changes. We want to be able to checkout the same library version of the tested branch, and run it for verification. In the below example we have usefulMethod that does some linting on our groovy code inside the shared library repo. The method will fail because it expects the shared library code to be available on "builder1" but the code was checked-out to the Jenkins maser node. 

          node("builder1") {

            // This will checkout the shared library into the master node, despite being inside an
            //explicit node block:
            lib = library("jenkins-shared-library@${env.BRANCH_NAME}")

            // test that usefulMethod did not break:
            lib.usefulMethod()
          }

          In order to workaround this we need to checkout the shared library code twice, first time to load the library classes and methods and a second time to bring the code to the current workspace. This of course is not very efficient and puts unnecessary stress on our source control system.

          Amir Barkal added a comment - - edited I concur. Here's another use case for this request: When we're developing our CI pipeline, we want to make sure we don't introduce regressions to Jenkins shared libraries, so we have a pipeline that's defined as a gate for our shared library changes. We want to be able to checkout the same library version of the tested branch, and run it for verification. In the below example we have usefulMethod that does some linting on our groovy code inside the shared library repo. The method will fail because it expects the shared library code to be available on "builder1" but the code was checked-out to the Jenkins maser node.  node("builder1") {   // This will checkout the shared library into the master node, despite being inside an   //explicit node block:   lib = library("jenkins-shared-library@${env.BRANCH_NAME}")   // test that usefulMethod did not break:   lib.usefulMethod() } In order to workaround this we need to checkout the shared library code twice, first time to load the library classes and methods and a second time to bring the code to the current workspace. This of course is not very efficient and puts unnecessary stress on our source control system.

          Steven G added a comment -

          jglick

          We've just run into this issue in trying to build a utility library with straight groovy. Everything was working flawlessly until we started trying to manipulate files on the node from the groovy library.

          Is there really no valid use-case? We have a pretty large CI workflow we are developing and our Jenkinsfile is going to get large pretty quickly. The idea in outsourcing work to the groovy lib was to utilize some object hierarchies.

          I know this isn't a support forum, but shared libs are intended to only run on master?

          Steven G added a comment - jglick We've just run into this issue in trying to build a utility library with straight groovy. Everything was working flawlessly until we started trying to manipulate files on the node from the groovy library. Is there really no valid use-case? We have a pretty large CI workflow we are developing and our Jenkinsfile is going to get large pretty quickly. The idea in outsourcing work to the groovy lib was to utilize some object hierarchies. I know this isn't a support forum, but shared libs are intended to only run on master?

          ShiMing Qu added a comment -

          We have the similar issue. Our requirement is running the shared lib code inside a Docker container( via Dockerfile agent )

          ShiMing Qu added a comment - We have the similar issue. Our requirement is running the shared lib code inside a Docker container( via Dockerfile agent )

          Nikita Kunets added a comment -

          Same problem.

          I have to write monstrous pipelines (instead of just encapsulate similar functionality in shared library methods and use them) to be able to run them on slaves in kubernetes cluster.

          Nikita Kunets added a comment - Same problem. I have to write monstrous pipelines (instead of just encapsulate similar functionality in shared library methods and use them) to be able to run them on slaves in kubernetes cluster.

          Steven G added a comment -

          dkder3k

          We found a workaround that may help you out. We've written Functions.groovy and at the start of each pipeline, we pull the groovy files from a separate git repo and run the following:

          def functions = load "./scripts/Functions.groovy"
          

          This allows you to call the functions defined in Functions.groovy using functions.functionName(). The only requirement is to write the functions in the groovy file and 

          return this
          

          at the very end of the script.

          At this point, we're basically re-creating the functionality of shared libraries. I'd like to hear how jglick imagines we should be doing this, we've got a few thousand lines of code spread across 6 external groovy files now, on top of our 500 line jenkinsfile that runs the CI flow. If there's a better way, I'm all ears.

          Steven G added a comment - dkder3k We found a workaround that may help you out. We've written Functions.groovy and at the start of each pipeline, we pull the groovy files from a separate git repo and run the following: def functions = load "./scripts/Functions.groovy" This allows you to call the functions defined in Functions.groovy using functions.functionName(). The only requirement is to write the functions in the groovy file and  return this at the very end of the script. At this point, we're basically re-creating the functionality of shared libraries. I'd like to hear how jglick imagines we should be doing this, we've got a few thousand lines of code spread across 6 external groovy files now, on top of our 500 line jenkinsfile that runs the CI flow. If there's a better way, I'm all ears.

          Nikita Kunets added a comment -

          steve_dev thank you!
          I know this workaround, but in my book if there is such functionality as shared libraries I'd prefer to use it instead of writing workarounds.
          I suppose, in master-slave system it is strange for the master to do something except job management and shared libraries are the way to

          share parts of Pipelines between various projects to reduce redundancies and keep code "DRY"

          So I can't understand why this code should be executed only on the master

          Nikita Kunets added a comment - steve_dev thank you! I know this workaround, but in my book if there is such functionality as shared libraries I'd prefer to use it instead of writing workarounds. I suppose, in master-slave system it is strange for the master to do something except job management and shared libraries are the way to share parts of Pipelines between various projects to reduce redundancies and keep code "DRY" So I can't understand why this code should be executed only on the master

          Jesse Glick added a comment -

          The issue title is misleading. Yes you can use the load step.

          The standard library retrievers could probably be made to use lightweight checkouts with supported SCMs like GitHub, thus avoiding the need to maintain a local copy anywhere. Not sure if retrieval of source files could be made lazy, though.

          Making the library step use a contextual node might work, with some API changes, though it would introduce security issues if not done carefully—could not be offered for trusted libraries. Also would not work for @Library which must be done prior to script parsing.

          Jesse Glick added a comment - The issue title is misleading. Yes you can use the load step. The standard library retrievers could probably be made to use lightweight checkouts with supported SCMs like GitHub, thus avoiding the need to maintain a local copy anywhere. Not sure if retrieval of source files could be made lazy, though. Making the library step use a contextual node might work, with some API changes, though it would introduce security issues if not done carefully—could not be offered for trusted libraries. Also would not work for @Library which must be done prior to script parsing.

          Similar use case here; I am trying to identify the files changed on my last (git scm) changeset, and perform some greps on them, but they are on a different node.

          Would it be possible to get an ETA for this? (in case it will be implemented) 

          Cesar Martinez added a comment - Similar use case here; I am trying to identify the files changed on my last (git scm) changeset, and perform some greps on them, but they are on a different node. Would it be possible to get an ETA for this? (in case it will be implemented) 

          Jesse Glick added a comment -

          No plans for anything like this, though see JENKINS-70869.

          Jesse Glick added a comment - No plans for anything like this, though see JENKINS-70869 .

            Unassigned Unassigned
            owood Owen Wood
            Votes:
            9 Vote for this issue
            Watchers:
            18 Start watching this issue

              Created:
              Updated:
              Resolved: