Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-37220

Job DSL analogue of multibranch projects & organization folders

      To provide a more scalable alternative to the endless requested branch-api customizations such as JENKINS-32396, or the tricks based on JENKINS-30519, it would be useful to have a simple way of calling into scm-api implementations from a Job DSL folder (JENKINS-33275). So you could (for example) easily construct the equivalent of a GitHub organization folder, but from a script loop that could do arbitrary customizations to each, such as setting job properties.

      Need to also implement SCMSourceOwner, HeadByItem, etc. so that the branch source knows that a given generated project is associated with a given branch, which is needed for various purposes.

      TBD how webhooks should be integrated. The script needs to somehow indicate that changes to a given repository or organization should trigger a regeneration of the folder. May need to create an alternative to SCMTrigger that is keyed off the same criteria that branch-api branch indexing uses, though there are also use cases for traditional SCM polling—it is a little less efficient, but has more configuration options.

      For Pipeline projects you could use Jenkinsfile, if SCMBinder and SCMVar were generalized a bit, or something else.

      Needs a lot of design work; at this point this is more of a placeholder issue to collect ideas.

          [JENKINS-37220] Job DSL analogue of multibranch projects & organization folders

          Jesse Glick created issue -
          Jesse Glick made changes -
          Epic Link New: JENKINS-35386 [ 171179 ]
          Jesse Glick made changes -
          Link New: This issue is blocked by JENKINS-33275 [ JENKINS-33275 ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-30519 [ JENKINS-30519 ]
          Jesse Glick made changes -
          Link New: This issue relates to JENKINS-32396 [ JENKINS-32396 ]
          Andrew Bayer made changes -
          Labels Original: multibranch workflow New: multibranch pipeline workflow
          Andrew Bayer made changes -
          Labels Original: multibranch pipeline workflow New: multibranch pipeline
          Jesse Glick made changes -
          Link New: This issue is duplicated by JENKINS-39750 [ JENKINS-39750 ]

          Use case:

          Joe creates a GitHub Org Folder.  His repos foo and bar are scanned and picked up because they have a Jenkinsfile in each.

          foo is a huge repository (2gb) and he only wants the last 10 commits when building (a shallow clone with depth 10) and without any tags.

          bar uses lots of submodules, but isn't big.  He wants the submodule extensions turned on and to use the GitHub Org credentials.

          This could be done by allow adding extensions at the GitHub Org Folder level, but that would mean forcing all builds to have a depth of 10, no tags, and submodules turned on.  This could cause people to split the GitHub Orgs up just to accommodate Jenkins (which may not be possible, if they are paying for the Orgs, etc.)

          An idea is we could change the scm from an Object to the list form (I don't know what it is called) that looks like [$class:GitSCM, extensions: ... ]

          I have some code I run in a trusted pipeline via the src/ directory that almost does this...

           class ScmEditablizer {
               static def makeScmEditable(scm, env) {
                  Integer i
          
                  def branches = []
                  for (i = 0; i < scm.branches.size(); i++) {
                      def it = scm.branches.get(i)
                      branches.add([name: it.getName()])
                  }
          
                  def userRemoteConfigs = []
                  for (i = 0; i < scm.userRemoteConfigs.size(); i++) {
                      def it = scm.userRemoteConfigs.get(i)
                      userRemoteConfigs.add([
                      name:          it.getName(),
                      url:           it.getUrl(),
                      credentialsId: it.getCredentialsId(),
                      refspec:       it.getRefspec()
                      ])
                  }
          
                  return [ $class: 'GitSCM', extensions: [], branches: branches, userRemoteConfigs: userRemoteConfigs ]
              }
          }
          

          But this doesn't work because the extensions are missing and it doesn't understand Pull Requests (I don't know why exactly).

          This allows getting an "editable" scm object that can have extensions, etc. added to it.

           

          Christian Höltje added a comment - Use case: Joe creates a GitHub Org Folder.  His repos foo and bar are scanned and picked up because they have a Jenkinsfile in each. foo is a huge repository (2gb) and he only wants the last 10 commits when building (a shallow clone with depth 10) and without any tags. bar  uses lots of submodules, but isn't big.  He wants the submodule extensions turned on and to use the GitHub Org credentials. This could be done by allow adding extensions at the GitHub Org Folder level, but that would mean forcing all builds to have a depth of 10, no tags, and submodules turned on.  This could cause people to split the GitHub Orgs up just to accommodate Jenkins (which may not be possible, if they are paying for the Orgs, etc.) An idea is we could change the scm from an Object to the list form (I don't know what it is called) that looks like [$class:GitSCM, extensions: ... ] I have some code I run in a trusted pipeline via the src/ directory that almost  does this...   class ScmEditablizer { static def makeScmEditable(scm, env) { Integer i def branches = [] for (i = 0; i < scm.branches.size(); i++) { def it = scm.branches.get(i) branches.add([name: it.getName()]) } def userRemoteConfigs = [] for (i = 0; i < scm.userRemoteConfigs.size(); i++) { def it = scm.userRemoteConfigs.get(i) userRemoteConfigs.add([ name: it.getName(), url: it.getUrl(), credentialsId: it.getCredentialsId(), refspec: it.getRefspec() ]) } return [ $class: 'GitSCM' , extensions: [], branches: branches, userRemoteConfigs: userRemoteConfigs ] } } But this doesn't work because the extensions are missing and it doesn't understand Pull Requests (I don't know why exactly). This allows getting an "editable" scm object that can have extensions, etc. added to it.  

          Michael Neale added a comment -

          Is the aim here to "kill job DSL"? 

           

          "have a simple way of calling into scm-api implementations from a Job DSL folder" - interesting use of "simple". I would say "for very advanced users". 

          Michael Neale added a comment - Is the aim here to "kill job DSL"?    "have a simple way of calling into scm-api implementations from a Job DSL folder" - interesting use of "simple". I would say "for very advanced users". 

            jamietanna Jamie Tanna
            jglick Jesse Glick
            Votes:
            11 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated: