Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43749

Support multiple Jenkinsfiles from the same repository

      This would support scenarios where different "configurations" of a pipeline cannot share the same Jenkinsfile.

      If I had multiple Jenkinsfiles... repository github.com/apple/swift

      /Package.jenkinsfile 
      /Incremental.jenkinsfile
      /Incremental-RA.jenkinsfile
      /Assert.jenkinsfile
      /src/…
      

      I would like to create multibranch Pipelines for each so I have the resulting structure:

      /Apple
      /Apple/Swift - Package
      /Apple/Swift - Incremental
      /Apple/Swift - Incremental-RA
      /Apple/Swfit - Assert
      

      Note that in this example I have an organization folder for github.com/apple and it is creating multiple multibranch pipelines for each Jenkinsfile discovered in each repository.

      I have written up examples and use cases in this doc

          [JENKINS-43749] Support multiple Jenkinsfiles from the same repository

          James Dumay created issue -
          James Dumay made changes -
          Description New: This would support scenarios where different "configurations" of a pipeline cannot share the same Jenkinsfile.

          If I had multiple Jenkinsfiles like so:
          {code}
          github.com/apple/swift
          /Package.jenkinsfile
          /Incremental.jenkinsfile
          /Incremental-RA.jenkinsfile
          /Assert.jenkinsfile
          /src/…
          {code}

          I would like to create multibranch Pipelines for each so that:
          {code}
          Item structure
          /Apple
          /Apple/Swift - Package
          /Apple/Swift - Incremental
          /Apple/Swift - Incremental-RA
          /Apple/Swfit - Assert
          {code}

          In depth [use cases are provided in this doc|https://docs.google.com/document/d/1a3Q8i15d9umxZjJVU7gygqmsHiFQX2lOT0l-xpZ5tT8/edit?usp=sharing]
          James Dumay made changes -
          Description Original: This would support scenarios where different "configurations" of a pipeline cannot share the same Jenkinsfile.

          If I had multiple Jenkinsfiles like so:
          {code}
          github.com/apple/swift
          /Package.jenkinsfile
          /Incremental.jenkinsfile
          /Incremental-RA.jenkinsfile
          /Assert.jenkinsfile
          /src/…
          {code}

          I would like to create multibranch Pipelines for each so that:
          {code}
          Item structure
          /Apple
          /Apple/Swift - Package
          /Apple/Swift - Incremental
          /Apple/Swift - Incremental-RA
          /Apple/Swfit - Assert
          {code}

          In depth [use cases are provided in this doc|https://docs.google.com/document/d/1a3Q8i15d9umxZjJVU7gygqmsHiFQX2lOT0l-xpZ5tT8/edit?usp=sharing]
          New: This would support scenarios where different "configurations" of a pipeline cannot share the same Jenkinsfile.

          If I had multiple Jenkinsfiles like so:
          {code}
          github.com/apple/swift
          /Package.jenkinsfile
          /Incremental.jenkinsfile
          /Incremental-RA.jenkinsfile
          /Assert.jenkinsfile
          /src/…
          {code}

          I would like to create multibranch Pipelines for each so I have the resulting structure:
          {code}
          /Apple
          /Apple/Swift - Package
          /Apple/Swift - Incremental
          /Apple/Swift - Incremental-RA
          /Apple/Swfit - Assert
          {code}

          Note that in this example I have an organization folder for github.com/apple and it is creating multiple multibranch pipelines for each Jenkinsfile discovered in each repository.

          In depth [use cases are provided in this doc|https://docs.google.com/document/d/1a3Q8i15d9umxZjJVU7gygqmsHiFQX2lOT0l-xpZ5tT8/edit?usp=sharing]
          James Dumay made changes -
          Description Original: This would support scenarios where different "configurations" of a pipeline cannot share the same Jenkinsfile.

          If I had multiple Jenkinsfiles like so:
          {code}
          github.com/apple/swift
          /Package.jenkinsfile
          /Incremental.jenkinsfile
          /Incremental-RA.jenkinsfile
          /Assert.jenkinsfile
          /src/…
          {code}

          I would like to create multibranch Pipelines for each so I have the resulting structure:
          {code}
          /Apple
          /Apple/Swift - Package
          /Apple/Swift - Incremental
          /Apple/Swift - Incremental-RA
          /Apple/Swfit - Assert
          {code}

          Note that in this example I have an organization folder for github.com/apple and it is creating multiple multibranch pipelines for each Jenkinsfile discovered in each repository.

          In depth [use cases are provided in this doc|https://docs.google.com/document/d/1a3Q8i15d9umxZjJVU7gygqmsHiFQX2lOT0l-xpZ5tT8/edit?usp=sharing]
          New: This would support scenarios where different "configurations" of a pipeline cannot share the same Jenkinsfile.

          If I had multiple Jenkinsfiles like so:
          {code}
          github.com/apple/swift
          /Package.jenkinsfile
          /Incremental.jenkinsfile
          /Incremental-RA.jenkinsfile
          /Assert.jenkinsfile
          /src/…
          {code}

          I would like to create multibranch Pipelines for each so I have the resulting structure:
          {code}
          /Apple
          /Apple/Swift - Package
          /Apple/Swift - Incremental
          /Apple/Swift - Incremental-RA
          /Apple/Swfit - Assert
          {code}

          Note that in this example I have an organization folder for github.com/apple and it is creating multiple multibranch pipelines for each Jenkinsfile discovered in each repository.

          I have written up examples and [use cases in this doc|https://docs.google.com/document/d/1a3Q8i15d9umxZjJVU7gygqmsHiFQX2lOT0l-xpZ5tT8/edit?usp=sharing]

          James Dumay added a comment -

          hrmpw this one keeps coming up all over the place (most recently on gitter) so I thought that we should share the internal document more widely with the community.

          James Dumay added a comment - hrmpw this one keeps coming up all over the place (most recently on gitter) so I thought that we should share the internal document more widely with the community.

          Alex Lourie added a comment -

          Just to add a bit more clarity for our use case:

           

          We have the main repository, src. In that repository, we have a code for all modules of our app, the code for packaging it in the distribution packages, and tests. We have a few distinct Jenkins jobs in that repo:

          • A job that runs tests. It can accept parameters (such as branch to checkout to run the tests on, the level of tests - small, large or medium, and components - for specifying a subset of the project to run the tests on). This job is run both manually as a part of some operational procedures, and automatically when executed from another job.
          • A build job, that builds artefacts. It always runs manually. It accepts parameters defining branch, environment (test/prod), set of modules to build and set of packages to create.
          • A "cron" job that runs subsets of tests periodically. These subsets are very resource-heavy, and cannot be run often. Mostly runs automatically on a scheduler, but also can be run manually if needed.
          • A subproject in a different repo that's being automatically or manually run by a build job with a set of dynamic parameters.

           

          At the moment, I cannot see a way to implement these in 1 Jenkins file. Especially combining a build job (which only should output success/failure) with tests (which should show results of passed/failed tests) into 1 pipeline doesn't make sense in our environment.

          Alex Lourie added a comment - Just to add a bit more clarity for our use case:   We have the main repository, src. In that repository, we have a code for all modules of our app, the code for packaging it in the distribution packages, and tests. We have a few distinct Jenkins jobs in that repo: A job that runs tests. It can accept parameters (such as branch to checkout to run the tests on, the level of tests - small, large or medium, and components - for specifying a subset of the project to run the tests on). This job is run both manually as a part of some operational procedures, and automatically when executed from another job. A build job, that builds artefacts. It always runs manually. It accepts parameters defining branch, environment (test/prod), set of modules to build and set of packages to create. A "cron" job that runs subsets of tests periodically. These subsets are very resource-heavy, and cannot be run often. Mostly runs automatically on a scheduler, but also can be run manually if needed. A subproject in a different repo that's being automatically or manually run by a build job with a set of dynamic parameters.   At the moment, I cannot see a way to implement these in 1 Jenkins file. Especially combining a build job (which only should output success/failure) with tests (which should show results of passed/failed tests) into 1 pipeline doesn't make sense in our environment.
          James Dumay made changes -
          Description Original: This would support scenarios where different "configurations" of a pipeline cannot share the same Jenkinsfile.

          If I had multiple Jenkinsfiles like so:
          {code}
          github.com/apple/swift
          /Package.jenkinsfile
          /Incremental.jenkinsfile
          /Incremental-RA.jenkinsfile
          /Assert.jenkinsfile
          /src/…
          {code}

          I would like to create multibranch Pipelines for each so I have the resulting structure:
          {code}
          /Apple
          /Apple/Swift - Package
          /Apple/Swift - Incremental
          /Apple/Swift - Incremental-RA
          /Apple/Swfit - Assert
          {code}

          Note that in this example I have an organization folder for github.com/apple and it is creating multiple multibranch pipelines for each Jenkinsfile discovered in each repository.

          I have written up examples and [use cases in this doc|https://docs.google.com/document/d/1a3Q8i15d9umxZjJVU7gygqmsHiFQX2lOT0l-xpZ5tT8/edit?usp=sharing]
          New: This would support scenarios where different "configurations" of a pipeline cannot share the same Jenkinsfile.

          If I had multiple Jenkinsfiles... repository github.com/apple/swift
          {code}
          /Package.jenkinsfile
          /Incremental.jenkinsfile
          /Incremental-RA.jenkinsfile
          /Assert.jenkinsfile
          /src/…
          {code}

          I would like to create multibranch Pipelines for each so I have the resulting structure:
          {code}
          /Apple
          /Apple/Swift - Package
          /Apple/Swift - Incremental
          /Apple/Swift - Incremental-RA
          /Apple/Swfit - Assert
          {code}

          Note that in this example I have an organization folder for github.com/apple and it is creating multiple multibranch pipelines for each Jenkinsfile discovered in each repository.

          I have written up examples and [use cases in this doc|https://docs.google.com/document/d/1a3Q8i15d9umxZjJVU7gygqmsHiFQX2lOT0l-xpZ5tT8/edit?usp=sharing]

          Seems today the only way to get this scenario running is by changing the build behavior entirely by switching based on the jobs naming pattern (wich does feel like bad design).

          Anyways I am curious about that internal document.

          Daniel Geißler added a comment - Seems today the only way to get this scenario running is by changing the build behavior entirely by switching based on the jobs naming pattern (wich does feel like bad design). Anyways I am curious about that internal document.

          Patrick Wolf added a comment - - edited

          Adding the ability to recognize other files is not difficult. The problem is that most people don't want to run N jobs every time a commit happens. 

          Pipeline was designed to have flow control using when and if to take different actions based on different conditions. The primary reason the Jenkinsfile exists is to tell Jenkins, create a new Pipeline for this branch. What happens in that Pipeline is entirely up to author of the Pipeline.

          This takes into account build parameters, triggers, branches, upstream stage success, running external jobs etc.  

          E.g.

          when { branch "master"}

          Pipeline was not meant to replace Freestyle jobs 1:1 but to collapse multiple Freestyle jobs into a single cohesive flow that adapted to the conditions. With all that is possible in Pipeline a single Jenkinsfile can account for most scenarios,

          Moving all of that flow control outside of the actual Pipeline essentially wraps Pipeline into a Freestyle configuration paradigm and we lose all of the advantages of Pipeline as Code

          1. The Pipelines are no longer portable 
          2. The UI becomes more complex by an order of magnitude
          3. Changes to the flow, triggers, parameters etc of the different Pipeline branches lose code review and audit capabilities

          I understand the desire to have separate build histories and change histories for discrete Pipelines, since those are the primary value-adds of having multiple files.  If you don't care about separate build histories you can already use the Jenkinsfile to determine the type of build to perform and then load a specific, discrete Pipeline file from the workspace to execute. You can even change the currentBuild.displayName = type of job that is being run.

          Just adding multiple Jenkinsfiles as a solution creates more problems than it solves. We need to examine the set of problems that are not satisfied by Jenkinsfile today and come up with an actual solution that also meets the goal of Pipeline-as-code.

           

           

          Patrick Wolf added a comment - - edited Adding the ability to recognize other files is not difficult. The problem is that most people don't want to run N jobs every time a commit happens.  Pipeline was designed to have flow control using when and if to take different actions based on different conditions. The primary reason the Jenkinsfile exists is to tell Jenkins, create a new Pipeline for this branch. What happens in that Pipeline is entirely up to author of the Pipeline. This takes into account build parameters, triggers, branches, upstream stage success, running external jobs etc.   E.g. when { branch "master" } Pipeline was not meant to replace Freestyle jobs 1:1 but to collapse multiple Freestyle jobs into a single cohesive flow that adapted to the conditions. With all that is possible in Pipeline a single Jenkinsfile can account for most scenarios, Moving all of that flow control outside of the actual Pipeline essentially wraps Pipeline into a Freestyle configuration paradigm and we lose all of the advantages of Pipeline as Code The Pipelines are no longer portable  The UI becomes more complex by an order of magnitude Changes to the flow, triggers, parameters etc of the different Pipeline branches lose code review and audit capabilities I understand the desire to have separate build histories and change histories for discrete Pipelines, since those are the primary value-adds of having multiple files.  If you don't care about separate build histories you can already use the Jenkinsfile to determine the type of build to perform and then load a specific, discrete Pipeline file from the workspace to execute. You can even change the currentBuild.displayName = type of job that is being run. Just adding multiple Jenkinsfiles as a solution creates more problems than it solves. We need to examine the set of problems that are not satisfied by Jenkinsfile today and come up with an actual solution that also meets the goal of Pipeline-as-code.    
          Alex Lourie made changes -
          Attachment New: image-2017-04-27-12-23-08-404.png [ 37461 ]

            Unassigned Unassigned
            jamesdumay James Dumay
            Votes:
            105 Vote for this issue
            Watchers:
            129 Start watching this issue

              Created:
              Updated: