• Icon: Epic Epic
    • Resolution: Fixed
    • Icon: Major Major
    • pipeline
    • 2.0: Pipeline as Code

      Problem definition

      The default interaction model with Jenkins has been very web UI driven, requiring users to manually create jobs, then manually fill in the details through a web browser. This requires large amounts of effort to create and manage jobs to test and build multiple projects and keeps the actual configuration of a job to build/test/deploy a project separate from the actual code being built/tested/deployed. This prevents users from applying their existing CI/CD best practices to the job configurations themselves.

      Proposal

      Introduce a new subsystem in Jenkins that:

      • lets you design a whole pipeline, not just a single linear set of tasks
      • stores the said pipeline configuration as human-editable Jenkinsfile in your SCM
      • makes it automatic to set up new pipelines when Jenkinsfile is added
      • differentiates multiple branches in the same repository

      This is the key new feature that positions Jenkins for continuous delivery use cases and other more complex automations of today.

      Overview

      (This overview uses GitHub as an example although the system is extensible to allow implementations for other SCMs and repository hostings.)

      1. In this new subsystem, you first let Jenkins know where you host your source code repositories. You do this by creating a new special folder called "Organization Folder" and selecting "GitHub organization." This sets up a new smart folder.
      2. In one of the repository in the chosen org, create a Jenkinsfile that defines your pipeline (or just a build process to start with.) This file is actually a workflow script. See workflow tutorial for details.
      3. Jenkins will automatically recognize this branch and create appropriate jobs by itself.
      4. Everytime a new change is pushed to this branch, the branch is built and the commit status gets updated.
      5. When the branch is destroyed in the repository or if Jenkinsfile is removed, the corresponding job gets destroyed from Jenkins as well automatically.

      In this use, there'll be a lot of Jenkinsfile in various branches & repositories. To keep them DRY, various mechanisms will be provided to promote reuse of workflow scripts, such as this.

      For more details, see jglick's slides and video recording that talks about this feature (in particular, where the demo starts.) There's also docker-based demo that you can play with on your own.

      Contents

      • Thus, this feature should be made available out of the box by default, unless users opt out (2.0 Out-of-the-box experience)
      • Provide better workflow visualization out of the box (JENKINS-31154)
      • Workflow gets renamed to Pipeline, so that users understand what it is without explanation (JENKINS-31153)
      • Shared library improvements to simplify Jenkinsfile (JENKINS-31155)

      Internals

      • The execution engine of this is Workflow plugin (see JENKINS-26129)
      • Multi-branch workflow project type defines a new kind of folder that is associated with a single source code repository and automatically create/destory a job inside as branches are created/destroyed in the repository.
      • Organization folder that defines a new kind of folder that is associated with a group of source code repositories (such as GitHub org) and automatically create/destory the above "multi-branch workflow project" as repositories are created/destroyed in the group.
      • Branch API plugin and SCM API plugin defines a contract for version control systems (such as Git) and hosting (such as GitHub) to define branch discovery, repository discovery, etc.
      • CloudBees GitHub Branch Source Plugin implements the above contract for GitHub, so that GitHub organization can be added as an organization folder to automatically scan every repo & branch in it. Other similar plugins can be developed for other hosting options and other source code management tools.

      Impact

      Open Questions

          [JENKINS-31152] 2.0: Pipeline as code in front & center

          John Mellor added a comment -

          I have one project that has ~130 sub-builds, in a fairly complex ordering tree with some build threads in parallel and then some in serial, and then some in parallel again. Almost all steps are freestyle builds running autotools and make or a modified debuild. Are we going to be able to bring the job steps together into several choke-points throughout the pipeline, or is the proposed solution going to be more simplistic?

          John Mellor added a comment - I have one project that has ~130 sub-builds, in a fairly complex ordering tree with some build threads in parallel and then some in serial, and then some in parallel again. Almost all steps are freestyle builds running autotools and make or a modified debuild. Are we going to be able to bring the job steps together into several choke-points throughout the pipeline, or is the proposed solution going to be more simplistic?

          Jesse Glick added a comment -

          jmellor the “proposed” solution already exists and is available for use, so I am not sure why the description was worded the way it was. This is not the place to ask for technical help.

          Jesse Glick added a comment - jmellor the “proposed” solution already exists and is available for use , so I am not sure why the description was worded the way it was. This is not the place to ask for technical help.

          FYI, we manage more than 1500 jobs at my client today and we have designed the Seed plugin to go even further in the simplification of the job generation and management, by providing development teams a way to provide at SCM level a shopping list (a property file) which refers to a versioned pipeline library maintained by a core team.

          This pipeline library uses the Job DSL plugin but could also use the Workflow plugin. The idea behind the Seed plugin is to manage branches and projects on a very large scale, for people who do not know very well Jenkins.

          Feel free to check it at https://wiki.jenkins-ci.org/display/JENKINS/Seed+Plugin

          I'm very interested in the development of this new core integration of the pipeline as code, and see how I could in the future integrate it with this concept of shopping list.

          Damien.

          Damien Coraboeuf added a comment - FYI, we manage more than 1500 jobs at my client today and we have designed the Seed plugin to go even further in the simplification of the job generation and management, by providing development teams a way to provide at SCM level a shopping list (a property file) which refers to a versioned pipeline library maintained by a core team. This pipeline library uses the Job DSL plugin but could also use the Workflow plugin. The idea behind the Seed plugin is to manage branches and projects on a very large scale, for people who do not know very well Jenkins. Feel free to check it at https://wiki.jenkins-ci.org/display/JENKINS/Seed+Plugin I'm very interested in the development of this new core integration of the pipeline as code, and see how I could in the future integrate it with this concept of shopping list. Damien.

          Jason Fowler added a comment - - edited

          I have two comments for "Pipeline as code":

          1) It's an excellent idea. A DSL for the "Jenkinsfile" is very important. There is a need for "Build and Deploy as a Service" for organizations who are limited by compliance to use cloud-based systems. Simply dropping a Jenkinsfile into a source code repo should be enough to trigger a build in Jenkins. The DSL should be simple to use like it is in, for example, TravisCI. Groupon created "DotCI" and would be a great reference for some of the features required to make it a success.

          2) Metrics. This is a somewhat related feature for "Pipeline as code" because the DSL in the Jenkinsfile should have the ability to configure build statistics and metrics. For example, let's say I have several jobs in Jenkins that perform builds, tests, and deployment to staging. This is my "Continuous Delivery Pipeline". I want to measure how long it takes from the initial commit to the code repo, to the time it is deployed on staging. And I want to track this over time for days, weeks, months, etc. I want to know if the build pipeline is slowing down or speeding up and to know why.

          Jason Fowler added a comment - - edited I have two comments for "Pipeline as code": 1) It's an excellent idea. A DSL for the "Jenkinsfile" is very important. There is a need for "Build and Deploy as a Service" for organizations who are limited by compliance to use cloud-based systems. Simply dropping a Jenkinsfile into a source code repo should be enough to trigger a build in Jenkins. The DSL should be simple to use like it is in, for example, TravisCI. Groupon created "DotCI" and would be a great reference for some of the features required to make it a success. 2) Metrics. This is a somewhat related feature for "Pipeline as code" because the DSL in the Jenkinsfile should have the ability to configure build statistics and metrics. For example, let's say I have several jobs in Jenkins that perform builds, tests, and deployment to staging. This is my "Continuous Delivery Pipeline". I want to measure how long it takes from the initial commit to the code repo, to the time it is deployed on staging. And I want to track this over time for days, weeks, months, etc. I want to know if the build pipeline is slowing down or speeding up and to know why.

          It would be totally awesome if Jenkins could come with an embedded Eclipse Orion (or similar) Web IDE for workflow scripts!
          Just imagine the perfect auto complete based on installed plug-ins, configuration,... that such an IDE could provide.
          See https://wiki.eclipse.org/Orion.

          Ing. Christoph Obexer added a comment - It would be totally awesome if Jenkins could come with an embedded Eclipse Orion (or similar) Web IDE for workflow scripts! Just imagine the perfect auto complete based on installed plug-ins, configuration,... that such an IDE could provide. See https://wiki.eclipse.org/Orion .

          Jesse Glick added a comment -

          Everyone commenting here—ideas are appreciated, but this issue is merely a placeholder for a feature which has already existed for over a year and which is under active development by several people, so it is not a good place for discussions. This query shows (as of this writing) 239 open issues relating to specific aspects.

          Jesse Glick added a comment - Everyone commenting here—ideas are appreciated, but this issue is merely a placeholder for a feature which has already existed for over a year and which is under active development by several people, so it is not a good place for discussions. This query shows (as of this writing) 239 open issues relating to specific aspects.

          Code changed in jenkins
          User: Daniel Beck
          Path:
          war/src/main/js/api/plugins.js
          http://jenkins-ci.org/commit/jenkins/47bfa4f71337ef6cda42c9d6e6eba2e1850d24aa
          Log:
          JENKINS-31152 Add Pipeline plugins, select by default

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: war/src/main/js/api/plugins.js http://jenkins-ci.org/commit/jenkins/47bfa4f71337ef6cda42c9d6e6eba2e1850d24aa Log: JENKINS-31152 Add Pipeline plugins, select by default

          Code changed in jenkins
          User: Daniel Beck
          Path:
          war/src/main/js/api/plugins.js
          http://jenkins-ci.org/commit/jenkins/6cb6e9d5423997160129dd91c6fc9dfd2ded5f1f
          Log:
          Merge pull request #2053 from daniel-beck/JENKINS-31152

          JENKINS-31152 Add Pipeline plugins, select by default

          Compare: https://github.com/jenkinsci/jenkins/compare/823f9bae7140...6cb6e9d54239

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: war/src/main/js/api/plugins.js http://jenkins-ci.org/commit/jenkins/6cb6e9d5423997160129dd91c6fc9dfd2ded5f1f Log: Merge pull request #2053 from daniel-beck/ JENKINS-31152 JENKINS-31152 Add Pipeline plugins, select by default Compare: https://github.com/jenkinsci/jenkins/compare/823f9bae7140...6cb6e9d54239

          2.0 has shipped, so closing this epic.

          Kohsuke Kawaguchi added a comment - 2.0 has shipped, so closing this epic.

            marcelfrei_ublox Marcel Frei
            kohsuke Kohsuke Kawaguchi
            Votes:
            25 Vote for this issue
            Watchers:
            24 Start watching this issue

              Created:
              Updated:
              Resolved: