Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-31152

2.0: Pipeline as code in front & center

    XMLWordPrintable

    Details

    • Epic Name:
      2.0: Pipeline as Code
    • Similar Issues:

      Description

      Problem definition

      The default interaction model with Jenkins has been very web UI driven, requiring users to manually create jobs, then manually fill in the details through a web browser. This requires large amounts of effort to create and manage jobs to test and build multiple projects and keeps the actual configuration of a job to build/test/deploy a project separate from the actual code being built/tested/deployed. This prevents users from applying their existing CI/CD best practices to the job configurations themselves.

      Proposal

      Introduce a new subsystem in Jenkins that:

      • lets you design a whole pipeline, not just a single linear set of tasks
      • stores the said pipeline configuration as human-editable Jenkinsfile in your SCM
      • makes it automatic to set up new pipelines when Jenkinsfile is added
      • differentiates multiple branches in the same repository

      This is the key new feature that positions Jenkins for continuous delivery use cases and other more complex automations of today.

      Overview

      (This overview uses GitHub as an example although the system is extensible to allow implementations for other SCMs and repository hostings.)

      1. In this new subsystem, you first let Jenkins know where you host your source code repositories. You do this by creating a new special folder called "Organization Folder" and selecting "GitHub organization." This sets up a new smart folder.
      2. In one of the repository in the chosen org, create a Jenkinsfile that defines your pipeline (or just a build process to start with.) This file is actually a workflow script. See workflow tutorial for details.
      3. Jenkins will automatically recognize this branch and create appropriate jobs by itself.
      4. Everytime a new change is pushed to this branch, the branch is built and the commit status gets updated.
      5. When the branch is destroyed in the repository or if Jenkinsfile is removed, the corresponding job gets destroyed from Jenkins as well automatically.

      In this use, there'll be a lot of Jenkinsfile in various branches & repositories. To keep them DRY, various mechanisms will be provided to promote reuse of workflow scripts, such as this.

      For more details, see Jesse Glick's slides and video recording that talks about this feature (in particular, where the demo starts.) There's also docker-based demo that you can play with on your own.

      Contents

      • Thus, this feature should be made available out of the box by default, unless users opt out (2.0 Out-of-the-box experience)
      • Provide better workflow visualization out of the box (JENKINS-31154)
      • Workflow gets renamed to Pipeline, so that users understand what it is without explanation (JENKINS-31153)
      • Shared library improvements to simplify Jenkinsfile (JENKINS-31155)

      Internals

      • The execution engine of this is Workflow plugin (see JENKINS-26129)
      • Multi-branch workflow project type defines a new kind of folder that is associated with a single source code repository and automatically create/destory a job inside as branches are created/destroyed in the repository.
      • Organization folder that defines a new kind of folder that is associated with a group of source code repositories (such as GitHub org) and automatically create/destory the above "multi-branch workflow project" as repositories are created/destroyed in the group.
      • Branch API plugin and SCM API plugin defines a contract for version control systems (such as Git) and hosting (such as GitHub) to define branch discovery, repository discovery, etc.
      • CloudBees GitHub Branch Source Plugin implements the above contract for GitHub, so that GitHub organization can be added as an organization folder to automatically scan every repo & branch in it. Other similar plugins can be developed for other hosting options and other source code management tools.

      Impact

      Open Questions

        Attachments

          Issue Links

            Activity

            kohsuke Kohsuke Kawaguchi created issue -
            kohsuke Kohsuke Kawaguchi made changes -
            Field Original Value New Value
            Description See [wiki page|http://TODO/] for details. *See [wiki page|https://wiki.jenkins-ci.org/display/JENKINS/2.0+Pipeline+as+Code] for the overview of this epic.
            kohsuke Kohsuke Kawaguchi made changes -
            Description *See [wiki page|https://wiki.jenkins-ci.org/display/JENKINS/2.0+Pipeline+as+Code] for the overview of this epic. *See [wiki page|https://wiki.jenkins-ci.org/display/JENKINS/2.0+Pipeline+as+Code] for the overview of this epic.*
            kohsuke Kohsuke Kawaguchi made changes -
            Epic Child JENKINS-31153 [ 165809 ]
            kohsuke Kohsuke Kawaguchi made changes -
            Description *See [wiki page|https://wiki.jenkins-ci.org/display/JENKINS/2.0+Pipeline+as+Code] for the overview of this epic.* *See [wiki page|https://wiki.jenkins-ci.org/display/JENKINS/2.0+Pipeline+as+Code] for the overview of this epic.*

            This epic is used to track 2.0 planning tickets that are related to pipeline as code in 2.0, so that the next level of details in this plan can be cross-referenced easily. There's no need to link bugs and other tactical changes.
            kohsuke Kohsuke Kawaguchi made changes -
            Epic Child JENKINS-31154 [ 165810 ]
            kohsuke Kohsuke Kawaguchi made changes -
            Epic Child JENKINS-31155 [ 165811 ]
            kohsuke Kohsuke Kawaguchi made changes -
            Epic Name Pipeline as Code 2.0: Pipeline as Code
            gembrotkan Gembrot Bokong made changes -
            Epic Child TEST-56 [ 165828 ]
            waterheaterjakarta waterheaterjakarta made changes -
            Epic Child MEETING-18 [ 165917 ]
            rtyler R. Tyler Croy made changes -
            Description *See [wiki page|https://wiki.jenkins-ci.org/display/JENKINS/2.0+Pipeline+as+Code] for the overview of this epic.*

            This epic is used to track 2.0 planning tickets that are related to pipeline as code in 2.0, so that the next level of details in this plan can be cross-referenced easily. There's no need to link bugs and other tactical changes.
            h1. Problem definition

            The default interaction model with Jenkins has been very web UI driven, requiring users to manually create jobs, then manually fill in the details through a web browser. This requires large amounts of effort to create and manage jobs to test and build multiple projects and keeps the actual configuration of a job to build/test/deploy a project separate from the actual code being built/tested/deployed. This prevents users from applying their existing CI/CD best practices to the job configurations themselves.

            h1. Proposal

            Introduce a new subsystem in Jenkins that:

            * lets you design a whole pipeline, not just a single linear set of tasks
            * stores the said pipeline configuration as human-editable Jenkinsfile in your SCM
            * makes it automatic to set up new pipelines when Jenkinsfile is added
            * differentiates multiple branches in the same repository

            This is the key new feature that positions Jenkins for continuous delivery use cases and other more complex automations of today.

            h2. Overview

            (This overview uses GitHub as an example although the system is extensible to allow implementations for other SCMs and repository hostings.)

            # In this new subsystem, you first let Jenkins know where you host your source code repositories. You do this by creating a new special folder called "Organization Folder" and selecting "GitHub organization." This sets up a new smart folder.
            # In one of the repository in the chosen org, create a {{Jenkinsfile}} that defines your pipeline (or just a build process to start with.) This file is actually a workflow script. See [workflow tutorial|https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md] for details.
            # Jenkins will automatically recognize this branch and create appropriate jobs by itself.
            # Everytime a new change is pushed to this branch, the branch is built and the commit status gets updated.
            # When the branch is destroyed in the repository or if {{Jenkinsfile}} is removed, the corresponding job gets destroyed from Jenkins as well automatically.

            In this use, there'll be a lot of {{Jenkinsfile}} in various branches & repositories. To keep them [DRY|https://en.wikipedia.org/wiki/Don%27t_repeat_yourself], various mechanisms will be provided to promote reuse of workflow scripts, such as [this|https://github.com/jenkinsci/workflow-plugin/blob/master/cps-global-lib/README.md].

            For more details, see [~jglick]'s [slides|https://www.cloudbees.com/sites/default/files/webinar-_continuous_delivery_as_code_with_jenkins_workflow.pdf] and [video recording|https://youtu.be/Q2pZdzaaCXg] that talks about this feature (in particular, where [the demo starts|http://www.youtube.com/watch?v=Q2pZdzaaCXg&t=20m30s].) There's also [docker-based demo|https://hub.docker.com/r/cloudbees/github-organization-demo/] that you can play with on your own.


            h2. Contents

            * Thus, this feature should be made available out of the box by default, unless users opt out ([2.0 Out-of-the-box experience])
            * Provide better workflow visualization out of the box (JENKINS-31154)
            * Workflow gets renamed to Pipeline, so that users understand what it is without explanation (JENKINS-31153)
            * Shared library improvements to simplify Jenkinsfile (JENKINS-31155)

            h2. Internals

            * The execution engine of this is [Workflow plugin] (see JENKINS-26129)
            * [Multi-branch workflow project type|https://github.com/jenkinsci/workflow-plugin/tree/master/multibranch] defines a new kind of [folder|CloudBees Folders Plugin] that is associated with a single source code repository and automatically create/destory a job inside as branches are created/destroied in the repository.
            * [Organization folder|https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/OrganizationFolder.java] that defines a new kind of [folder|CloudBees Folders Plugin] that is associated with a group of source code repositories (such as GitHub org) and automatically create/destory the above "multi-branch workflow project" as repositories are created/destroyed in the group.
            * [Branch API plugin] and [SCM API plugin] defines a contract for version control systems (such as Git) and hosting (such as GitHub) to define branch discovery, repository discovery, etc.
            * [CloudBees GitHub Branch Source Plugin] implements the above contract for GitHub, so that GitHub organization can be added as an organization folder to automatically scan every repo & branch in it. Other similar plugins can be developed for other hosting options and other source code management tools.


            h1. Impact


            h1. Open Question
            rtyler R. Tyler Croy made changes -
            Description h1. Problem definition

            The default interaction model with Jenkins has been very web UI driven, requiring users to manually create jobs, then manually fill in the details through a web browser. This requires large amounts of effort to create and manage jobs to test and build multiple projects and keeps the actual configuration of a job to build/test/deploy a project separate from the actual code being built/tested/deployed. This prevents users from applying their existing CI/CD best practices to the job configurations themselves.

            h1. Proposal

            Introduce a new subsystem in Jenkins that:

            * lets you design a whole pipeline, not just a single linear set of tasks
            * stores the said pipeline configuration as human-editable Jenkinsfile in your SCM
            * makes it automatic to set up new pipelines when Jenkinsfile is added
            * differentiates multiple branches in the same repository

            This is the key new feature that positions Jenkins for continuous delivery use cases and other more complex automations of today.

            h2. Overview

            (This overview uses GitHub as an example although the system is extensible to allow implementations for other SCMs and repository hostings.)

            # In this new subsystem, you first let Jenkins know where you host your source code repositories. You do this by creating a new special folder called "Organization Folder" and selecting "GitHub organization." This sets up a new smart folder.
            # In one of the repository in the chosen org, create a {{Jenkinsfile}} that defines your pipeline (or just a build process to start with.) This file is actually a workflow script. See [workflow tutorial|https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md] for details.
            # Jenkins will automatically recognize this branch and create appropriate jobs by itself.
            # Everytime a new change is pushed to this branch, the branch is built and the commit status gets updated.
            # When the branch is destroyed in the repository or if {{Jenkinsfile}} is removed, the corresponding job gets destroyed from Jenkins as well automatically.

            In this use, there'll be a lot of {{Jenkinsfile}} in various branches & repositories. To keep them [DRY|https://en.wikipedia.org/wiki/Don%27t_repeat_yourself], various mechanisms will be provided to promote reuse of workflow scripts, such as [this|https://github.com/jenkinsci/workflow-plugin/blob/master/cps-global-lib/README.md].

            For more details, see [~jglick]'s [slides|https://www.cloudbees.com/sites/default/files/webinar-_continuous_delivery_as_code_with_jenkins_workflow.pdf] and [video recording|https://youtu.be/Q2pZdzaaCXg] that talks about this feature (in particular, where [the demo starts|http://www.youtube.com/watch?v=Q2pZdzaaCXg&t=20m30s].) There's also [docker-based demo|https://hub.docker.com/r/cloudbees/github-organization-demo/] that you can play with on your own.


            h2. Contents

            * Thus, this feature should be made available out of the box by default, unless users opt out ([2.0 Out-of-the-box experience])
            * Provide better workflow visualization out of the box (JENKINS-31154)
            * Workflow gets renamed to Pipeline, so that users understand what it is without explanation (JENKINS-31153)
            * Shared library improvements to simplify Jenkinsfile (JENKINS-31155)

            h2. Internals

            * The execution engine of this is [Workflow plugin] (see JENKINS-26129)
            * [Multi-branch workflow project type|https://github.com/jenkinsci/workflow-plugin/tree/master/multibranch] defines a new kind of [folder|CloudBees Folders Plugin] that is associated with a single source code repository and automatically create/destory a job inside as branches are created/destroied in the repository.
            * [Organization folder|https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/OrganizationFolder.java] that defines a new kind of [folder|CloudBees Folders Plugin] that is associated with a group of source code repositories (such as GitHub org) and automatically create/destory the above "multi-branch workflow project" as repositories are created/destroyed in the group.
            * [Branch API plugin] and [SCM API plugin] defines a contract for version control systems (such as Git) and hosting (such as GitHub) to define branch discovery, repository discovery, etc.
            * [CloudBees GitHub Branch Source Plugin] implements the above contract for GitHub, so that GitHub organization can be added as an organization folder to automatically scan every repo & branch in it. Other similar plugins can be developed for other hosting options and other source code management tools.


            h1. Impact


            h1. Open Question
            h1. Problem definition

            The default interaction model with Jenkins has been very web UI driven, requiring users to manually create jobs, then manually fill in the details through a web browser. This requires large amounts of effort to create and manage jobs to test and build multiple projects and keeps the actual configuration of a job to build/test/deploy a project separate from the actual code being built/tested/deployed. This prevents users from applying their existing CI/CD best practices to the job configurations themselves.

            h1. Proposal

            Introduce a new subsystem in Jenkins that:

            * lets you design a whole pipeline, not just a single linear set of tasks
            * stores the said pipeline configuration as human-editable Jenkinsfile in your SCM
            * makes it automatic to set up new pipelines when Jenkinsfile is added
            * differentiates multiple branches in the same repository

            This is the key new feature that positions Jenkins for continuous delivery use cases and other more complex automations of today.

            h2. Overview

            (This overview uses GitHub as an example although the system is extensible to allow implementations for other SCMs and repository hostings.)

            # In this new subsystem, you first let Jenkins know where you host your source code repositories. You do this by creating a new special folder called "Organization Folder" and selecting "GitHub organization." This sets up a new smart folder.
            # In one of the repository in the chosen org, create a {{Jenkinsfile}} that defines your pipeline (or just a build process to start with.) This file is actually a workflow script. See [workflow tutorial|https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md] for details.
            # Jenkins will automatically recognize this branch and create appropriate jobs by itself.
            # Everytime a new change is pushed to this branch, the branch is built and the commit status gets updated.
            # When the branch is destroyed in the repository or if {{Jenkinsfile}} is removed, the corresponding job gets destroyed from Jenkins as well automatically.

            In this use, there'll be a lot of {{Jenkinsfile}} in various branches & repositories. To keep them [DRY|https://en.wikipedia.org/wiki/Don%27t_repeat_yourself], various mechanisms will be provided to promote reuse of workflow scripts, such as [this|https://github.com/jenkinsci/workflow-plugin/blob/master/cps-global-lib/README.md].

            For more details, see [~jglick]'s [slides|https://www.cloudbees.com/sites/default/files/webinar-_continuous_delivery_as_code_with_jenkins_workflow.pdf] and [video recording|https://youtu.be/Q2pZdzaaCXg] that talks about this feature (in particular, where [the demo starts|http://www.youtube.com/watch?v=Q2pZdzaaCXg&t=20m30s].) There's also [docker-based demo|https://hub.docker.com/r/cloudbees/github-organization-demo/] that you can play with on your own.


            h2. Contents

            * Thus, this feature should be made available out of the box by default, unless users opt out ([2.0 Out-of-the-box experience|https://wiki.jenkins-ci.org/display/JENKINS/2.0+Out-of-the-box+experience])
            * Provide better workflow visualization out of the box (JENKINS-31154)
            * Workflow gets renamed to Pipeline, so that users understand what it is without explanation (JENKINS-31153)
            * Shared library improvements to simplify Jenkinsfile (JENKINS-31155)

            h2. Internals

            * The execution engine of this is [Workflow plugin|https://wiki.jenkins-ci.org/display/JENKINS/Workflow+Plugin] (see JENKINS-26129)
            * [Multi-branch workflow project type|https://github.com/jenkinsci/workflow-plugin/tree/master/multibranch] defines a new kind of [folder|CloudBees Folders Plugin] that is associated with a single source code repository and automatically create/destory a job inside as branches are created/destroied in the repository.
            * [Organization folder|https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/OrganizationFolder.java] that defines a new kind of [folder|CloudBees Folders Plugin] that is associated with a group of source code repositories (such as GitHub org) and automatically create/destory the above "multi-branch workflow project" as repositories are created/destroyed in the group.
            * [Branch API plugin|https://wiki.jenkins-ci.org/display/JENKINS/Branch+API+Plugin] and [SCM API plugin|https://wiki.jenkins-ci.org/display/JENKINS/SCM+API+Plugin] defines a contract for version control systems (such as Git) and hosting (such as GitHub) to define branch discovery, repository discovery, etc.
            * [CloudBees GitHub Branch Source Plugin|https://wiki.jenkins-ci.org/display/JENKINS/CloudBees+GitHub+Branch+Source+Plugin] implements the above contract for GitHub, so that GitHub organization can be added as an organization folder to automatically scan every repo & branch in it. Other similar plugins can be developed for other hosting options and other source code management tools.


            h1. Impact


            h1. Open Question
            rtyler R. Tyler Croy made changes -
            Description h1. Problem definition

            The default interaction model with Jenkins has been very web UI driven, requiring users to manually create jobs, then manually fill in the details through a web browser. This requires large amounts of effort to create and manage jobs to test and build multiple projects and keeps the actual configuration of a job to build/test/deploy a project separate from the actual code being built/tested/deployed. This prevents users from applying their existing CI/CD best practices to the job configurations themselves.

            h1. Proposal

            Introduce a new subsystem in Jenkins that:

            * lets you design a whole pipeline, not just a single linear set of tasks
            * stores the said pipeline configuration as human-editable Jenkinsfile in your SCM
            * makes it automatic to set up new pipelines when Jenkinsfile is added
            * differentiates multiple branches in the same repository

            This is the key new feature that positions Jenkins for continuous delivery use cases and other more complex automations of today.

            h2. Overview

            (This overview uses GitHub as an example although the system is extensible to allow implementations for other SCMs and repository hostings.)

            # In this new subsystem, you first let Jenkins know where you host your source code repositories. You do this by creating a new special folder called "Organization Folder" and selecting "GitHub organization." This sets up a new smart folder.
            # In one of the repository in the chosen org, create a {{Jenkinsfile}} that defines your pipeline (or just a build process to start with.) This file is actually a workflow script. See [workflow tutorial|https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md] for details.
            # Jenkins will automatically recognize this branch and create appropriate jobs by itself.
            # Everytime a new change is pushed to this branch, the branch is built and the commit status gets updated.
            # When the branch is destroyed in the repository or if {{Jenkinsfile}} is removed, the corresponding job gets destroyed from Jenkins as well automatically.

            In this use, there'll be a lot of {{Jenkinsfile}} in various branches & repositories. To keep them [DRY|https://en.wikipedia.org/wiki/Don%27t_repeat_yourself], various mechanisms will be provided to promote reuse of workflow scripts, such as [this|https://github.com/jenkinsci/workflow-plugin/blob/master/cps-global-lib/README.md].

            For more details, see [~jglick]'s [slides|https://www.cloudbees.com/sites/default/files/webinar-_continuous_delivery_as_code_with_jenkins_workflow.pdf] and [video recording|https://youtu.be/Q2pZdzaaCXg] that talks about this feature (in particular, where [the demo starts|http://www.youtube.com/watch?v=Q2pZdzaaCXg&t=20m30s].) There's also [docker-based demo|https://hub.docker.com/r/cloudbees/github-organization-demo/] that you can play with on your own.


            h2. Contents

            * Thus, this feature should be made available out of the box by default, unless users opt out ([2.0 Out-of-the-box experience|https://wiki.jenkins-ci.org/display/JENKINS/2.0+Out-of-the-box+experience])
            * Provide better workflow visualization out of the box (JENKINS-31154)
            * Workflow gets renamed to Pipeline, so that users understand what it is without explanation (JENKINS-31153)
            * Shared library improvements to simplify Jenkinsfile (JENKINS-31155)

            h2. Internals

            * The execution engine of this is [Workflow plugin|https://wiki.jenkins-ci.org/display/JENKINS/Workflow+Plugin] (see JENKINS-26129)
            * [Multi-branch workflow project type|https://github.com/jenkinsci/workflow-plugin/tree/master/multibranch] defines a new kind of [folder|CloudBees Folders Plugin] that is associated with a single source code repository and automatically create/destory a job inside as branches are created/destroied in the repository.
            * [Organization folder|https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/OrganizationFolder.java] that defines a new kind of [folder|CloudBees Folders Plugin] that is associated with a group of source code repositories (such as GitHub org) and automatically create/destory the above "multi-branch workflow project" as repositories are created/destroyed in the group.
            * [Branch API plugin|https://wiki.jenkins-ci.org/display/JENKINS/Branch+API+Plugin] and [SCM API plugin|https://wiki.jenkins-ci.org/display/JENKINS/SCM+API+Plugin] defines a contract for version control systems (such as Git) and hosting (such as GitHub) to define branch discovery, repository discovery, etc.
            * [CloudBees GitHub Branch Source Plugin|https://wiki.jenkins-ci.org/display/JENKINS/CloudBees+GitHub+Branch+Source+Plugin] implements the above contract for GitHub, so that GitHub organization can be added as an organization folder to automatically scan every repo & branch in it. Other similar plugins can be developed for other hosting options and other source code management tools.


            h1. Impact


            h1. Open Question
            h1. Problem definition

            The default interaction model with Jenkins has been very web UI driven, requiring users to manually create jobs, then manually fill in the details through a web browser. This requires large amounts of effort to create and manage jobs to test and build multiple projects and keeps the actual configuration of a job to build/test/deploy a project separate from the actual code being built/tested/deployed. This prevents users from applying their existing CI/CD best practices to the job configurations themselves.

            h1. Proposal

            Introduce a new subsystem in Jenkins that:

            * lets you design a whole pipeline, not just a single linear set of tasks
            * stores the said pipeline configuration as human-editable Jenkinsfile in your SCM
            * makes it automatic to set up new pipelines when Jenkinsfile is added
            * differentiates multiple branches in the same repository

            This is the key new feature that positions Jenkins for continuous delivery use cases and other more complex automations of today.

            h2. Overview

            (This overview uses GitHub as an example although the system is extensible to allow implementations for other SCMs and repository hostings.)

            # In this new subsystem, you first let Jenkins know where you host your source code repositories. You do this by creating a new special folder called "Organization Folder" and selecting "GitHub organization." This sets up a new smart folder.
            # In one of the repository in the chosen org, create a {{Jenkinsfile}} that defines your pipeline (or just a build process to start with.) This file is actually a workflow script. See [workflow tutorial|https://github.com/jenkinsci/workflow-plugin/blob/master/TUTORIAL.md] for details.
            # Jenkins will automatically recognize this branch and create appropriate jobs by itself.
            # Everytime a new change is pushed to this branch, the branch is built and the commit status gets updated.
            # When the branch is destroyed in the repository or if {{Jenkinsfile}} is removed, the corresponding job gets destroyed from Jenkins as well automatically.

            In this use, there'll be a lot of {{Jenkinsfile}} in various branches & repositories. To keep them [DRY|https://en.wikipedia.org/wiki/Don%27t_repeat_yourself], various mechanisms will be provided to promote reuse of workflow scripts, such as [this|https://github.com/jenkinsci/workflow-plugin/blob/master/cps-global-lib/README.md].

            For more details, see [~jglick]'s [slides|https://www.cloudbees.com/sites/default/files/webinar-_continuous_delivery_as_code_with_jenkins_workflow.pdf] and [video recording|https://youtu.be/Q2pZdzaaCXg] that talks about this feature (in particular, where [the demo starts|http://www.youtube.com/watch?v=Q2pZdzaaCXg&t=20m30s].) There's also [docker-based demo|https://hub.docker.com/r/cloudbees/github-organization-demo/] that you can play with on your own.


            h2. Contents

            * Thus, this feature should be made available out of the box by default, unless users opt out ([2.0 Out-of-the-box experience|https://wiki.jenkins-ci.org/display/JENKINS/2.0+Out-of-the-box+experience])
            * Provide better workflow visualization out of the box (JENKINS-31154)
            * Workflow gets renamed to Pipeline, so that users understand what it is without explanation (JENKINS-31153)
            * Shared library improvements to simplify Jenkinsfile (JENKINS-31155)

            h2. Internals

            * The execution engine of this is [Workflow plugin|https://wiki.jenkins-ci.org/display/JENKINS/Workflow+Plugin] (see JENKINS-26129)
            * [Multi-branch workflow project type|https://github.com/jenkinsci/workflow-plugin/tree/master/multibranch] defines a new kind of [folder|https://wiki.jenkins-ci.org/display/JENKINS/CloudBees+Folders+Plugin] that is associated with a single source code repository and automatically create/destory a job inside as branches are created/destroyed in the repository.
            * [Organization folder|https://github.com/jenkinsci/branch-api-plugin/blob/master/src/main/java/jenkins/branch/OrganizationFolder.java] that defines a new kind of [folder|https://wiki.jenkins-ci.org/display/JENKINS/CloudBees+Folders+Plugin] that is associated with a group of source code repositories (such as GitHub org) and automatically create/destory the above "multi-branch workflow project" as repositories are created/destroyed in the group.
            * [Branch API plugin|https://wiki.jenkins-ci.org/display/JENKINS/Branch+API+Plugin] and [SCM API plugin|https://wiki.jenkins-ci.org/display/JENKINS/SCM+API+Plugin] defines a contract for version control systems (such as Git) and hosting (such as GitHub) to define branch discovery, repository discovery, etc.
            * [CloudBees GitHub Branch Source Plugin|https://wiki.jenkins-ci.org/display/JENKINS/CloudBees+GitHub+Branch+Source+Plugin] implements the above contract for GitHub, so that GitHub organization can be added as an organization folder to automatically scan every repo & branch in it. Other similar plugins can be developed for other hosting options and other source code management tools.


            h1. Impact


            h1. Open Questions
            Hide
            jmellor John Mellor added a comment -

            I have one project that has ~130 sub-builds, in a fairly complex ordering tree with some build threads in parallel and then some in serial, and then some in parallel again. Almost all steps are freestyle builds running autotools and make or a modified debuild. Are we going to be able to bring the job steps together into several choke-points throughout the pipeline, or is the proposed solution going to be more simplistic?

            Show
            jmellor John Mellor added a comment - I have one project that has ~130 sub-builds, in a fairly complex ordering tree with some build threads in parallel and then some in serial, and then some in parallel again. Almost all steps are freestyle builds running autotools and make or a modified debuild. Are we going to be able to bring the job steps together into several choke-points throughout the pipeline, or is the proposed solution going to be more simplistic?
            Hide
            jglick Jesse Glick added a comment -

            John Mellor the “proposed” solution already exists and is available for use, so I am not sure why the description was worded the way it was. This is not the place to ask for technical help.

            Show
            jglick Jesse Glick added a comment - John Mellor the “proposed” solution already exists and is available for use , so I am not sure why the description was worded the way it was. This is not the place to ask for technical help.
            solahartwater solahartwater made changes -
            Epic Child INFRA-442 [ 165957 ]
            solahartwater solahartwater made changes -
            Epic Child INFRA-443 [ 165958 ]
            solahartwater solahartwater made changes -
            Epic Child INFRA-447 [ 165962 ]
            solahartwater solahartwater made changes -
            Epic Child INFRA-451 [ 165966 ]
            solahartwater solahartwater made changes -
            Epic Child INFRA-452 [ 165967 ]
            solahartwater solahartwater made changes -
            Epic Child INFRA-453 [ 165968 ]
            Hide
            damien_coraboeuf Damien Coraboeuf added a comment -

            FYI, we manage more than 1500 jobs at my client today and we have designed the Seed plugin to go even further in the simplification of the job generation and management, by providing development teams a way to provide at SCM level a shopping list (a property file) which refers to a versioned pipeline library maintained by a core team.

            This pipeline library uses the Job DSL plugin but could also use the Workflow plugin. The idea behind the Seed plugin is to manage branches and projects on a very large scale, for people who do not know very well Jenkins.

            Feel free to check it at https://wiki.jenkins-ci.org/display/JENKINS/Seed+Plugin

            I'm very interested in the development of this new core integration of the pipeline as code, and see how I could in the future integrate it with this concept of shopping list.

            Damien.

            Show
            damien_coraboeuf Damien Coraboeuf added a comment - FYI, we manage more than 1500 jobs at my client today and we have designed the Seed plugin to go even further in the simplification of the job generation and management, by providing development teams a way to provide at SCM level a shopping list (a property file) which refers to a versioned pipeline library maintained by a core team. This pipeline library uses the Job DSL plugin but could also use the Workflow plugin. The idea behind the Seed plugin is to manage branches and projects on a very large scale, for people who do not know very well Jenkins. Feel free to check it at https://wiki.jenkins-ci.org/display/JENKINS/Seed+Plugin I'm very interested in the development of this new core integration of the pipeline as code, and see how I could in the future integrate it with this concept of shopping list. Damien.
            jakartaservice jakartaservice made changes -
            Epic Child WEBSITE-24 [ 166332 ]
            jakartaservice jakartaservice made changes -
            Epic Child WEBSITE-25 [ 166333 ]
            jakartaservice jakartaservice made changes -
            Epic Child WEBSITE-26 [ 166334 ]
            jakartaservice jakartaservice made changes -
            Epic Child WEBSITE-27 [ 166335 ]
            jakartaservice jakartaservice made changes -
            Epic Child WEBSITE-28 [ 166336 ]
            jakartaservice jakartaservice made changes -
            Epic Child WEBSITE-29 [ 166337 ]
            jakartaservice jakartaservice made changes -
            Epic Child WEBSITE-30 [ 166338 ]
            jakartaservice jakartaservice made changes -
            Epic Child WEBSITE-31 [ 166339 ]
            jakartaservice jakartaservice made changes -
            Epic Child WEBSITE-32 [ 166340 ]
            jakartaservice jakartaservice made changes -
            Epic Child WEBSITE-33 [ 166341 ]
            Hide
            preradio Jason Fowler added a comment - - edited

            I have two comments for "Pipeline as code":

            1) It's an excellent idea. A DSL for the "Jenkinsfile" is very important. There is a need for "Build and Deploy as a Service" for organizations who are limited by compliance to use cloud-based systems. Simply dropping a Jenkinsfile into a source code repo should be enough to trigger a build in Jenkins. The DSL should be simple to use like it is in, for example, TravisCI. Groupon created "DotCI" and would be a great reference for some of the features required to make it a success.

            2) Metrics. This is a somewhat related feature for "Pipeline as code" because the DSL in the Jenkinsfile should have the ability to configure build statistics and metrics. For example, let's say I have several jobs in Jenkins that perform builds, tests, and deployment to staging. This is my "Continuous Delivery Pipeline". I want to measure how long it takes from the initial commit to the code repo, to the time it is deployed on staging. And I want to track this over time for days, weeks, months, etc. I want to know if the build pipeline is slowing down or speeding up and to know why.

            Show
            preradio Jason Fowler added a comment - - edited I have two comments for "Pipeline as code": 1) It's an excellent idea. A DSL for the "Jenkinsfile" is very important. There is a need for "Build and Deploy as a Service" for organizations who are limited by compliance to use cloud-based systems. Simply dropping a Jenkinsfile into a source code repo should be enough to trigger a build in Jenkins. The DSL should be simple to use like it is in, for example, TravisCI. Groupon created "DotCI" and would be a great reference for some of the features required to make it a success. 2) Metrics. This is a somewhat related feature for "Pipeline as code" because the DSL in the Jenkinsfile should have the ability to configure build statistics and metrics. For example, let's say I have several jobs in Jenkins that perform builds, tests, and deployment to staging. This is my "Continuous Delivery Pipeline". I want to measure how long it takes from the initial commit to the code repo, to the time it is deployed on staging. And I want to track this over time for days, weeks, months, etc. I want to know if the build pipeline is slowing down or speeding up and to know why.
            Hide
            cobexer Ing. Christoph Obexer added a comment -

            It would be totally awesome if Jenkins could come with an embedded Eclipse Orion (or similar) Web IDE for workflow scripts!
            Just imagine the perfect auto complete based on installed plug-ins, configuration,... that such an IDE could provide.
            See https://wiki.eclipse.org/Orion.

            Show
            cobexer Ing. Christoph Obexer added a comment - It would be totally awesome if Jenkins could come with an embedded Eclipse Orion (or similar) Web IDE for workflow scripts! Just imagine the perfect auto complete based on installed plug-ins, configuration,... that such an IDE could provide. See https://wiki.eclipse.org/Orion .
            raphc Raphael CHAUMIER made changes -
            Comment [ +1 ]
            Hide
            jglick Jesse Glick added a comment -

            Everyone commenting here—ideas are appreciated, but this issue is merely a placeholder for a feature which has already existed for over a year and which is under active development by several people, so it is not a good place for discussions. This query shows (as of this writing) 239 open issues relating to specific aspects.

            Show
            jglick Jesse Glick added a comment - Everyone commenting here—ideas are appreciated, but this issue is merely a placeholder for a feature which has already existed for over a year and which is under active development by several people, so it is not a good place for discussions. This query shows (as of this writing) 239 open issues relating to specific aspects.
            yogiteches neel maity (Inactive) made changes -
            Epic Child INFRA-538 [ 167432 ]
            okram999 Niristotle Okram made changes -
            Epic Child JENKINS-32937 [ 168153 ]
            orrc Christopher Orr made changes -
            Epic Child JENKINS-32937 [ 168153 ]
            jglick Jesse Glick made changes -
            Epic Child JENKINS-33106 [ 168473 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Beck
            Path:
            war/src/main/js/api/plugins.js
            http://jenkins-ci.org/commit/jenkins/47bfa4f71337ef6cda42c9d6e6eba2e1850d24aa
            Log:
            JENKINS-31152 Add Pipeline plugins, select by default

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: war/src/main/js/api/plugins.js http://jenkins-ci.org/commit/jenkins/47bfa4f71337ef6cda42c9d6e6eba2e1850d24aa Log: JENKINS-31152 Add Pipeline plugins, select by default
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Daniel Beck
            Path:
            war/src/main/js/api/plugins.js
            http://jenkins-ci.org/commit/jenkins/6cb6e9d5423997160129dd91c6fc9dfd2ded5f1f
            Log:
            Merge pull request #2053 from daniel-beck/JENKINS-31152

            JENKINS-31152 Add Pipeline plugins, select by default

            Compare: https://github.com/jenkinsci/jenkins/compare/823f9bae7140...6cb6e9d54239

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Daniel Beck Path: war/src/main/js/api/plugins.js http://jenkins-ci.org/commit/jenkins/6cb6e9d5423997160129dd91c6fc9dfd2ded5f1f Log: Merge pull request #2053 from daniel-beck/ JENKINS-31152 JENKINS-31152 Add Pipeline plugins, select by default Compare: https://github.com/jenkinsci/jenkins/compare/823f9bae7140...6cb6e9d54239
            rtyler R. Tyler Croy made changes -
            Link This issue is blocking JENKINS-33281 [ JENKINS-33281 ]
            jglick Jesse Glick made changes -
            Assignee Jesse Glick [ jglick ] Daniel Beck [ danielbeck ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            michaelhuettermann Michael Hüttermann made changes -
            Epic Child JENKINS-33837 [ 169313 ]
            michaelhuettermann Michael Hüttermann made changes -
            Epic Child JENKINS-33840 [ 169316 ]
            michaelhuettermann Michael Hüttermann made changes -
            Epic Child JENKINS-33842 [ 169318 ]
            jglick Jesse Glick made changes -
            Epic Child JENKINS-33837 [ 169313 ]
            dpd_30 Daniel Daugherty made changes -
            Epic Child JENKINS-34150 [ 169676 ]
            drieselliott Dries Elliott made changes -
            Epic Child JENKINS-34163 [ 169690 ]
            jknurek J Knurek made changes -
            Epic Child JENKINS-34506 [ 170110 ]
            jknurek J Knurek made changes -
            Epic Child JENKINS-34507 [ 170111 ]
            m_k Michael Krause made changes -
            Epic Child JENKINS-34567 [ 170185 ]
            Hide
            kohsuke Kohsuke Kawaguchi added a comment -

            2.0 has shipped, so closing this epic.

            Show
            kohsuke Kohsuke Kawaguchi added a comment - 2.0 has shipped, so closing this epic.
            kohsuke Kohsuke Kawaguchi made changes -
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            jglick Jesse Glick made changes -
            Epic Child JENKINS-31155 [ 165811 ]
            jamesdelvin02 James Delvin made changes -
            Epic Child INFRA-817 [ 172188 ]
            rtyler R. Tyler Croy made changes -
            Workflow JNJira [ 166332 ] JNJira + In-Review [ 197979 ]
            abayer Andrew Bayer made changes -
            Component/s pipeline-general [ 21692 ]
            abayer Andrew Bayer made changes -
            Component/s workflow-plugin [ 18820 ]
            mkasberg Mike Kasberg made changes -
            Epic Child JENKINS-38046 [ 174194 ]
            peternhale Pete Hale made changes -
            Epic Child JENKINS-38454 [ 174634 ]
            abayer Andrew Bayer made changes -
            Epic Child JENKINS-38454 [ 174634 ]
            marcelfrei_ublox Marcel Frei made changes -
            Assignee Daniel Beck [ danielbeck ] Marcel Frei [ marcelfrei_ublox ]
            robert_ilg Robert Ilg made changes -
            Epic Child JENKINS-38742 [ 175142 ]
            asreenivas sree nivas made changes -
            Epic Child JENKINS-39900 [ 176519 ]
            deshari143 satish k made changes -
            Epic Child JENKINS-40889 [ 177611 ]
            atulsharma1989 Atul Sharma made changes -
            Epic Child EVENTS-43 [ 178147 ]
            saranya2293 Saranya U made changes -
            Epic Child JENKINS-43420 [ 180664 ]
            phani_kumar Phani Kumar made changes -
            Epic Child JENKINS-44695 [ 182559 ]
            phani_kumar Phani Kumar made changes -
            Epic Child JENKINS-44695 [ 182559 ]
            maneeshmp Maneesh M P made changes -
            Epic Child JENKINS-45418 [ 183586 ]
            jamesdumay James Dumay made changes -
            Epic Child JENKINS-43420 [ 180664 ]
            jamesdumay James Dumay made changes -
            Epic Status To Do [ 10000 ] Done [ 10002 ]

              People

              Assignee:
              marcelfrei_ublox Marcel Frei
              Reporter:
              kohsuke Kohsuke Kawaguchi
              Votes:
              25 Vote for this issue
              Watchers:
              24 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: