Problem definition

The default interaction model with Jenkins has been very web UI driven, requiring users to manually create jobs, then manually fill in the details through a web browser. This requires large amounts of effort to create and manage jobs to test and build multiple projects and keeps the actual configuration of a job to build/test/deploy a project separate from the actual code being built/tested/deployed. This prevents users from applying their existing CI/CD best practices to the job configurations themselves.

Proposal

Introduce a new subsystem in Jenkins that:

• lets you design a whole pipeline, not just a single linear set of tasks
• stores the said pipeline configuration as human-editable Jenkinsfile in your SCM
• makes it automatic to set up new pipelines when Jenkinsfile is added
• differentiates multiple branches in the same repository

This is the key new feature that positions Jenkins for continuous delivery use cases and other more complex automations of today.

Overview

(This overview uses GitHub as an example although the system is extensible to allow implementations for other SCMs and repository hostings.)

1. In this new subsystem, you first let Jenkins know where you host your source code repositories. You do this by creating a new special folder called "Organization Folder" and selecting "GitHub organization." This sets up a new smart folder.
2. In one of the repository in the chosen org, create a Jenkinsfile that defines your pipeline (or just a build process to start with.) This file is actually a workflow script. See workflow tutorial for details.
3. Jenkins will automatically recognize this branch and create appropriate jobs by itself.
4. Everytime a new change is pushed to this branch, the branch is built and the commit status gets updated.
5. When the branch is destroyed in the repository or if Jenkinsfile is removed, the corresponding job gets destroyed from Jenkins as well automatically.

In this use, there'll be a lot of Jenkinsfile in various branches & repositories. To keep them DRY, various mechanisms will be provided to promote reuse of workflow scripts, such as this.

For more details, see jglick's slides and video recording that talks about this feature (in particular, where the demo starts.) There's also docker-based demo that you can play with on your own.

Contents

• Thus, this feature should be made available out of the box by default, unless users opt out (2.0 Out-of-the-box experience)
• Provide better workflow visualization out of the box (JENKINS-31154)
• Workflow gets renamed to Pipeline, so that users understand what it is without explanation (JENKINS-31153)
• Shared library improvements to simplify Jenkinsfile (JENKINS-31155)

Internals

• The execution engine of this is Workflow plugin (see JENKINS-26129)
• Multi-branch workflow project type defines a new kind of folder that is associated with a single source code repository and automatically create/destory a job inside as branches are created/destroyed in the repository.
• Organization folder that defines a new kind of folder that is associated with a group of source code repositories (such as GitHub org) and automatically create/destory the above "multi-branch workflow project" as repositories are created/destroyed in the group.
• Branch API plugin and SCM API plugin defines a contract for version control systems (such as Git) and hosting (such as GitHub) to define branch discovery, repository discovery, etc.
• CloudBees GitHub Branch Source Plugin implements the above contract for GitHub, so that GitHub organization can be added as an organization folder to automatically scan every repo & branch in it. Other similar plugins can be developed for other hosting options and other source code management tools.

Impact

Open Questions