Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-38046

Gerrit Trigger Plugin Should be a Source for Multibranch Pipeline

      I want to use the new Multibranch Pipeline features of Jenkins with Gerrit source control.

      I should be able to choose Gerrit as the source for a multibranch pipeline. I should be able to select one or more repositories from Gerrit that each have their own Jenkinsfile. The pipeline would run the Jenkinsfile from the patch set that was pushed to Gerrit.

      There is no known workaround for this issue because one core requirement is to see each gerrit branch and change request as a different sub-job with its own history. This is essential because it allow you to know the build status of the branch instead of seeing a long queue of builds made on various branches and CRs, most likely full of failures.

      The behaviour described in this ticket is almost identical with the GitHub Multibranch implementation, the difference is that instead of having GitHub branches and PRs as the data source, we want to have Gerrit.

          [JENKINS-38046] Gerrit Trigger Plugin Should be a Source for Multibranch Pipeline

          Mike Kasberg added a comment - - edited

          Here is an (ugly) workaround:

          Use a normal pipeline job with the gerrit trigger. Have the pipeline execute a simple bootstrapping script, which loads the correct Jenkinsfile. The script might look like this:

          #!/usr/bin/env groovy
          
          node('label') {
              stage 'Checkout' {
                  // Checkout the patch set we're testing.
                  // See https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger#GerritTrigger-PipelineJobs
                  // Using Gerrit Trigger with Pipeline jobs.
          
                  git url: 'ssh://foo@gerrit/repository'
                      def changeBranch = "change-${GERRIT_CHANGE_NUMBER}-${GERRIT_PATCHSET_NUMBER}"
                      sh "git fetch origin ${GERRIT_REFSPEC}:${changeBranch}"
                      sh "git checkout ${changeBranch}"
                      sh "git clean -d -f -q -x"
              }
          
              // The rest comes from the repository itself!
              load 'Jenkinsfile'
          }
          

          Mike Kasberg added a comment - - edited Here is an (ugly) workaround: Use a normal pipeline job with the gerrit trigger. Have the pipeline execute a simple bootstrapping script, which loads the correct Jenkinsfile. The script might look like this: #!/usr/bin/env groovy node( 'label' ) { stage 'Checkout' { // Checkout the patch set we're testing. // See https://wiki.jenkins-ci.org/display/JENKINS/Gerrit+Trigger#GerritTrigger-PipelineJobs // Using Gerrit Trigger with Pipeline jobs. git url: 'ssh: //foo@gerrit/repository' def changeBranch = "change-${GERRIT_CHANGE_NUMBER}-${GERRIT_PATCHSET_NUMBER}" sh "git fetch origin ${GERRIT_REFSPEC}:${changeBranch}" sh "git checkout ${changeBranch}" sh "git clean -d -f -q -x" } // The rest comes from the repository itself! load 'Jenkinsfile' }

          Austin Phillips added a comment - - edited

          Here's another workaround using a normal pipeline job with the gerrit trigger plugin.

          • Set up Gerrit triggers as per normal
          • Set up Pipeline with 'Pipeline script from SCM', using the Gerrit environment variables as source information for the Jenkinsfile
          • Use a Jenkinsfile in your source as normal, complete with 'checkout scm' step.

          Configuration of the Pipeline SCM is as follows:

          • Set 'Repository URL' to $GERRIT_SCHEME://$GERRIT_HOST:$GERRIT_PORT/$GERRIT_PROJECT
          • Click the 'Advanced' button and set 'Refspec' to $GERRIT_REFSPEC:$GERRIT_REFSPEC
          • Set 'Branches to build' to $GERRIT_REFSPEC
          • If desired, add 'Advanced clone behaviours' with 'Shallow clone' enabled with a 'Shallow clone depth' set to 1.
          • Untick 'Lightweight checkout'

          Then, in your Gerrit repository Jenkins file:

           

           node {
              stage('Checkout') {
                  checkout scm
             }
             stage('Build') {
                 echo "Building...'
             }
          }

           

          Additional information:

          • The advantage of this method is that there is only a single Jenkinsfile specification which lives in your SCM.
          • The Pipeline configuration can be consistent across multiple projects as all of the SCM information is derived from Gerrit Trigger environment variables.  The Gerrit Trigger parameters become the source for the Jenkinsfile pipeline definition.
          • The Jenkins master will check out a copy of the repository on the master to get access to the Jenkinsfile (ie Not in a workspace)
          • The 'checkout scm' step inside the pipeline will check out the code according to the SCM information entered in the pipeline step.  Because the checkout occurs inside a 'node' block, the source repo ends up being checked out twice.  Once to obtain the Jenkinsfile by the master, once as part of the Jenkinsfile pipeline step.  The shallow clone options may help speed things up here.
          • This has only been tested with the 'Patchset Created' Gerrit Trigger type.
          • The 'Lightweight checkout' options appears not to work due to the way that git is used for this option. Following error is generated: 
            hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/$GERRIT_REFSPEC:refs/remotes/origin/$GERRIT_REFSPEC --prune" returned status code 128:

           

          Austin Phillips added a comment - - edited Here's another workaround using a normal pipeline job with the gerrit trigger plugin. Set up Gerrit triggers as per normal Set up Pipeline with 'Pipeline script from SCM', using the Gerrit environment variables as source information for the Jenkinsfile Use a Jenkinsfile in your source as normal, complete with 'checkout scm' step. Configuration of the Pipeline SCM is as follows: Set 'Repository URL' to $GERRIT_SCHEME://$GERRIT_HOST:$GERRIT_PORT/$GERRIT_PROJECT Click the 'Advanced' button and set 'Refspec' to $GERRIT_REFSPEC:$GERRIT_REFSPEC Set 'Branches to build' to $GERRIT_REFSPEC If desired, add 'Advanced clone behaviours' with 'Shallow clone' enabled with a 'Shallow clone depth' set to 1. Untick 'Lightweight checkout' Then, in your Gerrit repository Jenkins file:    node {     stage('Checkout') {         checkout scm    } stage('Build') { echo "Building...'    } }   Additional information: The advantage of this method is that there is only a single Jenkinsfile specification which lives in your SCM. The Pipeline configuration can be consistent across multiple projects as all of the SCM information is derived from Gerrit Trigger environment variables.  The Gerrit Trigger parameters become the source for the Jenkinsfile pipeline definition. The Jenkins master will check out a copy of the repository on the master to get access to the Jenkinsfile (ie Not in a workspace) The 'checkout scm' step inside the pipeline will check out the code according to the SCM information entered in the pipeline step.  Because the checkout occurs inside a 'node' block, the source repo ends up being checked out twice.  Once to obtain the Jenkinsfile by the master, once as part of the Jenkinsfile pipeline step.  The shallow clone options may help speed things up here. This has only been tested with the 'Patchset Created' Gerrit Trigger type. The 'Lightweight checkout' options appears not to work due to the way that git is used for this option. Following error is generated:  hudson.plugins.git.GitException: Command "git fetch --tags --progress origin +refs/heads/$GERRIT_REFSPEC:refs/remotes/origin/$GERRIT_REFSPEC --prune" returned status code 128:  

          Sorin Sbarnea added a comment -

          Does anyone have a Jenkins instance where we can see this working? If that's not public, can we see some screenshots?

          I am really interested about this because currently we do build all gerrit patches for multiple branches using a single job and this makes impossible to get a reliable build status because builds from different branches are one after another so the build history is polluted.

          Ideally I would like to see something similar to the github multi-branch implementation where we can see the status of each branch and also the status of each pull-request.

          Can we achieve something similar? ... or at least something that would allow us to see status of each branch individually.

          Sorin Sbarnea added a comment - Does anyone have a Jenkins instance where we can see this working? If that's not public, can we see some screenshots? I am really interested about this because currently we do build all gerrit patches for multiple branches using a single job and this makes impossible to get a reliable build status because builds from different branches are one after another so the build history is polluted. Ideally I would like to see something similar to the github multi-branch implementation where we can see the status of each branch and also the status of each pull-request. Can we achieve something similar? ... or at least something that would allow us to see status of each branch individually.

          Mike Kasberg added a comment -

          The workarounds that have been posted will still use a single Jenkins Pipeline job on multiple branches, which will result in a polluted build history. I don't think what you're asking for will be possible until this feature is implemented correctly, unless someone else has figured out a better workaround that I don't know about.

          I do agree with you about the pains of history pollution - it would be great to see some progress on this feature.

          Mike Kasberg added a comment - The workarounds that have been posted will still use a single Jenkins Pipeline job on multiple branches, which will result in a polluted build history. I don't think what you're asking for will be possible until this feature is implemented correctly, unless someone else has figured out a better workaround that I don't know about. I do agree with you about the pains of history pollution - it would be great to see some progress on this feature.

          Sorin Sbarnea added a comment - - edited

          Does anyone has an idea about which plugin(s) do need to be updated to make this happen? Having this issue addressed would be a life changer for OpenStack suite of projects which are all using Gerrit.

          If we really need a bounty to make this happen I am willing to contribute with my own money or to try to pursue my employer (RedHat) to contribute to it. Usually I would have spend time implementing a patch myself if it would not be too complex (my main coding language is Python and Java experience is sort-of limited).

          I am willing to spend some of my personal time on this but I doubt I could make it happen by myself. Anyone willing to step in/team up?

          One last note: in our current workflow we do not store the Pipeline definitions on the git repos themselves. We are using JJB to build the pipelines and to upload them to Jenkins. That's because we manage lots of projects with similar configurations that do change often and we don't want to push CI related changes to hundreds of branches. I know that this could be a problem if we would (probably never) like to use GitHub multi-branch because the plugin is looking for Jenkingfile(s) in order to create/destroy branches.

          Sorin Sbarnea added a comment - - edited Does anyone has an idea about which plugin(s) do need to be updated to make this happen? Having this issue addressed would be a life changer for OpenStack suite of projects which are all using Gerrit. If we really need a bounty to make this happen I am willing to contribute with my own money or to try to pursue my employer (RedHat) to contribute to it. Usually I would have spend time implementing a patch myself if it would not be too complex (my main coding language is Python and Java experience is sort-of limited). I am willing to spend some of my personal time on this but I doubt I could make it happen by myself. Anyone willing to step in/team up? One last note: in our current workflow we do not store the Pipeline definitions on the git repos themselves. We are using JJB to build the pipelines and to upload them to Jenkins. That's because we manage lots of projects with similar configurations that do change often and we don't want to push CI related changes to hundreds of branches. I know that this could be a problem if we would (probably never) like to use GitHub multi-branch because the plugin is looking for Jenkingfile(s) in order to create/destroy branches.

          Sorin Sbarnea added a comment - - edited

          I anyone else interesting in making this happen? Currently we can configure the multibranch-job to get branches from gerrit by using the git as the source branch but this is limited to normal branches, not change-requests.
          Doing this for change requests is impossible via git only because while Gerrit exposes all of them, it does in a way that makes them unusable for CI

          • the is no single branch per review, easy change being visible as another branch.
          • the abandoned or merged reviews are not removed from list of branches, so they only grow
          • most Gerrit servers to have rate limiters for "git ls-remote".

          So, what we need is to use Gerrit API in order to retrieve the current list or reviews that are supposed to be build and to map them to a single review branch. Initially I was considering to make each review visible as a temporary branch but I am starting to believe that this approach does not scale well, cluttering the user interface. My impression is that it could be better if we create a single review(pr) branch for each target branch, or even just a single one.

          Can we at least get some architecture guidance from someone that knows involved Jenkins plugins and API better? It would very useful to get a bill-of-materials that would look like: fork or clone this plugin and modify these classes, ...

          I digged a little bit last night and observed that branch detection is happening inside git-client-plugin, but I also know that there are two plugins that, based on their names, are supposed to provide exactly the same kind of feature but for other services: github-branch-source-plugin, bitbucket-branch-source-plugin and even an unmaintained proof of concept gitlab-branch-source-plugin. Both of them are quite complex in nature and I don't know if we should clone on of them and start to rewrite it to make it work with Gerrit. Or the solution is to add this functionality to the gerrit-trigger-plugin?

          Either way, I could use some guidance: what is the minimum change needed and where it should be implemented?

          Sorin Sbarnea added a comment - - edited I anyone else interesting in making this happen? Currently we can configure the multibranch-job to get branches from gerrit by using the git as the source branch but this is limited to normal branches, not change-requests. Doing this for change requests is impossible via git only because while Gerrit exposes all of them, it does in a way that makes them unusable for CI the is no single branch per review, easy change being visible as another branch. the abandoned or merged reviews are not removed from list of branches, so they only grow most Gerrit servers to have rate limiters for "git ls-remote". So, what we need is to use Gerrit API in order to retrieve the current list or reviews that are supposed to be build and to map them to a single review branch. Initially I was considering to make each review visible as a temporary branch but I am starting to believe that this approach does not scale well, cluttering the user interface. My impression is that it could be better if we create a single review(pr) branch for each target branch, or even just a single one. Can we at least get some architecture guidance from someone that knows involved Jenkins plugins and API better? It would very useful to get a bill-of-materials that would look like: fork or clone this plugin and modify these classes, ... I digged a little bit last night and observed that branch detection is happening inside git-client-plugin , but I also know that there are two plugins that, based on their names, are supposed to provide exactly the same kind of feature but for other services: github-branch-source-plugin , bitbucket-branch-source-plugin and even an unmaintained proof of concept gitlab-branch-source-plugin . Both of them are quite complex in nature and I don't know if we should clone on of them and start to rewrite it to make it work with Gerrit. Or the solution is to add this functionality to the gerrit-trigger-plugin? Either way, I could use some guidance: what is the minimum change needed and where it should be implemented?

          ssbarnea this would be very welcome. I think you are on the right track regarding the branch-source-plugin approach.

          I don't know how complex it has to be, though. It depends on what approach would be the best. Should reviews be built in their respective target branch? Or should they be built in a separate review branch? I.e, the "master" branch/job builds both from HEAD and from changes/12346/123 (or whatever) or should there be a origin/master and a changes/master (for the changes/****/**).

          Depending on which approach, I think the complexity will vary.

          Erik Hakansson added a comment - ssbarnea this would be very welcome. I think you are on the right track regarding the branch-source-plugin approach. I don't know how complex it has to be, though. It depends on what approach would be the best. Should reviews be built in their respective target branch? Or should they be built in a separate review branch? I.e, the "master" branch/job builds both from HEAD and from changes/12346/123 (or whatever) or should there be a origin/master and a changes/master (for the changes/**** / **). Depending on which approach, I think the complexity will vary.

          Sorin Sbarnea added a comment -

          Maybe someone from CloudBees could give us some hints here, like rsandell or kktest11.

          I will only describe the desired user experience UX:

          • incoming changes (PRs/CRs) should never pollute the branch build history.
          • if incoming changes are build as virtual-branches, we need to avoid seeing them in the normal list of branches (to avoid confusions due to too long lists)
          • keeping build history for an incoming change is not very important as we do also see it in Gerrit votes/comments, so I would find acceptable to combine all changes into one of few virtual branches.
          • be aware that having virtual branches for each open CR does not work well for any project, there are lots of them with more than 50-100 open and the list would be growing. If we go for one virtual branch per change request, we will spam Jenkins UI and even increase the load due to the way history is kept.

          I still hope I will get someone more senior giving a start on this, it would be much easier to fix issue on a proof of concept that got the right architecture decision from start.

          Sorin Sbarnea added a comment - Maybe someone from CloudBees could give us some hints here, like rsandell or kktest11 . I will only describe the desired user experience UX: incoming changes (PRs/CRs) should never pollute the branch build history. if incoming changes are build as virtual-branches, we need to avoid seeing them in the normal list of branches (to avoid confusions due to too long lists) keeping build history for an incoming change is not very important as we do also see it in Gerrit votes/comments, so I would find acceptable to combine all changes into one of few virtual branches. be aware that having virtual branches for each open CR does not work well for any project, there are lots of them with more than 50-100 open and the list would be growing. If we go for one virtual branch per change request, we will spam Jenkins UI and even increase the load due to the way history is kept. I still hope I will get someone more senior giving a start on this, it would be much easier to fix issue on a proof of concept that got the right architecture decision from start.

          I suppose my StackOverflow.com question can't be answered due to this not being implemented, correct?

          Is there anything we or I can do to help this process along?

          Michael Gardner added a comment - I suppose my StackOverflow.com question  can't be answered due to this not being implemented, correct? Is there anything we or I can do to help this process along?

          In reply to austin_phillips, I attempted your workaround, but after each triggered build the Build Triggers section of the build is reset. The "Gerrit event" checkbox becomes unticked and essentially makes the effort moot. Thoughts?

          Michael Gardner added a comment - In reply to austin_phillips , I attempted your workaround, but after each triggered build the Build Triggers section of the build is reset. The "Gerrit event" checkbox becomes unticked and essentially makes the effort moot. Thoughts?

          austin_phillips I appreciate the photos. The problem I'm running into is that when the build runs through the Gerrit Trigger section gets unchecked. It never fires again.

          Michael Gardner added a comment - austin_phillips I appreciate the photos. The problem I'm running into is that when the build runs through the Gerrit Trigger section gets unchecked. It never fires again.

          mikejr83 It would be strange for the Jenkins job configuration to change as a result of a build, so not sure what is happening there.

          The following images show a Pipeline job configuration which is triggered by Gerrit.

          As described by other comments, this configuration results in a build history which is shared (and therefore polluted) between changeset builds and builds from the target branch.

          As a workaround you can manually create two normal Pipeline jobs for each branch to be tracked, one job for changeset builds, another for builds off the target branch.  The pipeline configuration can be the same for both jobs, but the Gerrit trigger conditions change.  For changesets, trigger on 'Patchset Created' and set the Branches pattern to the branch to track.  For builds based on the target branch containing submitted changesets, trigger on 'Ref Updated' or 'Change Merged'.

          Austin Phillips added a comment - mikejr83 It would be strange for the Jenkins job configuration to change as a result of a build, so not sure what is happening there. The following images show a Pipeline job configuration which is triggered by Gerrit. As described by other comments, this configuration results in a build history which is shared (and therefore polluted) between changeset builds and builds from the target branch. As a workaround you can manually create two normal Pipeline jobs for each branch to be tracked, one job for changeset builds, another for builds off the target branch.  The pipeline configuration can be the same for both jobs, but the Gerrit trigger conditions change.  For changesets, trigger on 'Patchset Created' and set the Branches pattern to the branch to track.  For builds based on the target branch containing submitted changesets, trigger on 'Ref Updated' or 'Change Merged'.

          I have announced on the Jenkins Developers + Users mailing list that a new plugin is under development.

          See https://groups.google.com/forum/#!topic/jenkinsci-dev/E3pvb0WH4Ls

          It will behave very similarly to the GitHub integration, making Gerrit Changes / Patchsets buildable and discoverable automatically. Additionally, a Jenkinsfile-specific configuration could be defined to allow complete isolation across branches.

          Example Jekinsfile pipeline script:

          node {
              checkout scm
          
              gerrit.withServer("http://gerrit:8080/", "gerrit") {
          
                  try {
                      docker.image('gerritforge/play-sbt-8-jdk-alpine').inside {
                          stage('Build') {
                              sh 'sbt compile'
                          }
                          stage('Test') {
                              sh 'sbt 'test'
                          }
                      }
          
                      gerrit.review("Verified", 1, "It works !")
                  } catch (e) {
                      gerrit.review("Verified", -1, "Breaks the build ;-(")
                      throw e
                  }
              }
          }

          Luca Domenico Milanesio added a comment - I have announced on the Jenkins Developers + Users mailing list that a new plugin is under development. See https://groups.google.com/forum/#!topic/jenkinsci-dev/E3pvb0WH4Ls It will behave very similarly to the GitHub integration, making Gerrit Changes / Patchsets buildable and discoverable automatically. Additionally, a Jenkinsfile-specific configuration could be defined to allow complete isolation across branches. Example Jekinsfile pipeline script: node {     checkout scm     gerrit.withServer( "http: //gerrit:8080/" , "gerrit" ) {         try {             docker.image( 'gerritforge/play-sbt-8-jdk-alpine' ).inside {                 stage( 'Build' ) {                     sh 'sbt compile'                 }                 stage( 'Test' ) {                     sh 'sbt ' test'                 }             }             gerrit.review( "Verified" , 1, "It works !" )         } catch (e) {             gerrit.review( "Verified" , -1, "Breaks the build ;-(" )             throw e         }     } }

          Mike Kasberg added a comment -

          The new plugin sounds wonderful. Thank you for your work - I'm excited to try it out!

          Mike Kasberg added a comment - The new plugin sounds wonderful. Thank you for your work - I'm excited to try it out!

          Sorin Sbarnea added a comment -

          lucamilanesio Where is the codebase of this new plugin? I am more than interested and willing to test it and to contribute back to it.

          The fact that Gerrit branches and CRs are not working similarly to Git multi-branch, is a huge PITA.

          Sorin Sbarnea added a comment - lucamilanesio Where is the codebase of this new plugin? I am more than interested and willing to test it and to contribute back to it. The fact that Gerrit branches and CRs are not working similarly to Git multi-branch, is a huge PITA.

          Eric Isakson added a comment -

          Any word on the new plugin? I am also interested in testing/contributing to help get it off the ground.

          Eric Isakson added a comment - Any word on the new plugin? I am also interested in testing/contributing to help get it off the ground.

          I would also be interested and willing to test/contribute to the project.

          Philippe McLean added a comment - I would also be interested and willing to test/contribute to the project.

          Jesse Glick added a comment -

          Jesse Glick added a comment - https://github.com/GerritForge/gerrit-plugin/  apparently.

          jglick the new plugin has now been renamed to ([Gerrit Code Review plugin](https://github.com/jenkinsci/gerrit-code-review-plugin) and is now available for installation through the Jenkins Plugin Manager.

          Luca Domenico Milanesio added a comment - jglick the new plugin has now been renamed to ( [Gerrit Code Review plugin] ( https://github.com/jenkinsci/gerrit-code-review-plugin)  and is now available for installation through the Jenkins Plugin Manager.

          Taylor Patton added a comment -

          Will this capability be added to gerrit-trigger-plugin? I am weary of installing another gerrit plugin and having 2 ways of configuring pipelines (multi-branch and regular).

          Taylor Patton added a comment - Will this capability be added to gerrit-trigger-plugin? I am weary of installing another gerrit plugin and having 2 ways of configuring pipelines (multi-branch and regular).

          Jose Sa added a comment -

          I'm using this trigger in multi-branch but seems to be ignored:

            triggers {
              gerrit(
                serverName: 'server-name',
                gerritProjects: [[
                  compareType: 'PLAIN',
                  pattern: '_REPO_',
                  branches: [[ compareType: 'ANT', pattern: 'refs/heads/*' ]],
                  filePaths: [[ compareType: 'ANT', pattern: "_MODULE_/**" ]],
                  disableStrictForbiddenFileVerification: false
                ]],
                triggerOnEvents: [
                  refUpdated()
                ]
              )
            }
          

          The only thing that makes build happen is the polling scheduled every 5 minutes (as workaround) checking for branch changes.

          Should I use something differently to get multibranch jobs triggered ?

          Jose Sa added a comment - I'm using this trigger in multi-branch but seems to be ignored: triggers { gerrit( serverName: 'server-name' , gerritProjects: [[ compareType: 'PLAIN' , pattern: '_REPO_' , branches: [[ compareType: 'ANT' , pattern: 'refs/heads/*' ]], filePaths: [[ compareType: 'ANT' , pattern: "_MODULE_/**" ]], disableStrictForbiddenFileVerification: false ]], triggerOnEvents: [ refUpdated() ] ) } The only thing that makes build happen is the polling scheduled every 5 minutes (as workaround) checking for branch changes. Should I use something differently to get multibranch jobs triggered ?

          Mike added a comment -

          austin_phillips
          thanks for your solution! It's not multi-branch but it is working well for me.

          Mike added a comment - austin_phillips thanks for your solution! It's not multi-branch but it is working well for me.

          Lars Berntzon added a comment -

          Any news on this? We need it for multibranch stage0 builds.

          Lars Berntzon added a comment - Any news on this? We need it for multibranch stage0 builds.

            lucamilanesio Luca Domenico Milanesio
            mkasberg Mike Kasberg
            Votes:
            48 Vote for this issue
            Watchers:
            57 Start watching this issue

              Created:
              Updated: