Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46431

Multibranch pipelines should always show changes

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • Jenkins 2.60.3
      Pipeline Plugin 2.5
      Pipeline: Multibranch 2.16
      Git Plugin 3.5.1

      Given a Jenkinsfile of the form:

      node {
          sh "ls"
      }
      

      If I configure a multibranch pipeline pointing to a git repo with this Jenkinsfile, Jenkins does not show changes. I have to explicitly add a checkout step:

      node {
          checkout scm
          sh "ls"
      }
      

      If Jenkins cannot find changes, it cannot notify the committers that a build has failed. For example, CulpritsRecipientProvider in the email-ext ultimately calls WorkflowRun::getChangeSets. This returns an empty list, so Jenkins doesn't know who to email.

      Adding 'checkout scm' seems redundant, because Jenkins has found the Jenkinsfile from git already. In principle, Jenkins has the data it needs to work out the changes, as far as I can see.

          [JENKINS-46431] Multibranch pipelines should always show changes

          OK, so one of the classes in workflow-multibranch-plugin needs to inherit from SCMFileSystem? Is that correct?

          Wilfred Hughes added a comment - OK, so one of the classes in workflow-multibranch-plugin needs to inherit from SCMFileSystem? Is that correct?

          Nope, https://github.com/jenkinsci/workflow-multibranch-plugin/blob/c49261f827d032a637475071ba6742f0c40a8653/src/main/java/org/jenkinsci/plugins/workflow/multibranch/SCMBinder.java#L110 needs to call https://github.com/jenkinsci/scm-api-plugin/blob/5d7bc8376a565ef77667630047e2f73044952cee/src/main/java/jenkins/scm/api/SCMFileSystem.java#L155 with the output stream pointing to the changelog file...

          or/and https://github.com/jenkinsci/workflow-cps-plugin/blob/616e3ada5197a05972d585daab83d7c6c7ea819d/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java#L100 probably needs to do the same

          This is probably just a question of replicating this logic: https://github.com/jenkinsci/workflow-scm-step-plugin/blob/846b982d2c549038790ebb7d3ebef8d08efb6de0/src/main/java/org/jenkinsci/plugins/workflow/steps/scm/SCMStep.java#L91-L99 to figure out what file to write to and then just streaming the output.

          It seems noddy to me but jglick doesn't seem to want to try and use that API call even though I spend a good deal of effort coding up a reference impl for him: https://github.com/jenkinsci/git-plugin/blob/f85fccbac95eed2a4a6eff74912d3bcecf6bffec/src/main/java/jenkins/plugins/git/GitSCMFileSystem.java#L201-L243 and I am waiting for confirmation that the expected consumer works correctly before adding the implementations to GitHub Branch Source and Bitbucket Branch Source (though given my current work priorities, that would have to be a spare time effort... and also given Jesse's current work reprioritization it is likely the same for him)

           

          Perhaps svanoort or abayer could be convinced that this is a quick win

          Stephen Connolly added a comment - Nope, https://github.com/jenkinsci/workflow-multibranch-plugin/blob/c49261f827d032a637475071ba6742f0c40a8653/src/main/java/org/jenkinsci/plugins/workflow/multibranch/SCMBinder.java#L110  needs to call https://github.com/jenkinsci/scm-api-plugin/blob/5d7bc8376a565ef77667630047e2f73044952cee/src/main/java/jenkins/scm/api/SCMFileSystem.java#L155  with the output stream pointing to the changelog file... or/and https://github.com/jenkinsci/workflow-cps-plugin/blob/616e3ada5197a05972d585daab83d7c6c7ea819d/src/main/java/org/jenkinsci/plugins/workflow/cps/CpsScmFlowDefinition.java#L100  probably needs to do the same This is probably just a question of replicating this logic: https://github.com/jenkinsci/workflow-scm-step-plugin/blob/846b982d2c549038790ebb7d3ebef8d08efb6de0/src/main/java/org/jenkinsci/plugins/workflow/steps/scm/SCMStep.java#L91-L99  to figure out what file to write to and then just streaming the output. It seems noddy to me but jglick doesn't seem to want to try and use that API call even though I spend a good deal of effort coding up a reference impl for him: https://github.com/jenkinsci/git-plugin/blob/f85fccbac95eed2a4a6eff74912d3bcecf6bffec/src/main/java/jenkins/plugins/git/GitSCMFileSystem.java#L201-L243  and I am waiting for confirmation that the expected consumer works correctly before adding the implementations to GitHub Branch Source and Bitbucket Branch Source (though given my current work priorities, that would have to be a spare time effort... and also given Jesse's current work reprioritization it is likely the same for him)   Perhaps svanoort or abayer could be convinced that this is a quick win

          Jesse Glick added a comment -

          Mentioned in JENKINS-38659 though not an exact duplicate.

          Jesse Glick added a comment - Mentioned in  JENKINS-38659 though not an exact duplicate.

          Jesse Glick added a comment -

          Though the originally reported use case makes no sense to me, which is why I have always treated this as a very low priority. If you have no checkout scm, you are not using the rest of the repository at all—only Jenkinsfile. So what exactly is this Pipeline doing?

          Jesse Glick added a comment - Though the originally reported use case makes no sense to me, which is why I have always treated this as a very low priority. If you have no checkout scm , you are not using the rest of the repository at all—only Jenkinsfile . So what exactly is this Pipeline doing ?

          It's sometimes useful to just have a Jenkinsfile when you're using Jenkins to run cron jobs.

          However, I encountered this when trying to write a minimal Jenkinsfile to test configuring a new plugin. The behaviour of requiring 'checkout scm' initially confused me, because I thought it related to my plugin configuration.

          Wilfred Hughes added a comment - It's sometimes useful to just have a Jenkinsfile when you're using Jenkins to run cron jobs. However, I encountered this when trying to write a minimal Jenkinsfile to test configuring a new plugin. The behaviour of requiring 'checkout scm' initially confused me, because I thought it related to my plugin configuration.

          Wilfred Hughes added a comment - - edited

          I encountered this today when testing the StashNotifier plugin with this minimal Jenkinsfile:

          node {
              currentBuild.result = 'SUCCESS'
              step([$class: 'StashNotifier'])
          }
          

          This is tricky because when you have a pipeline library installed, you still get notifications to the current commit of the library, but not the current project. I solved this with:

          node {
              stage("Checkout") {
                  checkout scm
              }
              currentBuild.result = 'SUCCESS'
              step([$class: 'StashNotifier'])
          }
          

          Wilfred Hughes added a comment - - edited I encountered this today when testing the StashNotifier plugin with this minimal Jenkinsfile: node { currentBuild.result = 'SUCCESS' step([$class: 'StashNotifier' ]) } This is tricky because when you have a pipeline library installed, you still get notifications to the current commit of the library, but not the current project. I solved this with: node { stage( "Checkout" ) { checkout scm } currentBuild.result = 'SUCCESS' step([$class: 'StashNotifier' ]) }

          This has an interesting interaction with the stash notification plugin. Since we want to notify Stash/Bitbucket server as soon as the build has started, it's tempting to put the notification at the beginning:

          node {
              step([$class: 'StashNotifier'])
          
              stage("Checkout") {
                  checkout scm
              }
          
              try {
                  stage("Run tests") {
                      sh "bash tests.sh"
                  }
                  
                  // Set the current result, so we can send stash notifications.
                  currentBuild.result = 'SUCCESS'
              } catch (e) {
                  currentBuild.result = 'FAILURE'
                  throw e
              } finally {
                  // Notify that build completed.
                  step([$class: 'StashNotifier'])
              }
          }
          

          However, this doesn't work, because the plugin doesn't know the commit of the project under test. The correct version looks like this:

          node {
              stage("Checkout") {
                  checkout scm
              }
          
              step([$class: 'StashNotifier'])
          
              try {
                  stage("Run tests") {
                      sh "bash tests.sh"
                  }
                  
                  // Set the current result, so we can send stash notifications.
                  currentBuild.result = 'SUCCESS'
              } catch (e) {
                  currentBuild.result = 'FAILURE'
                  throw e
              } finally {
                  // Notify that build completed.
                  step([$class: 'StashNotifier'])
              }
          }
          

          Wilfred Hughes added a comment - This has an interesting interaction with the stash notification plugin. Since we want to notify Stash/Bitbucket server as soon as the build has started, it's tempting to put the notification at the beginning: node { step([$class: 'StashNotifier' ]) stage( "Checkout" ) { checkout scm } try { stage( "Run tests" ) { sh "bash tests.sh" } // Set the current result, so we can send stash notifications. currentBuild.result = 'SUCCESS' } catch (e) { currentBuild.result = 'FAILURE' throw e } finally { // Notify that build completed. step([$class: 'StashNotifier' ]) } } However, this doesn't work, because the plugin doesn't know the commit of the project under test. The correct version looks like this: node { stage( "Checkout" ) { checkout scm } step([$class: 'StashNotifier' ]) try { stage( "Run tests" ) { sh "bash tests.sh" } // Set the current result, so we can send stash notifications. currentBuild.result = 'SUCCESS' } catch (e) { currentBuild.result = 'FAILURE' throw e } finally { // Notify that build completed. step([$class: 'StashNotifier' ]) } }

          Psion Omikron added a comment - - edited

          I believe we're experiencing this problem. We have a multibranch project that is not reporting any changes or git hashes except for the shared library.

          How we got here: We ran out of disk space because of the git repository replicating on each branch, so we switched to a custom checkout approach where we use worktrees, as suggested in the comments on JENKINS-32246. Because of this we're not using checkout scm.

          Psion Omikron added a comment - - edited I believe we're experiencing this problem. We have a multibranch project that is not reporting any changes or git hashes except for the shared library. How we got here: We ran out of disk space because of the git repository replicating on each branch, so we switched to a custom checkout approach where we use worktrees, as suggested in the comments on JENKINS-32246 . Because of this we're not using checkout scm.

          We are seeing this issue in our PRs on the creation event any of the PRs has changes, we tried the following pipelines and none works, we used a Shared pipeline library configured globaly (I dunno if it matters), we use Jenkins 2.153,
          Pipeline 2.2, Declarative pipeline 1.33, Pipeline: Multibranch 2.20,
          Pipeline: SCM Step 2.7, SCM API Plugin 2.3.0, Github API 1.95, Basic Branch Build Strategies Plugin 1.1.1, Branch API Plugin 2.1.1,
          GitHub Branch Source Plugin 2.4.1

          regular checkout does not compute changes

          pipeline {
            agent any
            options {
              timeout(time: 1, unit: 'HOURS')
              buildDiscarder(logRotator(numToKeepStr: '20', artifactNumToKeepStr: '20', daysToKeepStr: '30'))
              timestamps()
              ansiColor('xterm')
              disableResume()
              durabilityHint('PERFORMANCE_OPTIMIZED')
            }
            stages {
              stage('Dummy'){
                steps {
                  sh 'export'
                  checkout scm
                  error "Please do not continue"
                }
              }
            }
          }
          

          using a reference repo checkout does not compute changes

          pipeline {
            agent any
            options {
              timeout(time: 1, unit: 'HOURS')
              buildDiscarder(logRotator(numToKeepStr: '20', artifactNumToKeepStr: '20', daysToKeepStr: '30'))
              timestamps()
              ansiColor('xterm')
              disableResume()
              durabilityHint('PERFORMANCE_OPTIMIZED')
            }
            stages {
              stage('Dummy'){
                steps {
                  checkout([$class: 'GitSCM', 
                  branches: [[name: "${env?.CHANGE_ID ? env?.GIT_COMMIT : env?.BRANCH_NAME}"]], 
                  doGenerateSubmoduleConfigurations: false, 
                  extensions: [[$class: 'CloneOption', 
                    noTags: false, 
                    reference: '/var/lib/jenkins/.git-references/apm-agent-go.git', 
                    shallow: false]], 
                  submoduleCfg: [], 
                  userRemoteConfigs: [[credentialsId: 'UserAndToken', 
                  url: "${env?.GIT_URL}"]]])
                  error "Please do not continue"
                }
              }
            }
          }
          

          using ChangelogToBranch checkout does not compute changes

          pipeline {
            agent any
            options {
              timeout(time: 1, unit: 'HOURS')
              buildDiscarder(logRotator(numToKeepStr: '20', artifactNumToKeepStr: '20', daysToKeepStr: '30'))
              timestamps()
              ansiColor('xterm')
              disableResume()
              durabilityHint('PERFORMANCE_OPTIMIZED')
            }
            stages {
              stage('Dummy'){
                steps {
                  checkout([$class: 'GitSCM', 
                    branches: [[name: "${env?.CHANGE_ID ? env?.GIT_COMMIT : env?.BRANCH_NAME}"]],
                    doGenerateSubmoduleConfigurations: false, 
                    extensions: [
                      [$class: 'ChangelogToBranch', 
                        options: [compareRemote: "${env?.GIT_URL}", 
                        compareTarget: "${env?.CHANGE_ID ? env?.CHANGE_TARGET : 'master'}"]]], 
                    submoduleCfg: [], 
                    userRemoteConfigs: [
                      [credentialsId: 'UserAndToken', 
                      url: "${env?.GIT_URL}"]]])
                  error "Please do not continue"
                }
              }
            }
          }
          

          Ivan Fernandez Calvo added a comment - We are seeing this issue in our PRs on the creation event any of the PRs has changes, we tried the following pipelines and none works, we used a Shared pipeline library configured globaly (I dunno if it matters), we use Jenkins 2.153, Pipeline 2.2, Declarative pipeline 1.33, Pipeline: Multibranch 2.20, Pipeline: SCM Step 2.7, SCM API Plugin 2.3.0, Github API 1.95, Basic Branch Build Strategies Plugin 1.1.1, Branch API Plugin 2.1.1, GitHub Branch Source Plugin 2.4.1 regular checkout does not compute changes pipeline { agent any options { timeout(time: 1, unit: 'HOURS' ) buildDiscarder(logRotator(numToKeepStr: '20' , artifactNumToKeepStr: '20' , daysToKeepStr: '30' )) timestamps() ansiColor( 'xterm' ) disableResume() durabilityHint( 'PERFORMANCE_OPTIMIZED' ) } stages { stage( 'Dummy' ){ steps { sh 'export' checkout scm error "Please do not continue " } } } } using a reference repo checkout does not compute changes pipeline { agent any options { timeout(time: 1, unit: 'HOURS' ) buildDiscarder(logRotator(numToKeepStr: '20' , artifactNumToKeepStr: '20' , daysToKeepStr: '30' )) timestamps() ansiColor( 'xterm' ) disableResume() durabilityHint( 'PERFORMANCE_OPTIMIZED' ) } stages { stage( 'Dummy' ){ steps { checkout([$class: 'GitSCM' , branches: [[name: "${env?.CHANGE_ID ? env?.GIT_COMMIT : env?.BRANCH_NAME}" ]], doGenerateSubmoduleConfigurations: false , extensions: [[$class: 'CloneOption' , noTags: false , reference: '/ var /lib/jenkins/.git-references/apm-agent-go.git' , shallow: false ]], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'UserAndToken' , url: "${env?.GIT_URL}" ]]]) error "Please do not continue " } } } } using ChangelogToBranch checkout does not compute changes pipeline { agent any options { timeout(time: 1, unit: 'HOURS' ) buildDiscarder(logRotator(numToKeepStr: '20' , artifactNumToKeepStr: '20' , daysToKeepStr: '30' )) timestamps() ansiColor( 'xterm' ) disableResume() durabilityHint( 'PERFORMANCE_OPTIMIZED' ) } stages { stage( 'Dummy' ){ steps { checkout([$class: 'GitSCM' , branches: [[name: "${env?.CHANGE_ID ? env?.GIT_COMMIT : env?.BRANCH_NAME}" ]], doGenerateSubmoduleConfigurations: false , extensions: [ [$class: 'ChangelogToBranch' , options: [compareRemote: "${env?.GIT_URL}" , compareTarget: "${env?.CHANGE_ID ? env?.CHANGE_TARGET : 'master' }" ]]], submoduleCfg: [], userRemoteConfigs: [ [credentialsId: 'UserAndToken' , url: "${env?.GIT_URL}" ]]]) error "Please do not continue " } } } }

          Ivan Fernandez Calvo added a comment - nice input about this one https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!msg/jenkinsci-users/Ew-7R2rToDQ/THz_c0z7BgAJ

            Unassigned Unassigned
            wilfredh Wilfred Hughes
            Votes:
            5 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated: