Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64719

checkout scm on the agent does not respect git 'Discover other refs' in multibranch pipeline

      I'm trying to use the git plugin to detect pull requests from my git repository which is located in Azure DevOps

      I'm using the following configuration in Multibranch Pipeline

      it seems that the initial git clone is actually works on the jenkins controller

      Started by user
      Replayed #11
       > git rev-parse --is-inside-work-tree # timeout=10
      Setting origin to https://dev.azure.com/<account>/<project>/_git/<repo>
       > git config remote.origin.url https://dev.azure.com/<account>/<project>/_git/<repo> # timeout=10
      Fetching origin...
      Fetching upstream changes from origin
       > git --version # timeout=10
       > git --version # 'git version 2.11.0'
       > git config --get remote.origin.url # timeout=10
      using GIT_ASKPASS to set credentials 
       > git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* +refs/pull/*/merge:refs/remotes/origin/pull/*/merge # timeout=10
       > git rev-parse refs/pull/74/merge^{commit} # timeout=10
      ERROR: Could not determine exact tip revision of pull/74/merge; falling back to nondeterministic checkout
      Running in Durability level: MAX_SURVIVABILITY
      [Pipeline] Start of Pipeline
      [Pipeline] library
      Loading library pipeline-library@master
      Attempting to resolve master from remote references...
       > git --version # timeout=10
       > git --version # 'git version 2.11.0'
      using GIT_ASKPASS to set credentials 
       > git ls-remote -h -- https://user@dev.azure.com/<account>/<project>/_git/<repo> # timeout=10
      Found match: refs/heads/master revision eeeacb692d0fc5e23c0c5ba66123fdbf95bbac0d
      The recommended git tool is: NONE
      using credential jenkins-token
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
       > git config remote.origin.url https://user@dev.azure.com/<account>/<project>/_git/<repo> # timeout=10
      Fetching without tags
      Fetching upstream changes from https://user@dev.azure.com/<account>/<project>/_git/<repo>
       > git --version # timeout=10
       > git --version # 'git version 2.11.0'
      using GIT_ASKPASS to set credentials 
       > git fetch --no-tags --progress -- https://user@dev.azure.com/<account>/<project>/_git/<repo> +refs/heads/*:refs/remotes/origin/* # timeout=10
      Checking out Revision eeeacb692d0fc5e23c0c5ba66123fdbf95bbac0d (master)
       > git config core.sparsecheckout # timeout=10
       > git checkout -f eeeacb692d0fc5e23c0c5ba66123fdbf95bbac0d # timeout=10
      Commit message: "align to pipeline"
       > git rev-list --no-walk eeeacb692d0fc5e23c0c5ba66123fdbf95bbac0d # timeout=10
      

      also in my local PC I can perform git checkout to this PR

      git checkout pull/74/merge
      # output:
      Branch 'pull/74/merge' set up to track remote branch 'pull/74/merge' from 'origin'.
      Switched to a new branch 'pull/74/merge'

       

      But When the jenkins is trying  to perform checkout scm on a different agent (node)

      stage('Generate') {
          node ('linux') {
              checkout scm
          }
      

      or 

      pipeline {
          agent { 
              label 'linux'
          }
          stages {
              stage('Initialize') { 
                steps {
                  checkout scm

      note: in declarative pipeline the jenkins will try automtically to perform checkout scm on the agent node and it will failed too.

      it's getting the following error:

       > git rev-parse --is-inside-work-tree # timeout=10
      Setting origin to https://dev.azure.com/<account>/<project>/_git/<repo>
       > git config remote.origin.url https://dev.azure.com/<account>/<project>/_git/<repo> # timeout=10
      Fetching origin...
      Fetching upstream changes from origin
       > git --version # timeout=10
       > git --version # 'git version 2.11.0'
       > git config --get remote.origin.url # timeout=10
      using GIT_ASKPASS to set credentials 
       > git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* +refs/pull/*/merge:refs/remotes/origin/pull/*/merge # timeout=10
       > git rev-parse refs/pull/74/merge^{commit} # timeout=10
      [Pipeline] }
      [Pipeline] // script
      [Pipeline] }
      [Pipeline] // stage
      [Pipeline] }
      [Pipeline] // timeout
      [Pipeline] }
      19:08:21  
      [Pipeline] // ansiColor
      [Pipeline] }
      [Pipeline] // timestamps
      [Pipeline] }
      [Pipeline] // withEnv
      [Pipeline] }
      [Pipeline] // node
      [Pipeline] End of Pipeline
      ERROR: Could not determine exact tip revision of pull/74/merge
      Finished: FAILURE
      

      why the step checkout SCM is not respecting the git Discover other refs?

      another way to workaround this? 

       

          [JENKINS-64719] checkout scm on the agent does not respect git 'Discover other refs' in multibranch pipeline

          dor s added a comment -

          dor s added a comment - related https://issues.jenkins.io/browse/JENKINS-59392  ? 

          Mark Waite added a comment - - edited

          I have duplicated the problem that you're describing in my docker-lfs repository. It appears to be the same scenario as JENKINS-59392.

          I don't plan to work on this issue any further. The multibranch pipeline works best with pull requests when using the more advanced branch source providers like GitHub, Bitbucket, GitLab, and Gitea. In each of those cases, the pull request (or merge request) is correctly discovered and is presented under a pull requests tab in the user interface.

          I was impressed that the "Hello World" pipeline was discovered and executed from the branch. Unfortunately, as soon as I included checkout scm, the pipeline failed.

          Mark Waite added a comment - - edited I have duplicated the problem that you're describing in my docker-lfs repository . It appears to be the same scenario as JENKINS-59392 . I don't plan to work on this issue any further. The multibranch pipeline works best with pull requests when using the more advanced branch source providers like GitHub, Bitbucket, GitLab, and Gitea. In each of those cases, the pull request (or merge request) is correctly discovered and is presented under a pull requests tab in the user interface. I was impressed that the "Hello World" pipeline was discovered and executed from the branch. Unfortunately, as soon as I included checkout scm , the pipeline failed.

          dor s added a comment - - edited

          Hi markewaite , Thank you for your fast response.

          The thing is that my source code is located under azure devops and I can't really move it to another provider. I couldn't find any other way to use azure devops PRs in a maintained plugin. (currently in order to install the latest tfs-plugin  you need to do it manually for god sake).

          Apparently azure devops  abandoned their tfs-plugin  (probably to push users to use their solutions), and because I'm using jenkins only I won't be able to get PRs which is very improtant for our CI jobs.

          it will be awesome to have a generic solution for source code providers that are not provinding a good integration with jenkins so we will not get a blocker on this scenario.

          I'm really frustrated from this issue and I really hope that you will reconsider to help the end user to get a solution/workaround for this scenario.

          more here: https://stackoverflow.com/questions/65157509/team-foundation-server-plugin-is-not-available-in-jenkins 

          dor s added a comment - - edited Hi markewaite  , Thank you for your fast response. The thing is that my source code is located under azure devops and I can't really move it to another provider. I couldn't find any other way to use azure devops PRs in a maintained plugin. (currently in order to install the latest tfs-plugin   you need to do it manually for god sake). Apparently azure devops  abandoned their  tfs-plugin  (probably to push users to use their solutions), and because I'm using jenkins only I won't be able to get PRs which is very improtant for our CI jobs. it will be awesome to have a generic solution for source code providers that are not provinding a good integration with jenkins so we will not get a blocker on this scenario. I'm really frustrated from this issue and I really hope that you will reconsider to help the end user to get a solution/workaround for this scenario. more here:  https://stackoverflow.com/questions/65157509/team-foundation-server-plugin-is-not-available-in-jenkins  

          hakamairi added a comment -

          any plans to work on this one?

          hakamairi added a comment - any plans to work on this one?

          Mark Waite added a comment -

          hakamairi I have no plans that I will work on this. If others would like to work on it, they are welcome to submit pull requests.

          Mark Waite added a comment - hakamairi I have no plans that I will work on this. If others would like to work on it, they are welcome to submit pull requests.

          hakamairi added a comment - - edited

          In declarative pipeline this checkout works, 52775 is just the pr number for a given PR.

          pipeline {
              agent any
              stages {
                  stage('Test') {
                      steps {
                          checkout scmGit(branches: [[name: '*/*52775*/*']],  extensions: [checkoutOption(), cloneOption(depth: 1, honorRefspec: true, noTags: true, reference: '', shallow: false)], userRemoteConfigs: [[credentialsId: 'creds', name: 'origin', refspec: '+refs/pull/*:refs/remotes/origin/pull-*', url: 'REPO_URL']])
                      }
                  }
              }
          }
           

          Outcome: 

          [Pipeline] {
          [Pipeline] stage
          [Pipeline] { (Test)
          [Pipeline] checkout
          The recommended git tool is: git
          using credential creds
          Fetching changes from the remote Git repository
           > git rev-parse --resolve-git-dir /var/jenkins/workspace/test/pr-build/.git # timeout=10
           > git config remote.origin.url REPO_URL # timeout=10
          Fetching upstream changes from REPO_URL
           > git --version # timeout=10
           > git --version # 'git version 2.31.1'
          [INFO] Currently running in a labeled security context
           > /usr/bin/chcon --type=ssh_home_t /var/jenkins/workspace/test/pr-build@tmp/jenkins-gitclient-ssh14081999047372580134.key
          Verifying host key using known hosts file, will automatically accept unseen keys
           > git fetch --no-tags --force --progress -- REPO_URL +refs/pull/*:refs/remotes/origin/pull-* # timeout=10
          Checking out Revision 1cc64c0e9b9dfc1ce85dfd89ee4d0795dae5e8e1 (origin/pull-52775/merge)
          Seen branch in repository origin/...
          ...
          Seen 158 remote branches
           > git show-ref --tags -d # timeout=10
           > git config core.sparsecheckout # timeout=10
           > git checkout -f 1cc64c0e9b9dfc1ce85dfd89ee4d0795dae5e8e1 # timeout=10
          Commit message: "Merge pull request 52775 from master into TEST_BRANCH"
          First time build. Skipping changelog.
          [Pipeline] }
          [Pipeline] // stage
          [Pipeline] }
          [Pipeline] // node
          [Pipeline] End of Pipeline
          Finished: SUCCESS 

          hakamairi added a comment - - edited In declarative pipeline this checkout works, 52775 is just the pr number for a given PR. pipeline {     agent any     stages {         stage( 'Test' ) {             steps {                 checkout scmGit(branches: [[name: '* /*52775*/ *' ]],  extensions: [checkoutOption(), cloneOption(depth: 1, honorRefspec: true , noTags: true , reference: '', shallow: false )], userRemoteConfigs: [[credentialsId: ' creds ', name: ' origin ', refspec: ' +refs/pull/*:refs/remotes/origin/pull-* ', url: ' REPO_URL']])             }         }     } } Outcome:  [Pipeline] { [Pipeline] stage [Pipeline] { (Test) [Pipeline] checkout The recommended git tool is: git using credential creds Fetching changes from the remote Git repository  > git rev-parse --resolve-git-dir / var /jenkins/workspace/test/pr-build/.git # timeout=10  > git config remote.origin.url REPO_URL # timeout=10 Fetching upstream changes from REPO_URL  > git --version # timeout=10  > git --version # 'git version 2.31.1' [INFO] Currently running in a labeled security context  > /usr/bin/chcon --type=ssh_home_t / var /jenkins/workspace/test/pr-build@tmp/jenkins-gitclient-ssh14081999047372580134.key Verifying host key using known hosts file, will automatically accept unseen keys  > git fetch --no-tags --force --progress -- REPO_URL +refs/pull/*:refs/remotes/origin/pull-* # timeout=10 Checking out Revision 1cc64c0e9b9dfc1ce85dfd89ee4d0795dae5e8e1 (origin/pull-52775/merge) Seen branch in repository origin/... ... Seen 158 remote branches  > git show-ref --tags -d # timeout=10  > git config core.sparsecheckout # timeout=10  > git checkout -f 1cc64c0e9b9dfc1ce85dfd89ee4d0795dae5e8e1 # timeout=10 Commit message: "Merge pull request 52775 from master into TEST_BRANCH" First time build. Skipping changelog. [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS

          hakamairi added a comment -

          Seems that the way to do that for azure devops and multi branch pipeline is:
          discover other refs: 

          pull/*/merge

          specify ref specs:

          +refs/pull/*/merge:refs/pull/*/merge 

           

          hakamairi added a comment - Seems that the way to do that for azure devops and multi branch pipeline is: discover other refs:  pull/*/merge specify ref specs: +refs/pull /*/merge:refs/pull/*/ merge  

          Rob added a comment -

          hakamairi's comment helped me run CI on Azure DevOps Pull Requests.

          To get Jenkins CI to find the branch, I followed their suggestion of:

          discover other refs: 

          pull/*/merge

          specify ref specs:

          +refs/pull/*/merge:refs/pull/*/merge

          Then, to get my git scm checkout step to work, I had to ensure that the refspec was only "+refs/pull//merge:refs/pull//merge"

                  steps.checkout changelog: changelog, poll: false, scm: [
                      $class: 'GitSCM',
                      branches: [
                          [
                              name: "${remoteBranch}"
                          ]
                      ],
                      doGenerateSubmoduleConfigurations: false,
                      extensions: extensions,
                      submoduleCfg: [],
                      userRemoteConfigs: [
                          [
                              credentialsId: 'Jenkins',
                              url: "${gitUrl}",
                              name: "${remoteName}",
                              refspec: "+refs/pull//merge:refs/pull//merge"
                          ]
                      ]
                  ]

          Rob added a comment - hakamairi's comment helped me run CI on Azure DevOps Pull Requests. To get Jenkins CI to find the branch, I followed their suggestion of: discover other refs:  pull/*/merge specify ref specs: +refs/pull /*/merge:refs/pull/*/ merge Then, to get my git scm checkout step to work, I had to ensure that the refspec was only " +refs/pull/ /merge:refs/pull/ /merge"         steps.checkout changelog: changelog, poll: false, scm: [             $class: 'GitSCM',             branches: [                 [                     name: "${remoteBranch}"                 ]             ],             doGenerateSubmoduleConfigurations: false,             extensions: extensions,             submoduleCfg: [],             userRemoteConfigs: [                 [                     credentialsId: 'Jenkins',                     url: "${gitUrl}",                     name: "${remoteName}",                     refspec: "+refs/pull/ /merge:refs/pull/ /merge"                 ]             ]         ]

            Unassigned Unassigned
            dordor dor s
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: