Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45997

BitBucket PRs failing to checkout on initial run after PR updated with new commits

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Updating an existing Pull-Request with additional commits leads to broken builds of the Pull Request, because jenkins tries to check out a not existing ref in git.

      Reproduction:

      • create a branch in the repository
      • create a new commit 
      • create a multibranch job building all branches and pull requests
      • create a new pull request in Bitbucket server from the new branch to master
      • scan for changes in the multibranch job -> the first build of the PR succeeds
      • change some files on the branch and commit with the --amend flag set and push with force flag Edit: Found out, that a normal commit to the branch also crashes the build.
      • scan for changes in the multibranch job -> a build in the PR is triggered but fails due to git complaining that the ref does not exist
      • After visiting the Pull-Request overview in Bitbucket, following builds succeed

       

      This is the jenkins console output:

      Started by user admin
      Checking out git http://xstash.itd.meisterplan.io/scm/mp/failing-builds.git http://xstash.itd.meisterplan.io/scm/mp/failing-builds.git into /var/jenkins_home/workspace/failing-builds_PR-3-5GPUDFBBY64BPS5T63E3TRH74RJSNKWF5A6YELFFKHSOT5HTAQ4A@script to read Jenkinsfile
       > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from 2 remote Git repositories
       > git config remote.origin.url http://xstash.itd.meisterplan.io/scm/mp/failing-builds.git # timeout=10
      Fetching without tags
      Fetching upstream changes from http://xstash.itd.meisterplan.io/scm/mp/failing-builds.git
       > git --version # timeout=10
      using GIT_ASKPASS to set credentials Bitbucket-Server Account
       > git fetch --no-tags --progress http://xstash.itd.meisterplan.io/scm/mp/failing-builds.git +refs/pull-requests/3/from:refs/remotes/origin/PR-3
       > git config remote.upstream.url http://xstash.itd.meisterplan.io/scm/mp/failing-builds.git # timeout=10
      Fetching without tags
      Fetching upstream changes from http://xstash.itd.meisterplan.io/scm/mp/failing-builds.git
      using GIT_ASKPASS to set credentials Bitbucket-Server Account
       > git fetch --no-tags --progress http://xstash.itd.meisterplan.io/scm/mp/failing-builds.git +refs/heads/master:refs/remotes/upstream/master
      Merging remotes/upstream/master commit d4eaaccd2e0f61e87b2a48fcae8a8b6693038a48 into PR head commit cec48748dc421b835274476029a9c98123babc15
       > git config core.sparsecheckout # timeout=10
       > git checkout -f cec48748dc421b835274476029a9c98123babc15
      [Bitbucket] Notifying pull request build result
      [Bitbucket] Build result notified
      hudson.plugins.git.GitException: Command "git checkout -f cec48748dc421b835274476029a9c98123babc15" returned status code 128:
      stdout: 
      stderr: fatal: reference is not a tree: cec48748dc421b835274476029a9c98123babc15
      
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1924)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$800(CliGitAPIImpl.java:71)
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2214)
      Caused: hudson.plugins.git.GitException: Could not checkout cec48748dc421b835274476029a9c98123babc15
      	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2237)
      	at jenkins.plugins.git.MergeWithGitSCMExtension.checkout(MergeWithGitSCMExtension.java:146)
      	at jenkins.plugins.git.MergeWithGitSCMExtension.decorateRevisionToBuild(MergeWithGitSCMExtension.java:112)
      	at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:1023)
      	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1116)
      	at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:109)
      	at org.jenkinsci.plugins.workflow.cps.CpsScmFlowDefinition.create(CpsScmFlowDefinition.java:130)
      	at org.jenkinsci.plugins.workflow.multibranch.SCMBinder.create(SCMBinder.java:120)
      	at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:246)
      	at hudson.model.ResourceController.execute(ResourceController.java:97)
      	at hudson.model.Executor.run(Executor.java:405)
      Finished: FAILURE

      I opened an issue at the bitbucket atlassian support and got the answer, that this behavior is by design and one should never checkout refs, which are generated lazily.

      This is the full answer from atlassian:

      I'm one of the Bitbucket Server developers. The issue you're running into here is from fetching pull request refs. Here's a detailed response I wrote to someone else having a similar issue.
      When new commits are pushed to a branch which has an open pull request, the pull request hashes are eagerly updated in the database but the refs on disk are not. This behavior has existed unchanged since pull requests were introduced in Stash 1.3.0. If someone views the pull request, then the refs are updated as part of performing a new merge to show an up-to-date diff. Any integration which attempts to use refs/pull-requests/*/from is going to be racy and unreliable because that ref is not eagerly updated.

        Attachments

        1. screenshot-1.png
          8 kB
          Patrick Ruckstuhl
        2. Screenshot from 2018-07-13 11-33-59.png
          26 kB
          Kevin de Berk

          Issue Links

            Activity

            Hide
            kdberk Kevin de Berk added a comment -

            It seems to be working now! Thanks for the help and quick response!

            It took me a while before I could find the option that you and other people here were referring to. I don't understand why a google search for "Call can merge" (with quotes!) only generates 2 results, of which one is this page and the other completely irrelevant.

            Show
            kdberk Kevin de Berk added a comment - It seems to be working now! Thanks for the help and quick response! It took me a while before I could find the option that you and other people here were referring to. I don't understand why a google search for "Call can merge" (with quotes!) only generates 2 results, of which one is this page and the other completely irrelevant.
            Hide
            markewaite Mark Waite added a comment -

            It seems that the "Call can merge" selection is only available for Bitbucket Server connections, not for connections to Bitbucket.org. I assume that is because it is not needed when connecting to Bitbucket.org. I was unable to duplicate this bug report with Bitbucket.org.

            Show
            markewaite Mark Waite added a comment - It seems that the "Call can merge" selection is only available for Bitbucket Server connections, not for connections to Bitbucket.org. I assume that is because it is not needed when connecting to Bitbucket.org. I was unable to duplicate this bug report with Bitbucket.org.
            Hide
            damyan Damyan Yordanov added a comment -

            Confirmation: when the "call can merge" is not set, an error message "reference is not a tree" occurrs, when trying to build a PR. Setting the option seems to fix the issue on our internal Bitbucket Server. Thank you!

            Show
            damyan Damyan Yordanov added a comment - Confirmation: when the "call can merge" is not set, an error message "reference is not a tree" occurrs, when trying to build a PR. Setting the option seems to fix the issue on our internal Bitbucket Server. Thank you!
            Hide
            macdrega Joerg Schwaerzler added a comment - - edited

            Not sure whether this is the same issue. But this is the closet ticket I could find.
            For us Bitbucket + Branch Source plugin was working just find - until today. Now for one of our repositories I cannot convince Jenkins to clone the repository correctly ritght after a PR was updated or created.

            Now we always would have to re-trigger each PR build manually in order to get it running. Once triggered automatically the builds won't work (see below).
            Furthermore - other, unrelated jobs which are trying to clone the repository at the very same point in time when the PR is updated and which are not even trying to check out the pull request revision but master (of the same repository) - happen to fail with the very same error messages!

            Not sure whether this would be an issue in Bitbucket or in Jenkins git plugin or Bitbucket branch source plugin. Maybe somebody can comment.

            The error message is either:

            Cloning the remote Git repository
            Cloning with configured refspecs honoured and without tags
            ERROR: Error cloning remote repo 'origin'
            hudson.plugins.git.GitException: Command "git fetch --no-tags --progress https://******* +refs/pull-requests/66/from:refs/remotes/origin/PR-66" returned status code 128:
            stdout: 
            stderr: fatal: Couldn't find remote ref refs/pull-requests/66/from
            

            or something like (using a slightly different refspec):

            git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* +refs/pull-requests/*/merge:refs/remotes/origin/PR-*" returned status code 128:
            stdout: 
            stderr: error: refs/pull-requests/66/from does not point to a valid object!
            error: Could not read 7a99185626f667ab8c34d502417c47e99778e633
            error: Could not read 7a99185626f667ab8c34d502417c47e99778e633
            remote: error: Could not read 7a99185626f667ab8c34d502417c47e99778e633        
            remote: fatal: revision walk setup failed        
            error: git upload-pack: git-pack-objects died with error.
            fatal: git upload-pack: aborting due to possible repository corruption on the remote side.
            remote: aborting due to possible repository corruption on the remote side.
            fatal: protocol error: bad pack header
            

            Curiously the Jenkins can be cloned. However later when it comes to the `checkout scm` resp, git step it won't work.

            Show
            macdrega Joerg Schwaerzler added a comment - - edited Not sure whether this is the same issue. But this is the closet ticket I could find. For us Bitbucket + Branch Source plugin was working just find - until today. Now for one of our repositories I cannot convince Jenkins to clone the repository correctly ritght after a PR was updated or created. Now we always would have to re-trigger each PR build manually in order to get it running. Once triggered automatically the builds won't work (see below). Furthermore - other, unrelated jobs which are trying to clone the repository at the very same point in time when the PR is updated and which are not even trying to check out the pull request revision but master (of the same repository) - happen to fail with the very same error messages ! Not sure whether this would be an issue in Bitbucket or in Jenkins git plugin or Bitbucket branch source plugin. Maybe somebody can comment. The error message is either: Cloning the remote Git repository Cloning with configured refspecs honoured and without tags ERROR: Error cloning remote repo 'origin' hudson.plugins.git.GitException: Command "git fetch --no-tags --progress https: //******* +refs/pull-requests/66/from:refs/remotes/origin/PR-66" returned status code 128: stdout: stderr: fatal: Couldn't find remote ref refs/pull-requests/66/from or something like (using a slightly different refspec): git fetch --tags --progress origin +refs/heads /*:refs/remotes/origin/* +refs/pull-requests/*/ merge:refs/remotes/origin/PR-*" returned status code 128: stdout: stderr: error: refs/pull-requests/66/from does not point to a valid object! error: Could not read 7a99185626f667ab8c34d502417c47e99778e633 error: Could not read 7a99185626f667ab8c34d502417c47e99778e633 remote: error: Could not read 7a99185626f667ab8c34d502417c47e99778e633 remote: fatal: revision walk setup failed error: git upload-pack: git-pack-objects died with error. fatal: git upload-pack: aborting due to possible repository corruption on the remote side. remote: aborting due to possible repository corruption on the remote side. fatal: protocol error: bad pack header Curiously the Jenkins can be cloned. However later when it comes to the `checkout scm` resp, git step it won't work.
            Hide
            markewaite Mark Waite added a comment -

            Joerg Schwaerzler it is usually much less effective to request help by updating an existing closed bug report than to ask for help on the mailing lists or in the chat channels. There are far fewer people watching updates to Jenkins bug reports than watch the mailing lists and the chat channels.

            In your case, the messages may indicate that a cache repository on the Jenkins master has been damaged or left in an inconsistent state. You may want to look in the $JENKINS_HOME/caches directory to find the directory which is used as the cache for that job. One technique I've used has been to:

            $ cd $JENKINS_HOME/caches
            $ for repo in *; do (cd $repo && echo ==== $repo && git remote -v | grep fetch); done
            

            That will list the remote repository for each of those cache directories. Remove the directory for the remote repository that is having that problem. The next use of the cache will clone that repository and may be enough to resolve the issue.

            Show
            markewaite Mark Waite added a comment - Joerg Schwaerzler it is usually much less effective to request help by updating an existing closed bug report than to ask for help on the mailing lists or in the chat channels . There are far fewer people watching updates to Jenkins bug reports than watch the mailing lists and the chat channels. In your case, the messages may indicate that a cache repository on the Jenkins master has been damaged or left in an inconsistent state. You may want to look in the $JENKINS_HOME/caches directory to find the directory which is used as the cache for that job. One technique I've used has been to: $ cd $JENKINS_HOME/caches $ for repo in *; do (cd $repo && echo ==== $repo && git remote -v | grep fetch); done That will list the remote repository for each of those cache directories. Remove the directory for the remote repository that is having that problem. The next use of the cache will clone that repository and may be enough to resolve the issue.

              People

              Assignee:
              tario Patrick Ruckstuhl
              Reporter:
              mkaulig Michael Kaulig
              Votes:
              44 Vote for this issue
              Watchers:
              54 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: