Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46219

Should fetch origin/refs even if repository exists in work tree

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None

      Use case: Using git-plugin to run jobs on incoming pull requests, when the git repo exists in working tree, git-plugin fails to merge the branches because of a missing reference to the base branch.

      Behavior: git-plugin throws GitException
      Expected behavior: merge successfully the pr branch and the base branch

      Suggested solution: fetch origin references and not only the tags

      Stack trace:
      > git rev-parse --is-inside-work-tree # timeout=10
      Fetching changes from the remote Git repository
      > git config remote.origin.url https://github.com/ros/bond_core.git # timeout=10
      Fetching upstream changes from https://github.com/ros/bond_core.git
      > git --version # timeout=10
      > git fetch --tags --progress https://github.com/ros/bond_core.git +refs/pull/:refs/remotes/origin/pr/
      > git rev-parse refs/remotes/origin/pr/31/merge^

      {commit} # timeout=10
      > git rev-parse refs/remotes/origin/origin/pr/31/merge^{commit}

      # timeout=10
      Merging Revision b0644a4eb62fde36776aae1a6ecd63a5a077838f (refs/remotes/origin/pr/31/merge) to origin/groovy-devel, UserMergeOptions

      {mergeRemote='origin', mergeTarget='groovy-devel', mergeStrategy='default', fastForwardMode='--ff'}

      > git rev-parse origin/groovy-devel^

      {commit} # timeout=10
      FATAL: Command "git rev-parse origin/groovy-devel^{commit}

      " returned status code 128:
      stdout: origin/groovy-devel^

      {commit}

      Workaround: wipe the workspace of the job

      Link to failing Jenkins job:
      http://build.ros.org/job/Ipr__bond_core__ubuntu_trusty_amd64/25/console

      This is my first issue on Jenkins please let me know if there is any missing information or if this is not the right way to describe an issue.
      Thanks!

      Edit: markewaite Sorry for the delay. For various reasons our buildfarm has an "old" version of Jenkins and some plugins. We're spinning up a new farm with the latest version of Jenkins and all plugins, I'll use that config to test and provide a reproducible example

          [JENKINS-46219] Should fetch origin/refs even if repository exists in work tree

          Mark Waite added a comment - - edited

          Thanks for the good details.

          We'll need more details to understand what's happening, and ultimately to understand what you're expecting. Refer to "How to report an issue" for the types of information that will be needed.

          Some of the things that I expect will be needed include:

          • What Jenkins version?
          • What plugin versions?
          • What is the job configuration (config.xml file)?
          • What is the job type (freestyle, pipeline, multi-branch pipeline, matrix, maven, etc.)?
          • What sequence of steps will show the problem on a freshly configured system?
          • Are there any relevant differences which might make your configuration unique compared to others?
          • Are submodules involved in the checkout?
          • Is large file support (LFS) involved in the checkout?
          • If this is a pipeline job, have you made the mistake of setting doGenerateSubmoduleConfiguration to true? If you have, then you should set it to its default value of false.

          Any update on those questions?

          Mark Waite added a comment - - edited Thanks for the good details. We'll need more details to understand what's happening, and ultimately to understand what you're expecting. Refer to " How to report an issue " for the types of information that will be needed. Some of the things that I expect will be needed include: What Jenkins version? What plugin versions? What is the job configuration (config.xml file)? What is the job type (freestyle, pipeline, multi-branch pipeline, matrix, maven, etc.)? What sequence of steps will show the problem on a freshly configured system? Are there any relevant differences which might make your configuration unique compared to others? Are submodules involved in the checkout? Is large file support (LFS) involved in the checkout? If this is a pipeline job, have you made the mistake of setting doGenerateSubmoduleConfiguration to true? If you have, then you should set it to its default value of false. Any update on those questions?

          pjdarton added a comment -

          I've been trying to debug an problem on my Jenkins server, and this looks like it may be the same issue: If the build workspace already contains a .git folder, the checkout operation fails.

          We're seeing this on Jenkins 2.89.2 with git plugins Git = 3.6.4, Git client = 2.7.0, GIT server Plugin = 1.7, GitHub = 1.28.1, GitHub API Plugin = 1.90, GitHub Branch Source = 2.3.2, and the git client we're using is "PortableGit" version 2.10.0 for Windows.

          Background:
          We have some static slaves that persist between builds.
          We have builds that, at the start of the "Build Environment" phase, deletes the build workspace.
          When we first started using Git, we used to configure the "delete workspace" step to avoid deleting the .git folder and its contents so that any subsequent clone wouldn't have to pull as much data, and this seemed to work fine.
          Since updating Jenkins and the Git plugin to more recent versions, we've found that the git checkout operation fails unless we delete the entire .git folder.
          We have since reconfigured our build jobs so that their "delete workspace" step no longer excludes the .git folder and the issue has stopped hitting us but, as you might well guess, we've now got a lot more network traffic as we're doing a fresh clone every build instead of just pulling the latest changes.

          The problem seems to be that, if there's already a .git folder, the git plugin assumes that it's up to date and then tries to checkout the latest revision whereas the reality is that the .git folder contains an old clone of the code and thus does not contain the latest revision, causing the checkout operation to fail.

          pjdarton added a comment - I've been trying to debug an problem on my Jenkins server, and this looks like it may be the same issue: If the build workspace already contains a .git folder, the checkout operation fails. We're seeing this on Jenkins 2.89.2 with git plugins Git = 3.6.4, Git client = 2.7.0, GIT server Plugin = 1.7, GitHub = 1.28.1, GitHub API Plugin = 1.90, GitHub Branch Source = 2.3.2, and the git client we're using is "PortableGit" version 2.10.0 for Windows. Background: We have some static slaves that persist between builds. We have builds that, at the start of the "Build Environment" phase, deletes the build workspace. When we first started using Git, we used to configure the "delete workspace" step to avoid deleting the .git folder and its contents so that any subsequent clone wouldn't have to pull as much data, and this seemed to work fine. Since updating Jenkins and the Git plugin to more recent versions, we've found that the git checkout operation fails unless we delete the entire .git folder. We have since reconfigured our build jobs so that their "delete workspace" step no longer excludes the .git folder and the issue has stopped hitting us but, as you might well guess, we've now got a lot more network traffic as we're doing a fresh clone every build instead of just pulling the latest changes. The problem seems to be that, if there's already a .git folder, the git plugin assumes that it's up to date and then tries to checkout the latest revision whereas the reality is that the .git folder contains an old clone of the code and thus does not contain the latest revision, causing the checkout operation to fail.

          Mark Waite added a comment -

          @pjdarton I don't understand (based on your description) how to duplicate the conditions you're seeing. Can you create a repeatable series of steps so that others could see the same behavior?

          Mark Waite added a comment - @pjdarton I don't understand (based on your description) how to duplicate the conditions you're seeing. Can you create a repeatable series of steps so that others could see the same behavior?

          pjdarton added a comment - - edited

          Have a Git repository (that isn't empty).

          Have a Windows slave (we're using Windows 7 x64, in case that makes a difference) separate from the Jenkins master. Give it a label of "win64" plus anything else you want.

          Configure Jenkins' "Global Tool Configuration" "Git installations" section with a git tool with the URL of a zip of "PortableGit" version 2.10.0 for Windows so that Jenkins can auto-install Git on slaves that need it:

          Have a Jenkins build job configured to:

          • restrict where this project can run so that it'll only ever run on the Windows slave mentioned above.
          • Source Code Management = Git.
            • Specify one repository
            • name "origin"
            • no refspec
            • Branches to build is "*/master"
            • Repository browser is "(Auto)"
            • no additional behaviours.
          • Build Triggers = poll SCM
            • Schedule = "H/2 * * * *"
          • Do not "Delete workspace before build starts".
          • have a single build step that just lists all the files out.

          Jenkins should detect that it's never built the build before and build it:

          • It'll auto-install git into the slave
          • then use git to grab the source code from the git repository into the Jenkins workspace.
            That should all work and, in the console output of the build, you'll see the .git folder there plus the contents of the Git repository.
          • The git-checkout phase of the build should look something like the attachment success_when_no_git_folder_exists.txt.

          Now make and commit a change into the Git repository.

          After a while, Jenkins should detect the change and re-trigger the build.  That'll run on the same Windows slave as before, in the same workspace folder, which has the .git folder left over from the previous build and hence has a clone of the repository that contains all the changes except for the one that was just committed.
          What I see, in this scenario, is that the git checkout fails:

          • it doesn't pull changes from the git repo so that when it later tries to checkout the latest revision, it can't (as the local .git folder doesn't contain the latest code).
          • The git-checkout phase of the build should look something like the attachment fails_when_git_folder_exists.txt.

          (if there's any of the above that needs clarifying, let me know and I'll add more detail)

          pjdarton added a comment - - edited Have a Git repository (that isn't empty). Have a Windows slave (we're using Windows 7 x64, in case that makes a difference) separate from the Jenkins master. Give it a label of "win64" plus anything else you want. Configure Jenkins' "Global Tool Configuration" "Git installations" section with a git tool with the URL of a zip of "PortableGit" version 2.10.0 for Windows so that Jenkins can auto-install Git on slaves that need it: Git: Name = Default Path to Git executable = git Tick "Install automatically" One installer of type "Extract .zip/ .tar.gz": Label = win64 (so it'll match the windows slave above) Download URL = http://some.local.server/.../PortableGit-2.10.0-Win64.zip Subdirectory = bin/git.exe Have a Jenkins build job configured to: restrict where this project can run so that it'll only ever run on the Windows slave mentioned above. Source Code Management = Git. Specify one repository name "origin" no refspec Branches to build is "*/master" Repository browser is "(Auto)" no additional behaviours. Build Triggers = poll SCM Schedule = "H/2 * * * *" Do not "Delete workspace before build starts". have a single build step that just lists all the files out. Jenkins should detect that it's never built the build before and build it: It'll auto-install git into the slave it'll log something along the lines of "Unpacking http://some.local.server/.../PortableGit-2.10.0-Win64.zip to C:\J\tools\git on myslave") then use git to grab the source code from the git repository into the Jenkins workspace. That should all work and, in the console output of the build, you'll see the .git folder there plus the contents of the Git repository. The git-checkout phase of the build should look something like the attachment success_when_no_git_folder_exists.txt . Now make and commit a change into the Git repository. After a while, Jenkins should detect the change and re-trigger the build.  That'll run on the same Windows slave as before, in the same workspace folder, which has the .git folder left over from the previous build and hence has a clone of the repository that contains all the changes except for the one that was just committed. What I see, in this scenario, is that the git checkout fails: it doesn't pull changes from the git repo so that when it later tries to checkout the latest revision, it can't (as the local .git folder doesn't contain the latest code). The git-checkout phase of the build should look something like the attachment fails_when_git_folder_exists.txt . (if there's any of the above that needs clarifying, let me know and I'll add more detail)

            Unassigned Unassigned
            mikaelarguedas Mikael Arguedas
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: