Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45175

Multibranch Pipeline does not prune/track remote branches with git 1.7.1

      Today i was wondering why a branch that was deleted on the remote repository does not get marked as removed in the Pipeline Multibranch Projekt.

      So I did a couple of "Scan Multibranch Pipeline Now" and had a closer look at the log, but while the statements seemed ok in the first place, the branches are still all listed and even though a prune option is on the fetch they did not get removed.

       > /usr/bin/git rev-parse --is-inside-work-tree # timeout=10
      Setting origin to gitserver:/data/git/Repository.git
       > /usr/bin/git config remote.origin.url gitserver:/data/git/Repository.git # timeout=10
      Fetching origin...
      Fetching upstream changes from origin
       > /usr/bin/git --version # timeout=10
      using GIT_SSH to set credentials Jenkins private Key zur Authentifizierung auf gitserver
       > /usr/bin/git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/*
      using GIT_SSH to set credentials Jenkins private Key zur Authentifizierung auf gitserver
       > /usr/bin/git ls-remote gitserver:/data/git/Repository.git # timeout=10
       > /usr/bin/git rev-parse --is-inside-work-tree # timeout=10
      Setting origin to gitserver:/data/git/Repository.git
       > /usr/bin/git config remote.origin.url gitserver:/data/git/Repository.git # timeout=10
      Fetching & pruning origin...
      Fetching upstream changes from origin
       > /usr/bin/git --version # timeout=10
      using GIT_SSH to set credentials Jenkins private Key zur Authentifizierung auf gitserver
       > /usr/bin/git fetch --tags --progress origin +refs/heads/*:refs/remotes/origin/* --prune
      Getting remote branches...
      Seen branch in repository origin/bugfix/JIRA-6365
      Seen branch in repository origin/development
      Seen branch in repository origin/release/14.1
      ...

      so i was looking into the <JENKINS_HOME>/cache directory and fired a couple of git commands manually:

      git prune
      git fetch --prune
      git remote prune origin
      

      but nothing happened to the refs/remotes/origin/bugfix... (that should haved been removed)
      after more investigation I stumbled upon the following command and compared the output to my local copy and the surprise was that all branches in the jenkins cache does not seem to be tracked:

      # in the Jenkins cache (used for the multi branch indexing)
      git remote show origin
      * remote origin
        Fetch URL: gitserver:/data/git/Repository.git
        Push  URL: gitserver:/data/git/Repository.git
        HEAD branch: development
      
      # locally cloned repository
       git remote show origin
      * remote origin
        Fetch URL: gitserver:/data/git/Repository.git
        Push  URL: gitserver:/data/git/Repository.git
        HEAD branch: development
        Remote branches:
          bugfix/JIRA-6365 tracked
          development      tracked
          release/14.1     tracked
        Local branches configured for 'git pull':
      ...
        Local refs configured for 'git push':
      ...
      

      The only difference I could find was that in the .git/config

      [remote "origin"]
              url = gitserver:/data/git/Repository.git
              fetch = +refs/heads/*:refs/remotes/origin/*
      

      the fetch line was missing. After I added that in a local copy and made a git fetch --prune the stale branches got removed.

      git fetch --prune
      From gitserver:/data/git/Repository.git
       - [deleted]           (none)     -> origin/bugfix/JIRA-6365
      

      I also replayed the commands from the Multibranch Scan Log with a local git (git version 2.13.0.windows.1) and had the same problems - the remote references where fetched but not tracked.

      I hope I did not miss a thing here and you can fix this.

      Update: Jenkins is using the command line git in version 1.7.1

          [JENKINS-45175] Multibranch Pipeline does not prune/track remote branches with git 1.7.1

          Mark Waite added a comment -

          I think that the multi-branch pipeline does not depend on the setting of "prune stale remote tracking branches". There are settings in the multi-branch pipeline job configuration ("Advanced" section, I believe) which control what should be done when remote branches have been deleted. I don't think that you want to set the git plugin option "Prune stale remote-tracking branches". Rather, you want to configure the expiration period for deleted branches to be 0 so that they will be removed immediately.

          You could enable JGit on your master, configure the multi-branch job to use it, and see if the behavior improves. JGit is a pure Java implementation of a subset of the command line git features. It works well in many use cases (though not all).

          Command line git version 1.7.1 is old enough that it is not officially supported by the git plugin. The git plugin wiki page notes that it needs at least 1.7.10. I continue to test with command line git 1.7.1 in my automation environment because it is still available on CentOS 6. However, there are many git operations which simply don't work on command line git 1.7.1. I frequently annotate tests to avoid running on older versions like git 1.7.1.

          There are authentication use cases which don't work, submodule use cases which don't work, sparse checkout use cases which don't work, large file support uses cases which don't work, and shallow clone use cases which don't work. I'm sure there are others.

          I think that the case you're seeing is not related to command line git version number, but wanted to be sure you understand that command line git versions older than 1.7.10 are only tracked on a best effort basis, and there are many functions which don't work and won't ever work.

          Mark Waite added a comment - I think that the multi-branch pipeline does not depend on the setting of "prune stale remote tracking branches". There are settings in the multi-branch pipeline job configuration ("Advanced" section, I believe) which control what should be done when remote branches have been deleted. I don't think that you want to set the git plugin option "Prune stale remote-tracking branches". Rather, you want to configure the expiration period for deleted branches to be 0 so that they will be removed immediately. You could enable JGit on your master, configure the multi-branch job to use it, and see if the behavior improves. JGit is a pure Java implementation of a subset of the command line git features. It works well in many use cases (though not all). Command line git version 1.7.1 is old enough that it is not officially supported by the git plugin. The git plugin wiki page notes that it needs at least 1.7.10. I continue to test with command line git 1.7.1 in my automation environment because it is still available on CentOS 6. However, there are many git operations which simply don't work on command line git 1.7.1. I frequently annotate tests to avoid running on older versions like git 1.7.1. There are authentication use cases which don't work, submodule use cases which don't work, sparse checkout use cases which don't work, large file support uses cases which don't work, and shallow clone use cases which don't work. I'm sure there are others. I think that the case you're seeing is not related to command line git version number, but wanted to be sure you understand that command line git versions older than 1.7.10 are only tracked on a best effort basis, and there are many functions which don't work and won't ever work.

          Hi markewaite, thank you for your quick reply.

          I think you are right, that the advanced git behavior "prune stale remote tracking branches" is only passed to the actual branch build jobs and not related to the index scan behavior of the Multibranch Pipeline Project. For all the other options you mentioned, the Job actually does not use them (submodules, sparse checkouts etc.).

          Relating the advanced options when a branch is removed you mean the "Orphaned Item Strategy"? This one is checked and "Days to keep old items" is 30, while "Max # of old items to keep" was set to 0, now empty. I like keeping old branches visible for some days, but they are actually not build anymore and the text is striked trough, so it's clear they are gone. I'll retest if the Max # = 0 may be actually the problem.

          For JGit, we found some usecases where that does not work (.gitattributes for example) and I've never seen a option to select the Git implementation (maybe cause i had only one configured, but I'll check this).

          Thank you a lot for keeping the 1.7.1 tests around, i know it's very old but the Jenkins runs on CentOS 6, so we're stuck with it for now.

          What left me helpless was that the log always showed that the branch is still around, while he was not and i could not see a mistake in the fired git commands and when testing them locally the results made no sense either.

          Daniel Geißler added a comment - Hi  markewaite , thank you for your quick reply. I think you are right, that the advanced git behavior "prune stale remote tracking branches" is only passed to the actual branch build jobs and not related to the index scan behavior of the Multibranch Pipeline Project. For all the other options you mentioned, the Job actually does not use them (submodules, sparse checkouts etc.). Relating the advanced options when a branch is removed you mean the "Orphaned Item Strategy"? This one is checked and "Days to keep old items" is 30, while "Max # of old items to keep" was set to 0, now empty. I like keeping old branches visible for some days, but they are actually not build anymore and the text is striked trough, so it's clear they are gone. I'll retest if the Max # = 0 may be actually the problem. For JGit, we found some usecases where that does not work (.gitattributes for example) and I've never seen a option to select the Git implementation (maybe cause i had only one configured, but I'll check this). Thank you a lot for keeping the 1.7.1 tests around, i know it's very old but the Jenkins runs on CentOS 6, so we're stuck with it for now. What left me helpless was that the log always showed that the branch is still around, while he was not and i could not see a mistake in the fired git commands and when testing them locally the results made no sense either.

          Ok first test as mentioned: CLI git, but "Max # of old items to keep" empty
          Result: old branch is still around and Jenkins thinks it's "alive"

          2nd test with JGit (after configuring it i can select it in the project configuration)
          Result: deleting branches in the central repository works exactly as it should

          So I retested with CLI git on ubuntu with git 2.7.4 and it just works.

          In the end there seems to be something off with git 1.7.1.

          Thanks for guiding me into the right direction, now I'll have to see what i can make out of that.

          Daniel Geißler added a comment - Ok first test as mentioned: CLI git, but "Max # of old items to keep" empty Result: old branch is still around and Jenkins thinks it's "alive" 2nd test with JGit (after configuring it i can select it in the project configuration) Result: deleting branches in the central repository works exactly as it should So I retested with CLI git on ubuntu with git 2.7.4 and it just works. In the end there seems to be something off with git 1.7.1. Thanks for guiding me into the right direction, now I'll have to see what i can make out of that.

          Mark Waite added a comment -

          You could experiment with making git an automatically installed tool. In order to do that, you'd need to compile it for your target platform, package it as a compressed tar file, then configure Jenkins to install that for you.

          I say "could" because that is a technique I've used with other tools (ant, maven, cmake), but have not used it with git. I think it should work, but I have not verified for myself that it does work. Compiling git from source is not nearly as easy as installing it from a yum repository.

          Other options may be available in this stackoverflow article.

          Mark Waite added a comment - You could experiment with making git an automatically installed tool. In order to do that, you'd need to compile it for your target platform, package it as a compressed tar file, then configure Jenkins to install that for you. I say "could" because that is a technique I've used with other tools (ant, maven, cmake), but have not used it with git. I think it should work, but I have not verified for myself that it does work. Compiling git from source is not nearly as easy as installing it from a yum repository. Other options may be available in this stackoverflow article .

          Mark Waite added a comment -

          I've closed this as "Won't fix" since it seems to be specific to git version 1.7.1 and that is not a supported version. The git version needs to be at least 1.7.10. Far better if it is 1.9 or greater.

          There are known poor interactions between the git plugin and command line git versions prior to 1.9. They won't be fixed. For example, command line git 1.8 and 1.7 sometimes report that the data stream from a GitHub server sparse checkout is incorrect. Newer git versions do not have that problem.

          Mark Waite added a comment - I've closed this as "Won't fix" since it seems to be specific to git version 1.7.1 and that is not a supported version. The git version needs to be at least 1.7.10. Far better if it is 1.9 or greater. There are known poor interactions between the git plugin and command line git versions prior to 1.9. They won't be fixed. For example, command line git 1.8 and 1.7 sometimes report that the data stream from a GitHub server sparse checkout is incorrect. Newer git versions do not have that problem.

            Unassigned Unassigned
            dageissl Daniel Geißler
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: