Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45586

Recent Changes not updated when using GIT Shallow clone option

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • Jenkins 2.60.1 LTS, all latest plugins on July 17 2017.
      Jenkins server running Cent OS and Client running on a Windows Server 2012R2 VM.

      In a Multibranch job, if the Jenkins File contains the following:

      checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\{env.BRANCH_NAME}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: +*true*+]], submoduleCfg: [], userRemoteConfigs: [url: '[http://My]]

      The "Recent Changes" of the branch will not update.  Previous version (LTS 2.32.1) with old plugins did not had this problem.

      Changing the checkout to:

      checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\{env.BRANCH_NAME}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: +*false*+]], submoduleCfg: [], userRemoteConfigs: [url: '[http://My]]

      allows to get the list of "Recent Changes" populated again, but at the cost of a considerably higher build time and stress on the GIT server as the repository need to be cloned from scratch every time (as we are running agents on VMs that are always started from the same snapshot).

       

          [JENKINS-45586] Recent Changes not updated when using GIT Shallow clone option

          Mark Waite added a comment -

          Since a shallow clone may only clone the most recent change, it seems (at least to me) that the change list may be incorrect for a shallow clone that follows another shallow clone. If a shallow clone is followed by 2 or more commits to the repository, then another shallow clone is performed, the commits in the working repository may only be the original commit and the most recent commit in the repository.

          You might try reducing load on your git server by using a narrow refspec instead of a shallow clone. A narrow refspec allows the git server to only return the history of a single branch, while a shallow clone allows the server to return the history of a limited number of commits for each branch that matches the refspec. The refspec setting is modified in the "Advanced" section of the repository definition.

          Mark Waite added a comment - Since a shallow clone may only clone the most recent change, it seems (at least to me) that the change list may be incorrect for a shallow clone that follows another shallow clone. If a shallow clone is followed by 2 or more commits to the repository, then another shallow clone is performed, the commits in the working repository may only be the original commit and the most recent commit in the repository. You might try reducing load on your git server by using a narrow refspec instead of a shallow clone. A narrow refspec allows the git server to only return the history of a single branch, while a shallow clone allows the server to return the history of a limited number of commits for each branch that matches the refspec. The refspec setting is modified in the "Advanced" section of the repository definition.

          This is a repo with many years of commits (and all commit being merged to master).  So a shallow copy (with maybe a depth of 5) allow us to copy much less data.  I know that many commits between build could end up in the list of "Recent Changes" missing some entries, but this is a price we are ready to pay to reduce build time.

          We will however experiment with refspec in the mean time.

          Frédérick St-Laurent added a comment - This is a repo with many years of commits (and all commit being merged to master).  So a shallow copy (with maybe a depth of 5) allow us to copy much less data.  I know that many commits between build could end up in the list of "Recent Changes" missing some entries, but this is a price we are ready to pay to reduce build time. We will however experiment with refspec in the mean time.

          Frédérick St-Laurent added a comment - - edited

          I look like refspec will not be "totally respected" on the first checkout.  With the following script:

          node( "MyWindowsSlave" )
          {
          def branch = "bugfix/FPP-1229-use-refspec"
          def myRepo = "http://mtlstash.miranda.com:7990/scm/fpp/fpp.git"
          checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\{branch\}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: false]], submoduleCfg: [], userRemoteConfigs: [[refspec: "+refs/heads/$\{branch\}:refs/remotes/origin/$\{branch\}", url: myRepo]]]
          }

          The first checkout will give me something that looks like:

          Cloning the remote Git repository
          Avoid fetching tags
          Cloning repository http://myserver/myrepo.git
          > git init d:\Jenkins\Workspaces\workspace\TestFred # timeout=10
          Fetching upstream changes from http://myserver/myrepo.git
          > git --version # timeout=10
          > git fetch --no-tags --progress http://myserver/myrepo.git +refs/heads/:refs/remotes/origin/**
          > git config remote.origin.url http://myserver/myrepo.git # timeout=10
          > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
          > git config remote.origin.url http://myserver/myrepo.git # timeout=10
          Fetching upstream changes from http://myserver/myrepo.git
          > git fetch --no-tags --progress http://myserver/myrepo.git +refs/heads/bugfix/FPP-1229-use-refspec:refs/remotes/origin/bugfix/FPP-1229-use-refspec

           

          As we can see, git fetch is called twice, the first time without my refspec.  However, if I run the job again without clearing my workspace I get:

          > git rev-parse --is-inside-work-tree # timeout=10
          Fetching changes from the remote Git repository
          > git config remote.origin.url http://myserver/myrepo.git # timeout=10
          Fetching upstream changes from http://myserver/myrepo.git
          > git --version # timeout=10
          > git fetch --no-tags --progress http://myserver/myrepo.git +refs/heads/bugfix/FPP-1229-use-refspec:refs/remotes/origin/bugfix/FPP-1229-use-refspec

           

          However, we cannot keep the workspace as each of our build normally run from a cleared VM snapshot.

          Frédérick St-Laurent added a comment - - edited I look like refspec will not be "totally respected" on the first checkout.  With the following script: node( "MyWindowsSlave" ) { def branch = "bugfix/FPP-1229-use-refspec" def myRepo = "http://mtlstash.miranda.com:7990/scm/fpp/fpp.git" checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [ [name: "$\{branch\}"] ], doGenerateSubmoduleConfigurations: false, extensions: [ [$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: false] ], submoduleCfg: [], userRemoteConfigs: [ [refspec: "+refs/heads/$\{branch\}:refs/remotes/origin/$\{branch\}", url: myRepo] ]] } The first checkout will give me something that looks like: Cloning the remote Git repository Avoid fetching tags Cloning repository http://myserver/myrepo.git > git init d:\Jenkins\Workspaces\workspace\TestFred # timeout=10 Fetching upstream changes from http://myserver/myrepo.git > git --version # timeout=10 > git fetch --no-tags --progress http://myserver/myrepo.git +refs/heads/ :refs/remotes/origin/** > git config remote.origin.url http://myserver/myrepo.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/ :refs/remotes/origin/ # timeout=10 > git config remote.origin.url http://myserver/myrepo.git # timeout=10 Fetching upstream changes from http://myserver/myrepo.git > git fetch --no-tags --progress http://myserver/myrepo.git +refs/heads/bugfix/FPP-1229-use-refspec:refs/remotes/origin/bugfix/FPP-1229-use-refspec   As we can see, git fetch is called twice, the first time without my refspec.  However, if I run the job again without clearing my workspace I get: > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://myserver/myrepo.git # timeout=10 Fetching upstream changes from http://myserver/myrepo.git > git --version # timeout=10 > git fetch --no-tags --progress http://myserver/myrepo.git +refs/heads/bugfix/FPP-1229-use-refspec:refs/remotes/origin/bugfix/FPP-1229-use-refspec   However, we cannot keep the workspace as each of our build normally run from a cleared VM snapshot.

          Mark Waite added a comment -

          you need to add the additional behaviour to honor refspec on checkout.

          Mark Waite added a comment - you need to add the additional behaviour to honor refspec on checkout.

          Adding "honorRefspec: true" in the clone options fixed the problem, however as expected there is virtually no gain when doing a clone of the master branch (but some gain when doing a checkout of older branches, so it proves it is working).

          Frédérick St-Laurent added a comment - Adding "honorRefspec: true" in the clone options fixed the problem, however as expected there is virtually no gain when doing a clone of the master branch (but some gain when doing a checkout of older branches, so it proves it is working).

          I'm having the same issue and would like to stick to a shallow clone. Could it be an option to ask the gitlab/github server via the API to send us the changelog if we are doing a shallow clone instead of calculating itself?

          Benjamin Fuchs added a comment - I'm having the same issue and would like to stick to a shallow clone. Could it be an option to ask the gitlab/github server via the API to send us the changelog if we are doing a shallow clone instead of calculating itself?

          Mark Waite added a comment -

          vollmilch the only way that I know to compute a changelog is to get the repository history by clone or fetch.

          I'm not aware of any API (GitHub or Bitbucket or Gitea or GoGS or Gitweb or ...) which will provide the changelog without cloning the repository.

          Mark Waite added a comment - vollmilch the only way that I know to compute a changelog is to get the repository history by clone or fetch. I'm not aware of any API (GitHub or Bitbucket or Gitea or GoGS or Gitweb or ...) which will provide the changelog without cloning the repository.

          Hi markewaite,

          At least for Bitbucket there is a wonderful REST API which allows you to get a change log. See https://docs.atlassian.com/bitbucket-server/rest/5.6.2/bitbucket-rest.html#idm46036344586576

          Matthias Tietz added a comment - Hi markewaite , At least for Bitbucket there is a wonderful REST API which allows you to get a change log. See https://docs.atlassian.com/bitbucket-server/rest/5.6.2/bitbucket-rest.html#idm46036344586576

          We just hit this problem as well. It seems to me that this is some sort of regress. We recently upgraded our Jenkins build farm to the latest weekly build and latest plugin versions. Prior to the upgrade the behavior of shallow clones was as expected (ie: showing the correct list of changes) and afterwards it breaks.

          Prior to the upgrade we had the following versions:

          • Git plugin v3.3.2
          • Jenkins core v2.46.3
          • Pipeline v2.5

          After the upgrade we are now using:

          • Git plugin v3.7.0
          • Jenkins core v2.110
          • Pipeline v2.5

          Also I should mention that enabling the honorRefspec option didn't have any effect in our environment. Not sure why it has worked for others and not for us.

          Kevin Phillips added a comment - We just hit this problem as well. It seems to me that this is some sort of regress. We recently upgraded our Jenkins build farm to the latest weekly build and latest plugin versions. Prior to the upgrade the behavior of shallow clones was as expected (ie: showing the correct list of changes) and afterwards it breaks. Prior to the upgrade we had the following versions: Git plugin v3.3.2 Jenkins core v2.46.3 Pipeline v2.5 After the upgrade we are now using: Git plugin v3.7.0 Jenkins core v2.110 Pipeline v2.5 Also I should mention that enabling the honorRefspec option didn't have any effect in our environment. Not sure why it has worked for others and not for us.

          I probably should also mention that adjusting the depth of the shallow clone does have an effect. For example, if you are certain that you are running a build after every commit, you can set the clone depth to 2 and you get the expected list of modified files. Presumably this works because the local checkout has sufficient history to be able to correctly calculate the diff. However, if your builds may vary in frequency and thus have varying numbers of commits in each build this solution will not work. For example, if you set the depth to 2 and there are 4 commits in the current build, you'll only see the changes for the last commit.

          I just thought I'd mention it here in case it helps some users. 

          Kevin Phillips added a comment - I probably should also mention that adjusting the depth of the shallow clone does have an effect. For example, if you are certain that you are running a build after every commit, you can set the clone depth to 2 and you get the expected list of modified files. Presumably this works because the local checkout has sufficient history to be able to correctly calculate the diff. However, if your builds may vary in frequency and thus have varying numbers of commits in each build this solution will not work. For example, if you set the depth to 2 and there are 4 commits in the current build, you'll only see the changes for the last commit. I just thought I'd mention it here in case it helps some users. 

          Suggestion: unless / untill an appropriate fix can be made for this problem, would it be reasonable to simply ignore the list of modified files when doing a shallow checkout? Currently the information being displayed on the Jenkins UI is completely incorrect in these situations, which leads to frustration and confusion to users. At the very least if the list was empty, or maybe a simple message was displayed saying changed files can not be shown because job uses a shallow clone.

          For users that link their repository browsers to their Jenkins jobs can still refer back to the source repo to get a correct list of changes as a fallback anyway. And having no information as opposed to incorrect information would be the lesser of two evils imo.

          Kevin Phillips added a comment - Suggestion: unless / untill an appropriate fix can be made for this problem, would it be reasonable to simply ignore the list of modified files when doing a shallow checkout? Currently the information being displayed on the Jenkins UI is completely incorrect in these situations, which leads to frustration and confusion to users. At the very least if the list was empty, or maybe a simple message was displayed saying changed files can not be shown because job uses a shallow clone. For users that link their repository browsers to their Jenkins jobs can still refer back to the source repo to get a correct list of changes as a fallback anyway. And having no information as opposed to incorrect information would be the lesser of two evils imo.

          Mark Waite added a comment -

          leedega thanks for the suggestion. If the plugin ignores the changes when performing a shallow checkout, I'm reasonably confident there will be users which (correctly) declare that it is a regression from previous behavior. They had already accepted that the shallow clone could not show all the changes because they did not request all the changes from the origin repository and they were OK with it in their environment.

          I don't plan to modify the git plugin changes list handling for shallow clones.

          Users that want accurate lists of changes will need to avoid shallow clone. When the changes aren't available in the repository, the plugin won't show those changes.

          Mark Waite added a comment - leedega thanks for the suggestion. If the plugin ignores the changes when performing a shallow checkout, I'm reasonably confident there will be users which (correctly) declare that it is a regression from previous behavior. They had already accepted that the shallow clone could not show all the changes because they did not request all the changes from the origin repository and they were OK with it in their environment. I don't plan to modify the git plugin changes list handling for shallow clones. Users that want accurate lists of changes will need to avoid shallow clone. When the changes aren't available in the repository, the plugin won't show those changes.

          markewaite

          haha - I'm guessing you are absolutely correct. However, if someone complains about a regression (aka: a change in previous behavior) when the previous behavior was incorrect is, well, questionable anyway. It's hard to maintain backwards compatibility AND fix incorrect behavior at the same time since the latter, by definition, changes the previous behavior.

          That being said, what if there were some way to control that at least... like maybe an option in the Git config that says "ignore modified files" or some-such? It could be disabled by default to maintain backwards compatibility and yet could be enabled for the rest of us who want to have accurate information on the Jenkins UI.

          Kevin Phillips added a comment - markewaite haha - I'm guessing you are absolutely correct. However, if someone complains about a regression (aka: a change in previous behavior) when the previous behavior was incorrect is, well, questionable anyway. It's hard to maintain backwards compatibility AND fix incorrect behavior at the same time since the latter, by definition, changes the previous behavior. That being said, what if there were some way to control that at least... like maybe an option in the Git config that says "ignore modified files" or some-such? It could be disabled by default to maintain backwards compatibility and yet could be enabled for the rest of us who want to have accurate information on the Jenkins UI.

            Unassigned Unassigned
            fstlaure Frédérick St-Laurent
            Votes:
            4 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: