Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45586

Recent Changes not updated when using GIT Shallow clone option

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • Jenkins 2.60.1 LTS, all latest plugins on July 17 2017.
      Jenkins server running Cent OS and Client running on a Windows Server 2012R2 VM.

      In a Multibranch job, if the Jenkins File contains the following:

      checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\{env.BRANCH_NAME}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: +*true*+]], submoduleCfg: [], userRemoteConfigs: [url: '[http://My]]

      The "Recent Changes" of the branch will not update.  Previous version (LTS 2.32.1) with old plugins did not had this problem.

      Changing the checkout to:

      checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\{env.BRANCH_NAME}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: +*false*+]], submoduleCfg: [], userRemoteConfigs: [url: '[http://My]]

      allows to get the list of "Recent Changes" populated again, but at the cost of a considerably higher build time and stress on the GIT server as the repository need to be cloned from scratch every time (as we are running agents on VMs that are always started from the same snapshot).

       

          [JENKINS-45586] Recent Changes not updated when using GIT Shallow clone option

          Frédérick St-Laurent created issue -
          Frédérick St-Laurent made changes -
          Description Original: In a Multibranch job, if the Jenkins File contains the following:

          {{checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\{env.BRANCH_NAME}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: +*true*+]], submoduleCfg: [], userRemoteConfigs: [[url: '[http://My|http://my/]BitbucketServer/someting/else/project.git']]]}}

          The "Recent Changes" of the branch will not update.  Previous version (LTS 2.32.1) with old plugins did not had this problem.

          Changing the checkout to:

          {{checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\{env.BRANCH_NAME}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: +*false*+]], submoduleCfg: [], userRemoteConfigs: [[url: '[http://My|http://my/]BitbucketServer/someting/else/project.git']]]}}

          allows to get the list of "Recent Changes" populated again, but at the cost of a considerably higher build time and stress on the GIT server as the repository need to be cloned from scratch every time (as we are running on VM that are always started from the same snapshot).

           
          New: In a Multibranch job, if the Jenkins File contains the following:

          {{checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\\{env.BRANCH_NAME}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: +*true*+]], submoduleCfg: [], userRemoteConfigs: [[url: '[http://My|http://my/]BitbucketServer/someting/else/project.git']]]}}

          The "Recent Changes" of the branch will not update.  Previous version (LTS 2.32.1) with old plugins did not had this problem.

          Changing the checkout to:

          {{checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\\{env.BRANCH_NAME}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: +*false*+]], submoduleCfg: [], userRemoteConfigs: [[url: '[http://My|http://my/]BitbucketServer/someting/else/project.git']]]}}

          allows to get the list of "Recent Changes" populated again, but at the cost of a considerably higher build time and stress on the GIT server as the repository need to be cloned from scratch every time (as we are running agents on VMs that are always started from the same snapshot).

           
          Frédérick St-Laurent made changes -
          Description Original: In a Multibranch job, if the Jenkins File contains the following:

          {{checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\\{env.BRANCH_NAME}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: +*true*+]], submoduleCfg: [], userRemoteConfigs: [[url: '[http://My|http://my/]BitbucketServer/someting/else/project.git']]]}}

          The "Recent Changes" of the branch will not update.  Previous version (LTS 2.32.1) with old plugins did not had this problem.

          Changing the checkout to:

          {{checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\\{env.BRANCH_NAME}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: +*false*+]], submoduleCfg: [], userRemoteConfigs: [[url: '[http://My|http://my/]BitbucketServer/someting/else/project.git']]]}}

          allows to get the list of "Recent Changes" populated again, but at the cost of a considerably higher build time and stress on the GIT server as the repository need to be cloned from scratch every time (as we are running agents on VMs that are always started from the same snapshot).

           
          New: In a Multibranch job, if the Jenkins File contains the following:

          \{\{checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\{env.BRANCH_NAME}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: +*true*+]], submoduleCfg: [], userRemoteConfigs: [[url: '[http://My|http://my/]BitbucketServer/someting/else/project.git']]]}}
           
           The "Recent Changes" of the branch will not update.  Previous version (LTS 2.32.1) with old plugins did not had this problem.
           
           Changing the checkout to:
           
           \{\{checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\{env.BRANCH_NAME}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: +*false*+]], submoduleCfg: [], userRemoteConfigs: [[url: '[http://My|http://my/]BitbucketServer/someting/else/project.git']]]}}

          allows to get the list of "Recent Changes" populated again, but at the cost of a considerably higher build time and stress on the GIT server as the repository need to be cloned from scratch every time (as we are running agents on VMs that are always started from the same snapshot).

           

          Mark Waite added a comment -

          Since a shallow clone may only clone the most recent change, it seems (at least to me) that the change list may be incorrect for a shallow clone that follows another shallow clone. If a shallow clone is followed by 2 or more commits to the repository, then another shallow clone is performed, the commits in the working repository may only be the original commit and the most recent commit in the repository.

          You might try reducing load on your git server by using a narrow refspec instead of a shallow clone. A narrow refspec allows the git server to only return the history of a single branch, while a shallow clone allows the server to return the history of a limited number of commits for each branch that matches the refspec. The refspec setting is modified in the "Advanced" section of the repository definition.

          Mark Waite added a comment - Since a shallow clone may only clone the most recent change, it seems (at least to me) that the change list may be incorrect for a shallow clone that follows another shallow clone. If a shallow clone is followed by 2 or more commits to the repository, then another shallow clone is performed, the commits in the working repository may only be the original commit and the most recent commit in the repository. You might try reducing load on your git server by using a narrow refspec instead of a shallow clone. A narrow refspec allows the git server to only return the history of a single branch, while a shallow clone allows the server to return the history of a limited number of commits for each branch that matches the refspec. The refspec setting is modified in the "Advanced" section of the repository definition.
          Mark Waite made changes -
          Component/s Original: git-changelog-plugin [ 20855 ]
          Mark Waite made changes -
          Assignee Original: Paul Wellner Bou [ paulwellnerbou ]

          This is a repo with many years of commits (and all commit being merged to master).  So a shallow copy (with maybe a depth of 5) allow us to copy much less data.  I know that many commits between build could end up in the list of "Recent Changes" missing some entries, but this is a price we are ready to pay to reduce build time.

          We will however experiment with refspec in the mean time.

          Frédérick St-Laurent added a comment - This is a repo with many years of commits (and all commit being merged to master).  So a shallow copy (with maybe a depth of 5) allow us to copy much less data.  I know that many commits between build could end up in the list of "Recent Changes" missing some entries, but this is a price we are ready to pay to reduce build time. We will however experiment with refspec in the mean time.

          Frédérick St-Laurent added a comment - - edited

          I look like refspec will not be "totally respected" on the first checkout.  With the following script:

          node( "MyWindowsSlave" )
          {
          def branch = "bugfix/FPP-1229-use-refspec"
          def myRepo = "http://mtlstash.miranda.com:7990/scm/fpp/fpp.git"
          checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [[name: "$\{branch\}"]], doGenerateSubmoduleConfigurations: false, extensions: [[$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: false]], submoduleCfg: [], userRemoteConfigs: [[refspec: "+refs/heads/$\{branch\}:refs/remotes/origin/$\{branch\}", url: myRepo]]]
          }

          The first checkout will give me something that looks like:

          Cloning the remote Git repository
          Avoid fetching tags
          Cloning repository http://myserver/myrepo.git
          > git init d:\Jenkins\Workspaces\workspace\TestFred # timeout=10
          Fetching upstream changes from http://myserver/myrepo.git
          > git --version # timeout=10
          > git fetch --no-tags --progress http://myserver/myrepo.git +refs/heads/:refs/remotes/origin/**
          > git config remote.origin.url http://myserver/myrepo.git # timeout=10
          > git config --add remote.origin.fetch +refs/heads/:refs/remotes/origin/ # timeout=10
          > git config remote.origin.url http://myserver/myrepo.git # timeout=10
          Fetching upstream changes from http://myserver/myrepo.git
          > git fetch --no-tags --progress http://myserver/myrepo.git +refs/heads/bugfix/FPP-1229-use-refspec:refs/remotes/origin/bugfix/FPP-1229-use-refspec

           

          As we can see, git fetch is called twice, the first time without my refspec.  However, if I run the job again without clearing my workspace I get:

          > git rev-parse --is-inside-work-tree # timeout=10
          Fetching changes from the remote Git repository
          > git config remote.origin.url http://myserver/myrepo.git # timeout=10
          Fetching upstream changes from http://myserver/myrepo.git
          > git --version # timeout=10
          > git fetch --no-tags --progress http://myserver/myrepo.git +refs/heads/bugfix/FPP-1229-use-refspec:refs/remotes/origin/bugfix/FPP-1229-use-refspec

           

          However, we cannot keep the workspace as each of our build normally run from a cleared VM snapshot.

          Frédérick St-Laurent added a comment - - edited I look like refspec will not be "totally respected" on the first checkout.  With the following script: node( "MyWindowsSlave" ) { def branch = "bugfix/FPP-1229-use-refspec" def myRepo = "http://mtlstash.miranda.com:7990/scm/fpp/fpp.git" checkout changelog: true, poll: false, scm: [$class: 'GitSCM', branches: [ [name: "$\{branch\}"] ], doGenerateSubmoduleConfigurations: false, extensions: [ [$class: 'CloneOption', depth: 0, noTags: true, reference: '', shallow: false] ], submoduleCfg: [], userRemoteConfigs: [ [refspec: "+refs/heads/$\{branch\}:refs/remotes/origin/$\{branch\}", url: myRepo] ]] } The first checkout will give me something that looks like: Cloning the remote Git repository Avoid fetching tags Cloning repository http://myserver/myrepo.git > git init d:\Jenkins\Workspaces\workspace\TestFred # timeout=10 Fetching upstream changes from http://myserver/myrepo.git > git --version # timeout=10 > git fetch --no-tags --progress http://myserver/myrepo.git +refs/heads/ :refs/remotes/origin/** > git config remote.origin.url http://myserver/myrepo.git # timeout=10 > git config --add remote.origin.fetch +refs/heads/ :refs/remotes/origin/ # timeout=10 > git config remote.origin.url http://myserver/myrepo.git # timeout=10 Fetching upstream changes from http://myserver/myrepo.git > git fetch --no-tags --progress http://myserver/myrepo.git +refs/heads/bugfix/FPP-1229-use-refspec:refs/remotes/origin/bugfix/FPP-1229-use-refspec   As we can see, git fetch is called twice, the first time without my refspec.  However, if I run the job again without clearing my workspace I get: > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > git config remote.origin.url http://myserver/myrepo.git # timeout=10 Fetching upstream changes from http://myserver/myrepo.git > git --version # timeout=10 > git fetch --no-tags --progress http://myserver/myrepo.git +refs/heads/bugfix/FPP-1229-use-refspec:refs/remotes/origin/bugfix/FPP-1229-use-refspec   However, we cannot keep the workspace as each of our build normally run from a cleared VM snapshot.

          Mark Waite added a comment -

          you need to add the additional behaviour to honor refspec on checkout.

          Mark Waite added a comment - you need to add the additional behaviour to honor refspec on checkout.

          Adding "honorRefspec: true" in the clone options fixed the problem, however as expected there is virtually no gain when doing a clone of the master branch (but some gain when doing a checkout of older branches, so it proves it is working).

          Frédérick St-Laurent added a comment - Adding "honorRefspec: true" in the clone options fixed the problem, however as expected there is virtually no gain when doing a clone of the master branch (but some gain when doing a checkout of older branches, so it proves it is working).

            Unassigned Unassigned
            fstlaure Frédérick St-Laurent
            Votes:
            4 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated: