Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-45771

Disable shallow clone when we know a merge will take place

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      There is an issue when building pull requests and the clone trait is present with shallow clone option.
      Apparently you need some history in order to be able to do a merge.

      We need to disable shallow clone when we know a merge will take place.

      At the first look, this can be fixed in
      com.cloudbees.jenkins.plugins.bitbucket.BitbucketGitSCMBuilder
      org.jenkinsci.plugins.github_branch_source.GitHubSCMBuilder

      I had identified the problem using bitbucket-branch-source-plugin but the should be visible on github-branch-source-plugin also..

        Attachments

          Issue Links

            Activity

            drealecs Alexandru Pătrănescu created issue -
            markewaite Mark Waite made changes -
            Field Original Value New Value
            Issue Type Bug [ 1 ] New Feature [ 2 ]
            markewaite Mark Waite made changes -
            Assignee Mark Waite [ markewaite ]
            stephenconnolly Stephen Connolly made changes -
            Description There is an issue when building pull requests and the clone trait is present with shallow clone option.
            Apparently you need some history in order to be able to do a merge.

            We need to disable shallow clone when we know a merge will take place.

            At the first look, this can be fixed in
            {{com.cloudbees.jenkins.plugins.bitbucket.BitbucketGitSCMBuilder}}
            {{org.jenkinsci.plugins.github_branch_source.GitHubSCMBuilder}}

            I had identified the problem using bitbucket-branch-source-plugin but the should be visible on github-branch-source-plugin also.
            There is an issue when building pull requests and the clone trait is present with shallow clone option.
             Apparently you need some history in order to be able to do a merge.

            We need to disable shallow clone when we know a merge will take place.

            At the first look, this can be fixed in
             {{com.cloudbees.jenkins.plugins.bitbucket.BitbucketGitSCMBuilder}}
             {{org.jenkinsci.plugins.github_branch_source.GitHubSCMBuilder}}

            I had identified the problem using bitbucket-branch-source-plugin but the should be visible on github-branch-source-plugin also..
            stephenconnolly Stephen Connolly made changes -
            Remote Link This issue links to "git-client-plugin#255 (Web Link)" [ 17335 ]
            stephenconnolly Stephen Connolly made changes -
            Remote Link This issue links to "git-plugin#516 (Web Link)" [ 17336 ]
            stephenconnolly Stephen Connolly made changes -
            Assignee Stephen Connolly [ stephenconnolly ]
            stephenconnolly Stephen Connolly made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            stephenconnolly Stephen Connolly made changes -
            Remote Link This issue links to "gitea-plugin#4 (Web Link)" [ 17337 ]
            stephenconnolly Stephen Connolly made changes -
            Remote Link This issue links to "bitbucket-branch-source-plugin#64 (Web Link)" [ 17338 ]
            stephenconnolly Stephen Connolly made changes -
            Remote Link This issue links to "github-branch-source-plugin#150 (Web Link)" [ 17339 ]
            stephenconnolly Stephen Connolly made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            stephenconnolly Stephen Connolly made changes -
            Issue Type New Feature [ 2 ] Bug [ 1 ]
            stephenconnolly Stephen Connolly made changes -
            Link This issue blocks JENKINS-44598 [ JENKINS-44598 ]
            Hide
            markewaite Mark Waite added a comment - - edited

            Since the "shallowness" of a git repository is captured in the local git repository and the --shallow and --depth x arguments to command line git only set the configuration for future command executions, the git client internals may need to be extended to include some form of check to switch a repository from "shallow" to "unshallow".

            Some of the complexities of converting from shallow to unshallow are described in stackoverflow.

            Show
            markewaite Mark Waite added a comment - - edited Since the "shallowness" of a git repository is captured in the local git repository and the --shallow and --depth x arguments to command line git only set the configuration for future command executions, the git client internals may need to be extended to include some form of check to switch a repository from "shallow" to "unshallow". Some of the complexities of converting from shallow to unshallow are described in stackoverflow .
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Mark Waite my fix sets the shallowness from the start... a PR-merge will always be non-shallow and not chop and change back and forth

            Show
            stephenconnolly Stephen Connolly added a comment - Mark Waite my fix sets the shallowness from the start... a PR-merge will always be non-shallow and not chop and change back and forth
            Hide
            markewaite Mark Waite added a comment -

            Stephen Connolly that sounds great to me. The general case of a user toggling between shallow and not shallow will continue to be an open issue, and the easy work around for that case is to wipe the workspace.

            Show
            markewaite Mark Waite added a comment - Stephen Connolly that sounds great to me. The general case of a user toggling between shallow and not shallow will continue to be an open issue, and the easy work around for that case is to wipe the workspace.
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/CloneCommand.java
            src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/PushCommand.java
            src/main/java/org/jenkinsci/plugins/gitclient/RevListCommand.java
            http://jenkins-ci.org/commit/git-client-plugin/d2243416d262ac9b442945d05cc2cf827e5e5373
            Log:
            JENKINS-45771 Ensure all command boolean options can be toggled, not just set once

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: pom.xml src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/CloneCommand.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/PushCommand.java src/main/java/org/jenkinsci/plugins/gitclient/RevListCommand.java http://jenkins-ci.org/commit/git-client-plugin/d2243416d262ac9b442945d05cc2cf827e5e5373 Log: JENKINS-45771 Ensure all command boolean options can be toggled, not just set once
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
            http://jenkins-ci.org/commit/git-client-plugin/1a7d53e90e23c6184b9b54003ef0d1b6379f8c30
            Log:
            JENKINS-45771 Code review catches bugs (who knew)

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java http://jenkins-ci.org/commit/git-client-plugin/1a7d53e90e23c6184b9b54003ef0d1b6379f8c30 Log: JENKINS-45771 Code review catches bugs (who knew)
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/FetchCommand.java
            src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
            http://jenkins-ci.org/commit/git-client-plugin/f445161be299b3cf0d54e893cfa7df39bb0058ee
            Log:
            JENKINS-45771 Missed one boolean builder

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/FetchCommand.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java http://jenkins-ci.org/commit/git-client-plugin/f445161be299b3cf0d54e893cfa7df39bb0058ee Log: JENKINS-45771 Missed one boolean builder
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Mark Waite
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/CloneCommand.java
            src/main/java/org/jenkinsci/plugins/gitclient/FetchCommand.java
            src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
            src/main/java/org/jenkinsci/plugins/gitclient/PushCommand.java
            src/main/java/org/jenkinsci/plugins/gitclient/RevListCommand.java
            http://jenkins-ci.org/commit/git-client-plugin/d7ed207c99feecb5d0fc854e05ff0c3df2cd275e
            Log:
            Merge pull request #255 from stephenc/jenkins-45771

            JENKINS-45771 Ensure all command boolean options can be toggled, not just set once

            Compare: https://github.com/jenkinsci/git-client-plugin/compare/b0fa8071bdc1...d7ed207c99fe

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Mark Waite Path: pom.xml src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/CloneCommand.java src/main/java/org/jenkinsci/plugins/gitclient/FetchCommand.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/PushCommand.java src/main/java/org/jenkinsci/plugins/gitclient/RevListCommand.java http://jenkins-ci.org/commit/git-client-plugin/d7ed207c99feecb5d0fc854e05ff0c3df2cd275e Log: Merge pull request #255 from stephenc/jenkins-45771 JENKINS-45771 Ensure all command boolean options can be toggled, not just set once Compare: https://github.com/jenkinsci/git-client-plugin/compare/b0fa8071bdc1...d7ed207c99fe
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            pom.xml
            src/main/java/jenkins/plugins/git/MergeWithGitSCMExtension.java
            http://jenkins-ci.org/commit/git-plugin/b8c90a9461dc57633d97490d421a52796b0be2bb
            Log:
            [FIXED JENKINS-45771] Consolidate the MergeWithGitSCMExtension into git plugin

            With the update to git-client we can now fix the shallow clone issue on merge with

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: pom.xml src/main/java/jenkins/plugins/git/MergeWithGitSCMExtension.java http://jenkins-ci.org/commit/git-plugin/b8c90a9461dc57633d97490d421a52796b0be2bb Log: [FIXED JENKINS-45771] Consolidate the MergeWithGitSCMExtension into git plugin With the update to git-client we can now fix the shallow clone issue on merge with
            stephenconnolly Stephen Connolly made changes -
            Remote Link This issue links to "Page (Jenkins Wiki)" [ 17358 ]
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugin/gitea/GiteaSCMBuilder.java
            src/main/java/org/jenkinsci/plugin/gitea/MergeWithGitSCMExtension.java
            http://jenkins-ci.org/commit/gitea-plugin/ed08e75e55f227e7d001dfd84a234fd477c7b831
            Log:
            JENKINS-45771 Pick up consolidated MergeWithGitSCMExtension from git plugin

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: pom.xml src/main/java/org/jenkinsci/plugin/gitea/GiteaSCMBuilder.java src/main/java/org/jenkinsci/plugin/gitea/MergeWithGitSCMExtension.java http://jenkins-ci.org/commit/gitea-plugin/ed08e75e55f227e7d001dfd84a234fd477c7b831 Log: JENKINS-45771 Pick up consolidated MergeWithGitSCMExtension from git plugin
            Hide
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Stephen Connolly
            Path:
            pom.xml
            src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMBuilder.java
            src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java
            src/main/java/org/jenkinsci/plugins/github_branch_source/MergeWithGitSCMExtension.java
            src/test/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMBuilderTest.java
            http://jenkins-ci.org/commit/github-branch-source-plugin/3256676fa6a47d342a0442c59555df9eecde0b68
            Log:
            JENKINS-45771 Pick up consolidated MergeWithGitSCMExtension from git plugin

            Show
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Stephen Connolly Path: pom.xml src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMBuilder.java src/main/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMSource.java src/main/java/org/jenkinsci/plugins/github_branch_source/MergeWithGitSCMExtension.java src/test/java/org/jenkinsci/plugins/github_branch_source/GitHubSCMBuilderTest.java http://jenkins-ci.org/commit/github-branch-source-plugin/3256676fa6a47d342a0442c59555df9eecde0b68 Log: JENKINS-45771 Pick up consolidated MergeWithGitSCMExtension from git plugin
            stephenconnolly Stephen Connolly made changes -
            Remote Link This issue links to "Page (Jenkins Wiki)" [ 17359 ]
            stephenconnolly Stephen Connolly made changes -
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Resolved [ 5 ]
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Required plugin updates:

            • git-client 2.5.0
            • git 3.5.0
            • At least one of the optional plugin updates

            Optional plugin updates: (update any of these if already have installed, do not need to install a plugin if not installed already)

            • github-branch-source 2.2.3
            • gitea 1.0.1
            • bitbucket-branch-source 2.2.3
            Show
            stephenconnolly Stephen Connolly added a comment - Required plugin updates: git-client 2.5.0 git 3.5.0 At least one of the optional plugin updates Optional plugin updates: (update any of these if already have installed, do not need to install a plugin if not installed already) github-branch-source 2.2.3 gitea 1.0.1 bitbucket-branch-source 2.2.3
            stephenconnolly Stephen Connolly made changes -
            Remote Link This issue links to "Page (Jenkins Wiki)" [ 17360 ]
            stephenconnolly Stephen Connolly made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            Hide
            shiqiyang Shiqi Yang added a comment - - edited

            I found a case where this bug still applies in parallel stages, reopen this one since I think it is the same issue but please let me know if it is not and I can open another one

            environment:
            jenkins 2.89.3
            github-branch-source-plugin 2.32.2

            I have a multibranch project setup with
            `discover the pull requests from origin`
            `merge pull request with target`
            fetch all tags
            shallow clone

            I also have Basic Branch Build Strategies Plugin installed to stop the build storm when we scan repo but I don't think it is related

            my pipeline file has 2 parallel stages and one of the stages has its own node {} section to specify the builder label and workspace path.
            it looks like this (this is almost the same example of https://jenkins.io/blog/2017/09/25/declarative-1, only difference here is that I have customWorkspace specified on the nodes to reuse the local code repo in the hope to save some time on git clone, our git repo is huge):

            pipeline {
                 agent none
                stages {
                    stage('Run Tests') {
                        parallel {
                            stage('Test 1') {
                                agent {
                                    label "windows"
                                    customWorkspace "${env.HOME}/TEST1"
                                }
                                steps {
                                    bat "run-tests.bat"
                                }
                                post {
                                    always {
                                        junit "**/TEST-*.xml"
                                    }
                                }
                            }
                            stage('Test On Linux') {
                                agent {
                                    label "linux"
                                    customWorkspace "${env.HOME}/TEST2"
                                }
                                steps {
                                    sh "run-tests.sh"
                                }
                                post {
                                    always {
                                        junit "**/TEST-*.xml"
                                    }
                                }
                            }
                        }
                    }
                }
            }
            

            the error I am seeing is that when the pipeline enters the parallel stages, especially after the workspace switch, it will fail to merge if the PR is couple of commits behind master because the head of master is checked out without git history. ( I logged in to the builder after the failure and manually verified that in the workspace)

            it seems that removing two ways to resolve this for now:
            1 remove the the customWorkspace this is not going to work well for us because the lack of self clean on the branches of multibranch (without customworkspace different branch runs will not share workspace in current design of multibranch pipeline)
            2. add 'wipe out workspace and clone again in advanced' settings. (this will increase our build time tremendously because our github repo is big, and unnecessary)

            > /usr/local/bin/git rev-parse --is-inside-work-tree # timeout=10
            
            Fetching changes from the remote Git repository
            
             > /usr/local/bin/git config remote.origin.url !INFO_EXTRACTED! # timeout=10
            
            Fetching upstream changes from !INFO_EXTRACTED!
            
             > /usr/local/bin/git --version # timeout=10
            
            using GIT_ASKPASS to set credentials JENKINS_GH
            
             > /usr/local/bin/git fetch --tags --progress !INFO_EXTRACTED! +refs/pull/!INFO_EXTRACTED!/head:refs/remotes/origin/PR-!INFO_EXTRACTED! +refs/heads/master:refs/remotes/origin/master # timeout=60
            
            Merging remotes/origin/master commit !INFO_EXTRACTED! into PR head commit !INFO_EXTRACTED!
            
             > /usr/local/bin/git config core.sparsecheckout # timeout=10
            
             > /usr/local/bin/git checkout -f !INFO_EXTRACTED!
            
             > /usr/local/bin/git merge !INFO_EXTRACTED! # timeout=10
            
             > /usr/local/bin/git config core.sparsecheckout # timeout=10
            
             > /usr/local/bin/git checkout -f !INFO_EXTRACTED!
            
            Command "/usr/local/bin/git merge !INFO_EXTRACTED!" returned status code 128:
            
            stdout: 
            
            stderr: fatal: refusing to merge unrelated histories
            

            FYI I remove the sensitive date from the output and replaced with ('!INFO_EXTRACTED')

            Show
            shiqiyang Shiqi Yang added a comment - - edited I found a case where this bug still applies in parallel stages, reopen this one since I think it is the same issue but please let me know if it is not and I can open another one environment: jenkins 2.89.3 github-branch-source-plugin 2.32.2 I have a multibranch project setup with `discover the pull requests from origin` `merge pull request with target` fetch all tags shallow clone I also have Basic Branch Build Strategies Plugin installed to stop the build storm when we scan repo but I don't think it is related my pipeline file has 2 parallel stages and one of the stages has its own node {} section to specify the builder label and workspace path. it looks like this (this is almost the same example of https://jenkins.io/blog/2017/09/25/declarative-1 , only difference here is that I have customWorkspace specified on the nodes to reuse the local code repo in the hope to save some time on git clone, our git repo is huge): pipeline { agent none stages { stage( 'Run Tests' ) { parallel { stage( 'Test 1' ) { agent { label "windows" customWorkspace "${env.HOME}/TEST1" } steps { bat "run-tests.bat" } post { always { junit "**/TEST-*.xml" } } } stage( 'Test On Linux' ) { agent { label "linux" customWorkspace "${env.HOME}/TEST2" } steps { sh "run-tests.sh" } post { always { junit "**/TEST-*.xml" } } } } } } } the error I am seeing is that when the pipeline enters the parallel stages, especially after the workspace switch, it will fail to merge if the PR is couple of commits behind master because the head of master is checked out without git history. ( I logged in to the builder after the failure and manually verified that in the workspace) it seems that removing two ways to resolve this for now: 1 remove the the customWorkspace this is not going to work well for us because the lack of self clean on the branches of multibranch (without customworkspace different branch runs will not share workspace in current design of multibranch pipeline) 2. add 'wipe out workspace and clone again in advanced' settings. (this will increase our build time tremendously because our github repo is big, and unnecessary) > /usr/local/bin/git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository > /usr/local/bin/git config remote.origin.url !INFO_EXTRACTED! # timeout=10 Fetching upstream changes from !INFO_EXTRACTED! > /usr/local/bin/git --version # timeout=10 using GIT_ASKPASS to set credentials JENKINS_GH > /usr/local/bin/git fetch --tags --progress !INFO_EXTRACTED! +refs/pull/!INFO_EXTRACTED!/head:refs/remotes/origin/PR-!INFO_EXTRACTED! +refs/heads/master:refs/remotes/origin/master # timeout=60 Merging remotes/origin/master commit !INFO_EXTRACTED! into PR head commit !INFO_EXTRACTED! > /usr/local/bin/git config core.sparsecheckout # timeout=10 > /usr/local/bin/git checkout -f !INFO_EXTRACTED! > /usr/local/bin/git merge !INFO_EXTRACTED! # timeout=10 > /usr/local/bin/git config core.sparsecheckout # timeout=10 > /usr/local/bin/git checkout -f !INFO_EXTRACTED! Command "/usr/local/bin/git merge !INFO_EXTRACTED!" returned status code 128: stdout: stderr: fatal: refusing to merge unrelated histories FYI I remove the sensitive date from the output and replaced with ('!INFO_EXTRACTED')
            shiqiyang Shiqi Yang made changes -
            Resolution Fixed [ 1 ]
            Status Closed [ 6 ] Reopened [ 4 ]
            Hide
            stephenconnolly Stephen Connolly added a comment -

            Removing myself as assignee. My current work assignments do not provide sufficient bandwidth to review these issues and in the majority of cases I am only assigned by virtue of being the default assignee. For the credentials-api and scm-api related plugins I have permission to allocate time reviewing changes to these APIs themselves to ensure these APIs remain cohesive, but that can be handled through PR reviews rather than assigning issues in JIRA

            Show
            stephenconnolly Stephen Connolly added a comment - Removing myself as assignee. My current work assignments do not provide sufficient bandwidth to review these issues and in the majority of cases I am only assigned by virtue of being the default assignee. For the credentials-api and scm-api related plugins I have permission to allocate time reviewing changes to these APIs themselves to ensure these APIs remain cohesive, but that can be handled through PR reviews rather than assigning issues in JIRA
            stephenconnolly Stephen Connolly made changes -
            Assignee Stephen Connolly [ stephenconnolly ]
            markewaite Mark Waite made changes -
            Status Reopened [ 4 ] Open [ 1 ]

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              drealecs Alexandru Pătrănescu
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated: