Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-26660

Add another '-f' to git clean to remove deleted submodules or other repos checked out in workspace in pipeline

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • None
    • Jenkins and multi SCM plugin configured with git repository using submodules
      Alternatively: A pipeline that checks out a second repository in a sub-dir
    • git-client-plugin 3.0.0, git-plugin 4.1.0

      Submodules clean up

      Jenkins SCM plugin can handle git repositories with submodules except for clean up.

      When "Clean before checkout" is selected, it runs 'git reset --hard && git clean -dxf' on the repository and submodules if "Recursively update submodules" is selected. This works for most stuff but it does not work if git submodule is removed from the tree which leaves the submodule directory in workspace even after 'git clean -dxf'.

      If another '-f' is added, also the removed submodule will be deleted which is the behavior I'd expect when git submodules are in use.

      So Jenkins git plugin should call 'git clean -dxf -f' in that case.

      Pipelines workspace reuse

      An alternative use case, which I believe is very common, is that a pipeline checks out a repository in a sub-directory. Since the workspaces are reused between builds, there is a big risk that code or generated files in that sub-directory from previous builds are still there in the next run. This can seriously impact reliability of build or even fail them. One case is that old cucumber test reports are found.

      Completely wiping the workspace has a performance cost that is unreasonable.

          [JENKINS-26660] Add another '-f' to git clean to remove deleted submodules or other repos checked out in workspace in pipeline

          Is there a current workaround until a fix for this has landed?

          I'm getting

          Command "git submodule init" returned status code 128:
          stdout: 
          stderr: fatal: No url found for submodule path 'mysubmodule' in .gitmodules
          

          after deleting a submodule and would like this to work without having to SSH to the Jenkins machine.

          Niklas Hambuechen added a comment - Is there a current workaround until a fix for this has landed? I'm getting Command "git submodule init" returned status code 128: stdout: stderr: fatal: No url found for submodule path 'mysubmodule' in .gitmodules after deleting a submodule and would like this to work without having to SSH to the Jenkins machine.

          Mark Waite added a comment -

          One alternative is to use "Wipe workspace" either within the job definition (so that it happens at the start of each job).

          Another alternative might be to add a build step as a "pre-SCM" build step.

          Mark Waite added a comment - One alternative is to use "Wipe workspace" either within the job definition (so that it happens at the start of each job). Another alternative might be to add a build step as a "pre-SCM" build step.

          Any updates on this? The fix seems quite simple and it's a really annoying bug. Thanks!

          Leandro Lucarella added a comment - Any updates on this? The fix seems quite simple and it's a really annoying bug. Thanks!

          Mark Waite added a comment -

          I'd love to have help evaluating the two different pull requests which propose to add an optional additional "f" to the git clean command. Refer to

          • PR227
          • PR222 where you said it would fix it for you, but did not indicate that you've deployed it

          Mark Waite added a comment - I'd love to have help evaluating the two different pull requests which propose to add an optional additional "f" to the git clean command. Refer to PR227 PR222 where you said it would fix it for you, but did not indicate that you've deployed it

          Sorry, I forgot about the duplicated issue. Since this only bite us very seldom, and it takes some time to figure out what the problem was, is hard to remember all the details about the issue.

          Is there any way to get an easily installable plugin that contains the fix to test it? Thanks!

          Leandro Lucarella added a comment - Sorry, I forgot about the duplicated issue. Since this only bite us very seldom, and it takes some time to figure out what the problem was, is hard to remember all the details about the issue. Is there any way to get an easily installable plugin that contains the fix to test it? Thanks!

          Code changed in jenkins
          User: Samantha Schwarz
          Path:
          src/main/java/hudson/plugins/git/GitAPI.java
          src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
          src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java
          src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
          src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java
          src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java
          http://jenkins-ci.org/commit/git-client-plugin/58a322534f70180f5cba17e8f09c0e40df7fd962
          Log:
          Clean takes a flag that will add a --force flag to clean.

          See bug report https://issues.jenkins-ci.org/browse/JENKINS-26660 for more details

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Samantha Schwarz Path: src/main/java/hudson/plugins/git/GitAPI.java src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java http://jenkins-ci.org/commit/git-client-plugin/58a322534f70180f5cba17e8f09c0e40df7fd962 Log: Clean takes a flag that will add a --force flag to clean. See bug report https://issues.jenkins-ci.org/browse/JENKINS-26660 for more details

          René Scheibe added a comment - - edited

          Current State

          René Scheibe added a comment - - edited Current State Step 1 is done. With git-client-plugin 3.0.0 https://github.com/jenkinsci/git-client-plugin/pull/222 is released now. Step 2 is nearly done. Using the new feature from git-client-plugin in git-plugin ( https://github.com/jenkinsci/git-plugin/pull/792 ). TODO: release

          Mark Waite added a comment -

          Built hpi of a pre-release of the plugin can be pulled from the master branch of the ci.jenkins.io job.

          Plan is to release git plugin 4.1 before Dec 31, 2019 with this fix included.

          Mark Waite added a comment - Built hpi of a pre-release of the plugin can be pulled from the master branch of the ci.jenkins.io job . Plan is to release git plugin 4.1 before Dec 31, 2019 with this fix included.

          Mark Waite added a comment -

          Fixed in git plugin 4.1.0, released Jan 18, 2020.

          Mark Waite added a comment - Fixed in git plugin 4.1.0, released Jan 18, 2020.

          René Scheibe added a comment -

          NOTE

          This change breaks backward compatibility regarding Job DSL for the methods cleanBeforeCheckoutTrait() and cleanAfterCheckoutTrait().

          The changes in the UI are backward compatible.

          old syntax

          multibranchPipelineJob('example-old') {
            branchSources {
              branchSource {
                source {
                  git {
                    id('example-old')
                    remote('https://github.com/jenkinsci/job-dsl-plugin.git')
                    traits {
                      cleanBeforeCheckoutTrait() // <--- fails with git-plugin v4.1.0
                      cleanAfterCheckoutTrait()  // <--- fails with git-plugin v4.1.0
                    }
                  }
                }
              }
            }
          }

          new syntax

          multibranchPipelineJob('example-new') {
            branchSources { 
              branchSource { 
                source { 
                  git { 
                    id('example-new')
                    remote('https://github.com/jenkinsci/job-dsl-plugin.git')
                    traits { 
                      cleanBeforeCheckoutTrait {
                        extension {
                          deleteUntrackedNestedRepositories(false)
                        }
                      }
                      cleanAfterCheckoutTrait {
                        extension {
                          deleteUntrackedNestedRepositories(false)
                        }
                      }
                    }
                  }
                }
              }
            }
          }

          René Scheibe added a comment - NOTE This change breaks backward compatibility regarding Job DSL for the methods cleanBeforeCheckoutTrait() and cleanAfterCheckoutTrait() . The changes in the UI are backward compatible. old syntax multibranchPipelineJob( 'example-old' ) { branchSources { branchSource { source { git { id( 'example-old' ) remote( 'https: //github.com/jenkinsci/job-dsl-plugin.git' ) traits { cleanBeforeCheckoutTrait() // <--- fails with git-plugin v4.1.0 cleanAfterCheckoutTrait() // <--- fails with git-plugin v4.1.0 } } } } } } new syntax multibranchPipelineJob( 'example- new ' ) { branchSources { branchSource { source { git { id( 'example- new ' ) remote( 'https: //github.com/jenkinsci/job-dsl-plugin.git' ) traits { cleanBeforeCheckoutTrait { extension { deleteUntrackedNestedRepositories( false ) } } cleanAfterCheckoutTrait { extension { deleteUntrackedNestedRepositories( false ) } } } } } } } }

            Unassigned Unassigned
            mcfrisk Mikko Rapeli
            Votes:
            23 Vote for this issue
            Watchers:
            31 Start watching this issue

              Created:
              Updated:
              Resolved: