Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-10131

Git polling shouldn't need a workspace on a slave.

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Major Major
    • git-plugin
    • None

      What happened:

      I have two slaves (1 & 2). I took slave 1 down for maintenance and a bunch of old, rarely updated, git builds kicked off. When I checked the git polling log, I saw a message (it's gone now, darn it) that said it was rebuilding to get workspace for git polling.

      A workspace shouldn't be needed.

      I'm unclear what git needs for this, but if you're tracking only a single branch (master) then you just need the HEAD and can compare the SHA1s.

      If it technically really really needs a git checkout, then I'd prefer if they were kept on an assigned host (the jenkins server in my case, master) instead of using the workspaces for this check. I'd want this for a couple reasons:

      • Slaves come and go. Rebuilding all my projects because a slave went down is unproductive.
      • Occasionally, I need to go into a slave and monkey with a workspace to troubleshoot a weird problem. I don't want that to impact polling.
      • It's a waste of space on the slaves. I'd rather control where the space is wasted.

      Ciao!

          [JENKINS-10131] Git polling shouldn't need a workspace on a slave.

          jake bishop added a comment - - edited

          Any news on this? I can turn on fast remote polling to get around this. But that is not really a solution because then I can only use one branch (without wildcards).

          jake bishop added a comment - - edited Any news on this? I can turn on fast remote polling to get around this. But that is not really a solution because then I can only use one branch (without wildcards).

          ls-remote is the default on 2.0 for "single branch" case.
          investigating how to use for general use

          Nicolas De Loof added a comment - ls-remote is the default on 2.0 for "single branch" case. investigating how to use for general use

          Code changed in jenkins
          User: Nicolas De Loof
          Path:
          src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java
          src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java
          src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java
          src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java
          src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java
          http://jenkins-ci.org/commit/git-client-plugin/649c08617ff01191028e2e744013ed64e8a14431
          Log:
          JENKINS-10131 ls-remote for general usage to lookup all branches

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Nicolas De Loof Path: src/main/java/org/jenkinsci/plugins/gitclient/CliGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/GitClient.java src/main/java/org/jenkinsci/plugins/gitclient/JGitAPIImpl.java src/main/java/org/jenkinsci/plugins/gitclient/RemoteGitImpl.java src/test/java/org/jenkinsci/plugins/gitclient/GitAPITestCase.java http://jenkins-ci.org/commit/git-client-plugin/649c08617ff01191028e2e744013ed64e8a14431 Log: JENKINS-10131 ls-remote for general usage to lookup all branches

          Marc Günther added a comment -

          I don't agree that this is an improvement. It's a genuine bug, and at the moment there is not even a real workaround for it.

          For the multiple branch case, we ended up using a conditional buildstep, which compares GIT_COMMIT and GIT_PREVIOUS_COMMIT. But then you can't do manual builds anymore, because there is no way to find out the build cause inside of a build....

          Marc Günther added a comment - I don't agree that this is an improvement. It's a genuine bug, and at the moment there is not even a real workaround for it. For the multiple branch case, we ended up using a conditional buildstep, which compares GIT_COMMIT and GIT_PREVIOUS_COMMIT. But then you can't do manual builds anymore, because there is no way to find out the build cause inside of a build....

          Owen Jacobson added a comment -

          It's Worse Than You Think.

          The ls-remote polling, as written, makes a hard build/no-build decision based on the git ls-remote output only. This interacts ... poorly with the GitSCM plugin features that filter commits (UserExclusion and so on), since they need to look at the individual commits to make their own go/no-go decisions - but that can't be done remotely.

          Forcing workspace polling fixes that (and I've got a related PR open for JENKINS-20607 anyways) but reintroduces the problem Christian et al are complaining about. I suspect the real solution is to always do the polling on the master, possibly outside the workspace, and have the build runs re-fetch and check out the correct commit.

          From working on JENKINS-20607 I'm well aware of how much work that is.

          Owen Jacobson added a comment - It's Worse Than You Think. The ls-remote polling, as written, makes a hard build/no-build decision based on the git ls-remote output only. This interacts ... poorly with the GitSCM plugin features that filter commits ( UserExclusion and so on), since they need to look at the individual commits to make their own go/no-go decisions - but that can't be done remotely. Forcing workspace polling fixes that (and I've got a related PR open for JENKINS-20607 anyways) but reintroduces the problem Christian et al are complaining about. I suspect the real solution is to always do the polling on the master, possibly outside the workspace, and have the build runs re-fetch and check out the correct commit. From working on JENKINS-20607 I'm well aware of how much work that is.

          Mark Waite added a comment -

          The 1.6.2 release of git-client-plugin fixed a bug in ls-remote polling for the command line git implementation. The bug would have manifest itself as having ls-remote based polling show that there were changes when there were no changes.

          Refer to https://github.com/jenkinsci/git-client-plugin/commit/e5e75d5a100905c0d5b332bbde3642e96ffe10a9 for more details on the bug.

          That fix does not change the behavior described by Owen Jacobson. Remote polling does not apply commit filtering, so if you're using inclusion or exclusion filters in your job definition, you can't use remote polling.

          Mark Waite added a comment - The 1.6.2 release of git-client-plugin fixed a bug in ls-remote polling for the command line git implementation. The bug would have manifest itself as having ls-remote based polling show that there were changes when there were no changes. Refer to https://github.com/jenkinsci/git-client-plugin/commit/e5e75d5a100905c0d5b332bbde3642e96ffe10a9 for more details on the bug. That fix does not change the behavior described by Owen Jacobson. Remote polling does not apply commit filtering, so if you're using inclusion or exclusion filters in your job definition, you can't use remote polling.

          Don Ross added a comment -

          Now that the default behavior of the plug-in is to use remote polling, does this mean we need to not use exclusion filters?

          Don Ross added a comment - Now that the default behavior of the plug-in is to use remote polling, does this mean we need to not use exclusion filters?

          Mark Waite added a comment - - edited

          drdt you may continue to use exclusion filters and inclusion filters, but jobs which use them must be configured to "Force polling using workspace".

          Effectively, you're providing the compatibility with past behavior by forcing polling to use the workspace. Had you enabled fast remote polling with inclusion or exclusion filtering with git plugin releases prior to 2.0, you would have seen the same types of problems as exist today. The remote polling technique does not receive enough information to evaluate inclusion or exclusion filters.

          Mark Waite added a comment - - edited drdt you may continue to use exclusion filters and inclusion filters, but jobs which use them must be configured to "Force polling using workspace". Effectively, you're providing the compatibility with past behavior by forcing polling to use the workspace. Had you enabled fast remote polling with inclusion or exclusion filtering with git plugin releases prior to 2.0, you would have seen the same types of problems as exist today. The remote polling technique does not receive enough information to evaluate inclusion or exclusion filters.

          Don Ross added a comment -

          Thanks, Mark. I actually never got exclusions to work in prior versions, so I wanted to try them out now I upgraded.

          I am having trouble getting forced workspace polling to work at any rate; see my comments on https://issues.jenkins-ci.org/browse/JENKINS-20427.

          Don Ross added a comment - Thanks, Mark. I actually never got exclusions to work in prior versions, so I wanted to try them out now I upgraded. I am having trouble getting forced workspace polling to work at any rate; see my comments on https://issues.jenkins-ci.org/browse/JENKINS-20427 .

          Mark Waite added a comment -

          As described in the bug report, this is resolved. The git plugin does not require a workspace for polling, and the default (as of git plugin 2.0) is to not use a workspace for polling.

          The problems described by Owen Jacobson (and others) still exist for the somewhat more advanced use cases like excluding users, excluding files, and including files, but the basic issue reported by the bug is fixed.

          Mark Waite added a comment - As described in the bug report, this is resolved. The git plugin does not require a workspace for polling, and the default (as of git plugin 2.0) is to not use a workspace for polling. The problems described by Owen Jacobson (and others) still exist for the somewhat more advanced use cases like excluding users, excluding files, and including files, but the basic issue reported by the bug is fixed.

            abayer Andrew Bayer
            docwhat Christian Höltje
            Votes:
            22 Vote for this issue
            Watchers:
            27 Start watching this issue

              Created:
              Updated:
              Resolved: