Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20569

Polling with excluded regions don't prevent job triggered

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • git-plugin
    • None
    • Git plugin 2.0

      I've three jobs that trigger git with the same repository and branch.
      In then Excluded Regions I add specific folders for each job.

      Example:
      project/common
      project/projectA
      project/projectB
      project/projectC

      If I made change on projectB folder, all the three job are build.

          [JENKINS-20569] Polling with excluded regions don't prevent job triggered

          Mark Waite added a comment -

          I attempted various combinations of patterns and could not find any pattern which would exclude all the changes in a subdirectory (like project/common). The behavior was the same for the JGit and the command line implementations.

          I'm reasonably confident this worked in the past, since we used it with older versions of the plugin.

          Mark Waite added a comment - I attempted various combinations of patterns and could not find any pattern which would exclude all the changes in a subdirectory (like project/common). The behavior was the same for the JGit and the command line implementations. I'm reasonably confident this worked in the past, since we used it with older versions of the plugin.

          This feature worked at our location in the 1.4 version of the plugin.
          When I upgraded to the 2.0 version of the plugin, I experienced the issue listed above. The value that was formerly in 'Included Regions' was still present...but, it didn't result in the polling to be specific to the directory.
          So, I downgraded back to the 1.4 version of the plugin in hopes of getting the functionality back. Unfortunately, when I downgraded, the value present in the 'Included Regions' disappeared. Yikes.

          Sonia Shepherd added a comment - This feature worked at our location in the 1.4 version of the plugin. When I upgraded to the 2.0 version of the plugin, I experienced the issue listed above. The value that was formerly in 'Included Regions' was still present...but, it didn't result in the polling to be specific to the directory. So, I downgraded back to the 1.4 version of the plugin in hopes of getting the functionality back. Unfortunately, when I downgraded, the value present in the 'Included Regions' disappeared. Yikes.

          Mark Waite added a comment -

          Sonia, the 2.0 release was a major release which included changes to the job definition XML file. Those changes make the new data values unreadable by the old plugin. You will probably need to restore an old copy of the job definition XML file (jobs/<jobname>/config.xml).

          Mark Waite added a comment - Sonia, the 2.0 release was a major release which included changes to the job definition XML file. Those changes make the new data values unreadable by the old plugin. You will probably need to restore an old copy of the job definition XML file (jobs/<jobname>/config.xml).

          Yesterday, I downgrade from 2.0 to 1.5.
          Exculded regions work as expected but included regions as no effect.
          If I set "projector/.*\.cpp" in the included regions and I changed a .txt in git, a job was triggered...

          Anthony FLORANT added a comment - Yesterday, I downgrade from 2.0 to 1.5. Exculded regions work as expected but included regions as no effect. If I set "projector/.*\.cpp" in the included regions and I changed a .txt in git, a job was triggered...

          Joe Mihalich added a comment - - edited

          I am having the exact same issue with included regions not being honored. Every job on the branch that git sends gets built regardless of what's in the included regions. Was on 1.4 before this and it was working great.

          I have to downgrade to 1.4.0 because of this bug. I can't have the entire project building every time one file changes. Please fix this ASAP.

          Joe Mihalich added a comment - - edited I am having the exact same issue with included regions not being honored. Every job on the branch that git sends gets built regardless of what's in the included regions. Was on 1.4 before this and it was working great. I have to downgrade to 1.4.0 because of this bug. I can't have the entire project building every time one file changes. Please fix this ASAP.

          This happens due to remote polling. This is also the case with 1.x when remote polling is set
          By nature, remote polling only can compare remote HEAD with last build sha1, not consider commit filters

          Nicolas De Loof added a comment - This happens due to remote polling. This is also the case with 1.x when remote polling is set By nature, remote polling only can compare remote HEAD with last build sha1, not consider commit filters

          Joe Mihalich added a comment -

          I don't understand that. This feature was working fine in the 1.4.0 git plugin. Something must have been broken.

          Joe Mihalich added a comment - I don't understand that. This feature was working fine in the 1.4.0 git plugin. Something must have been broken.

          Erik Cokeley added a comment - - edited

          I also need the Included Regions. This is failing for me and many builds are being kicked off; I'm not sure if all my builds within the git and branch but at least many. What version has Included Region (or excluded region) working?

          Erik Cokeley added a comment - - edited I also need the Included Regions. This is failing for me and many builds are being kicked off; I'm not sure if all my builds within the git and branch but at least many. What version has Included Region (or excluded region) working?

          @Joe change is that remote polling is the default behavior now - this makes polling a lightweight process.

          Nicolas De Loof added a comment - @Joe change is that remote polling is the default behavior now - this makes polling a lightweight process.

          Joe Mihalich added a comment -

          Ok, I get it. So how do i get the behavior back to not do remote polling and make it work like 1.4.0?

          Joe Mihalich added a comment - Ok, I get it. So how do i get the behavior back to not do remote polling and make it work like 1.4.0?

          Mark Waite added a comment -

          Select the "Force polling using workspace" option with the "Add" button in the Git plugin section of the job configuration page. That disables remote polling.

          Mark Waite added a comment - Select the "Force polling using workspace" option with the "Add" button in the Git plugin section of the job configuration page. That disables remote polling.

          Joe Mihalich added a comment -

          Confirmed this works for me. Thanks for the help!

          Joe Mihalich added a comment - Confirmed this works for me. Thanks for the help!

          Liya Katz added a comment -

          Polling using workspace should be the default behavior (backward compatibility!) and "remote Polling" should be an option

          Liya Katz added a comment - Polling using workspace should be the default behavior (backward compatibility!) and "remote Polling" should be an option

          Mark Waite added a comment -

          I disagree that polling using workspace should be the default.

          Polling using workspace is significantly slower than remote polling. When we have a new feature (like fast remote polling support) that significantly improves performance, it is better to make the faster performance the default so that new users and new jobs have the benefit of the faster performance without any special actions.

          It is unfortunate that fast remote polling does not work in all cases, but I think it is still better to use that as the default rather than polling using workspace by default, then allowing users the option to execute faster.

          Mark Waite added a comment - I disagree that polling using workspace should be the default. Polling using workspace is significantly slower than remote polling. When we have a new feature (like fast remote polling support) that significantly improves performance, it is better to make the faster performance the default so that new users and new jobs have the benefit of the faster performance without any special actions. It is unfortunate that fast remote polling does not work in all cases, but I think it is still better to use that as the default rather than polling using workspace by default, then allowing users the option to execute faster.

          Liya Katz added a comment -

          I agree that new features should be be "promoted", but in this case they cause other important features to stop working.
          You have to mention this limitation when include/exclude regions are chosen.

          Liya Katz added a comment - I agree that new features should be be "promoted", but in this case they cause other important features to stop working. You have to mention this limitation when include/exclude regions are chosen.

          So Included/Excluded Regions doesn't work without "Force polling using workspace", is it? If so, should the first option at least mention it or even forcibly enable the second one?

          Alexander Uvizhev added a comment - So Included/Excluded Regions doesn't work without "Force polling using workspace", is it? If so, should the first option at least mention it or even forcibly enable the second one?

          Mark Waite added a comment -

          Yes, included/excluded regions require "Force polling using workspace". You're welcome to submit a pull request with an improvement to the help, or with forcible enablement of using a workspace for polling.

          Currently there are pull requests under review and discussion around the issue, including https://github.com/jenkinsci/git-plugin/pull/93 and https://github.com/jenkinsci/git-plugin/pull/89 . Both pull requests may need further work before they can be pulled, and they may be mutually exclusive, but you could consider them . Neither of them do precisely what you've proposed, but they may give you an idea where to investigate in the code.

          Mark Waite added a comment - Yes, included/excluded regions require "Force polling using workspace". You're welcome to submit a pull request with an improvement to the help, or with forcible enablement of using a workspace for polling. Currently there are pull requests under review and discussion around the issue, including https://github.com/jenkinsci/git-plugin/pull/93 and https://github.com/jenkinsci/git-plugin/pull/89 . Both pull requests may need further work before they can be pulled, and they may be mutually exclusive, but you could consider them . Neither of them do precisely what you've proposed, but they may give you an idea where to investigate in the code.

          Trey Duskin added a comment -

          We are having this issue as well with Included Paths in our Git jobs. We have set up Included Paths for all our jobs, and each job points to a different path in our git repo. When there is a commit which only touches one of these paths, all of our jobs are getting triggered. I have enabled "Force Polling from Workspace" for all of the jobs, but this doesn't seem to help.

          Any idea when this issue might be addressed?

          Trey Duskin added a comment - We are having this issue as well with Included Paths in our Git jobs. We have set up Included Paths for all our jobs, and each job points to a different path in our git repo. When there is a commit which only touches one of these paths, all of our jobs are getting triggered. I have enabled "Force Polling from Workspace" for all of the jobs, but this doesn't seem to help. Any idea when this issue might be addressed?

          Mark Waite added a comment -

          tduskin I think you're having some other issue, not what is described in this bug report. When I enable "Force polling from workspace", I see the expected exclusion behavior from the jobs I've created.

          Here are the steps I took, and the results I observed:

          1. Create a git repository with project/common/common-file, project/projectA/A-file, project/projectB/B-file, project/projectC/C-file
          2. For testing speed, I added a post-receive "hook" to the repository (see Kohsuke's "polling must die" blog post for the technique)
          3. Define a Jenkins job JENKINS-20569-projectA using that repository
          4. Add "Force polling using workspace" behavior to that project
          5. Add "Polling ignores commits in certain paths" to that project
          6. Define the exclusion as
            project/projectB/.*
            project/projectC/.*
            
          7. Commit a change to the repository outside the exclusion (like project/common/common-file), confirm projectA builds on next poll
          8. Commit a change to the repository inside the exclusions (like project/projectB/B-file), confirm projectA does not build on next poll
          9. Copy the JENKINS-20569-projectA job to JENKINS-20569-projectB, define the exclusion as
            project/projectA/.*
            project/projectC/.*
            
          10. Commit a change to the repository outside the exclusion (like project/common/common-file), confirm projectA and projectB build on next poll
          11. Commit a change to the repository inside the exclusions (like project/projectB/B-file), confirm projectB builds, and projectA does not build on next poll
          12. Commit a change to the repository inside the exclusions (like project/projectB/A-file), confirm projectB does not build, and projectA builds on next poll

          So long as I use "Force polling using workspace", and use a regular expression to match the exclusion, then the exclusion works the way I expect.

          Can you provide more details on the failure case you found?

          Mark Waite added a comment - tduskin I think you're having some other issue, not what is described in this bug report. When I enable "Force polling from workspace", I see the expected exclusion behavior from the jobs I've created. Here are the steps I took, and the results I observed: Create a git repository with project/common/common-file, project/projectA/A-file, project/projectB/B-file, project/projectC/C-file For testing speed, I added a post-receive "hook" to the repository (see Kohsuke's "polling must die" blog post for the technique) Define a Jenkins job JENKINS-20569 -projectA using that repository Add "Force polling using workspace" behavior to that project Add "Polling ignores commits in certain paths" to that project Define the exclusion as project/projectB/.* project/projectC/.* Commit a change to the repository outside the exclusion (like project/common/common-file), confirm projectA builds on next poll Commit a change to the repository inside the exclusions (like project/projectB/B-file), confirm projectA does not build on next poll Copy the JENKINS-20569 -projectA job to JENKINS-20569 -projectB, define the exclusion as project/projectA/.* project/projectC/.* Commit a change to the repository outside the exclusion (like project/common/common-file), confirm projectA and projectB build on next poll Commit a change to the repository inside the exclusions (like project/projectB/B-file), confirm projectB builds, and projectA does not build on next poll Commit a change to the repository inside the exclusions (like project/projectB/A-file), confirm projectB does not build, and projectA builds on next poll So long as I use "Force polling using workspace", and use a regular expression to match the exclusion, then the exclusion works the way I expect. Can you provide more details on the failure case you found?

          Trey Duskin added a comment -

          My case differs from your example in a few ways.

          First, my jobs are set up to only check out a specific branch (e.g. '*/dev' instead of 'master'). There could be other commits in the history which are going into other branches, but I don't care about those and I only want to watch for commits in a specific branch which have changes in specific paths.

          Second, we are using git-flow style development, and so most of the commits are merges of feature branches into the the branch I am watching.

          If I look at the git polling log on a job which should not have been triggered, I can see it trying to get a history between the last build revision and the commit which triggered the poll (we are using web-hooks from our git provider, Assembla). However, if I repeat the command it uses to get the commit history (e.g. /usr/bin/git log --full-history --no-abbrev --format=raw -M -m --raw ff0ec94a066cee23c13faa16b38b247911cd5ee7..f47993565f701a4f28174c4ea81f35d0e1103590), I see the commits i expect, but weirdly, I also see a commit listed twice - once containing the paths which the commit affected, and another (same hash) which contain paths from some other commit in some other branch! Is this a problem with my git provider?

          Trey Duskin added a comment - My case differs from your example in a few ways. First, my jobs are set up to only check out a specific branch (e.g. '*/dev' instead of 'master'). There could be other commits in the history which are going into other branches, but I don't care about those and I only want to watch for commits in a specific branch which have changes in specific paths. Second, we are using git-flow style development, and so most of the commits are merges of feature branches into the the branch I am watching. If I look at the git polling log on a job which should not have been triggered, I can see it trying to get a history between the last build revision and the commit which triggered the poll (we are using web-hooks from our git provider, Assembla). However, if I repeat the command it uses to get the commit history (e.g. /usr/bin/git log --full-history --no-abbrev --format=raw -M -m --raw ff0ec94a066cee23c13faa16b38b247911cd5ee7..f47993565f701a4f28174c4ea81f35d0e1103590), I see the commits i expect, but weirdly, I also see a commit listed twice - once containing the paths which the commit affected, and another (same hash) which contain paths from some other commit in some other branch! Is this a problem with my git provider?

          Trey Duskin added a comment -

          Oh, and to be clear, I am only using Included Paths. I am not specifying any Excluded Paths.

          Trey Duskin added a comment - Oh, and to be clear, I am only using Included Paths. I am not specifying any Excluded Paths.

          Mark Waite added a comment -

          tduskin could you open a separate bug report for your use case?

          This bug report is focused on excluded regions and on the implicit requirement that include and exclude regions implicitly require "Force polling using workspace". As far as I can tell, the exclude regions are working as expected (at least as described by this bug report) so long as "Force polling using workspace" is enabled.

          Mark Waite added a comment - tduskin could you open a separate bug report for your use case? This bug report is focused on excluded regions and on the implicit requirement that include and exclude regions implicitly require "Force polling using workspace". As far as I can tell, the exclude regions are working as expected (at least as described by this bug report) so long as "Force polling using workspace" is enabled.

          Trey Duskin added a comment -

          Thanks. I did some more experimentation and I think my issue is specific to merge commits. My new bug is here: https://issues.jenkins-ci.org/browse/JENKINS-23606

          Trey Duskin added a comment - Thanks. I did some more experimentation and I think my issue is specific to merge commits. My new bug is here: https://issues.jenkins-ci.org/browse/JENKINS-23606

          Erik Shreve added a comment - - edited

          Excluded regions are also not working for me.

          I have a test repo with the following structure
          /src/main.c
          /docs/otherdocs/somefile.txt

          I've added "Force polling using workspace" as an Additional behavior.
          The excluded regions are set to:

          /docs/otherdocs/.*
          /docs/otherdocs/somefile.txt
          docs/otherdocs/somefile.txt
          docs/otherdocs/.*\.txt

          Included regions is empty.

          When the project polls the repo, it still builds when the only changed file is:
          /docs/otherdocs/somefile.txt

          Change to file is simply
          >echo hello >> somefile.txt
          >git commit -a -m "changed somefile"
          >git push origin master

          I'm on Jenkins 1.597, with git plugin 2.3.5

          Polling Log shows:
          Started on Jun 26, 2015 10:02:01 AM
          Polling SCM changes on mybuildmachine
          Using strategy: Default
          [poll] Last Built Revision: Revision c4ab323cf0ac778ed3b0e5fd13467be5587d3fe7 (refs/remotes/ErikTest/master)
          Fetching changes from the remote Git repositories
          Polling for changes in
          Done. Took 1.1 sec
          Changes found

          Erik Shreve added a comment - - edited Excluded regions are also not working for me. I have a test repo with the following structure /src/main.c /docs/otherdocs/somefile.txt I've added "Force polling using workspace" as an Additional behavior. The excluded regions are set to: /docs/otherdocs/.* /docs/otherdocs/somefile.txt docs/otherdocs/somefile.txt docs/otherdocs/.*\.txt Included regions is empty. When the project polls the repo, it still builds when the only changed file is: /docs/otherdocs/somefile.txt Change to file is simply >echo hello >> somefile.txt >git commit -a -m "changed somefile" >git push origin master I'm on Jenkins 1.597, with git plugin 2.3.5 Polling Log shows: Started on Jun 26, 2015 10:02:01 AM Polling SCM changes on mybuildmachine Using strategy: Default [poll] Last Built Revision: Revision c4ab323cf0ac778ed3b0e5fd13467be5587d3fe7 (refs/remotes/ErikTest/master) Fetching changes from the remote Git repositories Polling for changes in Done. Took 1.1 sec Changes found

          Erik Shreve added a comment -

          For me something is fundamentally broken. Exclusion by commit message also does not work (I haven't tried the other exclusion features).

          Also, I tried setting the excluded regions to just:
          .*

          If I read the PathRestriction class correctly, this should prevent all builds from occurring. However, my build still happens.

          My branch specifier is set to */master
          Build triggers section just has "Poll SCM" checked and it is set to poll once an hour. I'm visiting the git/notifyCommit?url= address of the jenkins server to start the polling.

          Erik Shreve added a comment - For me something is fundamentally broken. Exclusion by commit message also does not work (I haven't tried the other exclusion features). Also, I tried setting the excluded regions to just: .* If I read the PathRestriction class correctly, this should prevent all builds from occurring. However, my build still happens. My branch specifier is set to */master Build triggers section just has "Poll SCM" checked and it is set to poll once an hour. I'm visiting the git/notifyCommit?url= address of the jenkins server to start the polling.

          Erik Shreve added a comment -

          I noticed the following in Jenkins' log file:
          Jun 26, 2015 4:11:02 PM WARNING hudson.plugins.git.GitTool$DescriptorImpl getInstallation
          invalid gitTool selection Default

          The Git Client plugin was configured with JGit. I deleted the JGit entry on the configuration page and after reloading the page it had put a Git entry in as default.

          The exclude regions are now working.

          Erik Shreve added a comment - I noticed the following in Jenkins' log file: Jun 26, 2015 4:11:02 PM WARNING hudson.plugins.git.GitTool$DescriptorImpl getInstallation invalid gitTool selection Default The Git Client plugin was configured with JGit. I deleted the JGit entry on the configuration page and after reloading the page it had put a Git entry in as default. The exclude regions are now working.

          Mark Waite added a comment - - edited

          eshreveti I don't understand the conflict between your quote from the log file and your statement about the job using JGit. The message "invalid gitTool selection Default" is the message I would expect to see when using command line git, yet you indicate that you switched from JGit to command line git. Could you double check that to confirm?

          If you find that is the case, please open another bug report, since this bug report is focused on the requirement that if an include region is defined, or an exclude region is defined, then the job should implicitly require a workspace for polling.

          I double checked with my 3 jobs that I use to experiment with this bug report. I switched all three of them to use JGit, then committed first to only projectA, confirmed projectA was the only one built. Then I committed to projectB and confirmed projectB was the only one built. The same for projectC, then for the combination of A and B. All behaved as expected.

          I had enabled "Polling uses workspace" so that they exclusions and inclusions could be resolved.

          Mark Waite added a comment - - edited eshreveti I don't understand the conflict between your quote from the log file and your statement about the job using JGit. The message "invalid gitTool selection Default" is the message I would expect to see when using command line git, yet you indicate that you switched from JGit to command line git. Could you double check that to confirm? If you find that is the case, please open another bug report, since this bug report is focused on the requirement that if an include region is defined, or an exclude region is defined, then the job should implicitly require a workspace for polling. I double checked with my 3 jobs that I use to experiment with this bug report. I switched all three of them to use JGit, then committed first to only projectA, confirmed projectA was the only one built. Then I committed to projectB and confirmed projectB was the only one built. The same for projectC, then for the combination of A and B. All behaved as expected. I had enabled "Polling uses workspace" so that they exclusions and inclusions could be resolved.

          citizenkahn added a comment - - edited

          I'm seeing this as well and it seems to only happen when the job clears its workspace
          We have two jobs:

          Base line when workspace exists

          • change to master triggers only triggermaster job
          • change to myfeature only triggermyfeature job

          Modify jobs adding exclusion when workspace exists

          • Additional Behaviors : Polling ignores commts in certain paths
            • excluded regions:
              • master: multMergeCommits/.*
              • feature: myfeature/.*
          Tests when workspace exists:
          • master commit to root - trigger only master
          • master commit to file beneath excluded - no trigger
          • feature commit to root - trigger only feature
          • feature commit to file beneath excluded - no trigger
          After purge workspace

          Commit to master branch causes both jobs to trigger. The github payload contains commit data so the plugin goin analyze the payload and determine if the commit path matched the regex for exclusion but it looks like it wants a workspace in which to perform the checks.

          Recommend bump to at least major. This problem with the feature makes it unusable in our environment and I expect many cannot use this feature.

          citizenkahn added a comment - - edited I'm seeing this as well and it seems to only happen when the job clears its workspace We have two jobs: trigger-feature.xml feature branch, excludes region: myfeature/.* trigger-master.xml master, excludes region: multiMergeCommits/.* Base line when workspace exists change to master triggers only triggermaster job change to myfeature only triggermyfeature job Modify jobs adding exclusion when workspace exists Additional Behaviors : Polling ignores commts in certain paths excluded regions: master: multMergeCommits/.* feature: myfeature/.* Tests when workspace exists: master commit to root - trigger only master master commit to file beneath excluded - no trigger feature commit to root - trigger only feature feature commit to file beneath excluded - no trigger After purge workspace Commit to master branch causes both jobs to trigger. The github payload contains commit data so the plugin goin analyze the payload and determine if the commit path matched the regex for exclusion but it looks like it wants a workspace in which to perform the checks. Recommend bump to at least major . This problem with the feature makes it unusable in our environment and I expect many cannot use this feature.

            Unassigned Unassigned
            anthony Anthony FLORANT
            Votes:
            13 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated: