Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23606

Git Polling Jobs with Included Regions being triggered unnecessarily with merge commits

    XMLWordPrintable

Details

    Description

      We have a single Git repository with several projects in a directory structure. I have created build jobs with polling enabled for each project and have included the appropriate directory structure in the "Included Regions" option in the git plugin setup. I have also turned on "Force Polling using Workspace" as indicated in https://issues.jenkins-ci.org/browse/JENKINS-20569.

      We are using a git-flow type development process, and the jobs are all configured to watch the '*/dev' branch. All commits made to this branch are merges. It appears that whenever a merge commit is made, it will trigger all of the jobs, regardless of if the merge only contained changes to one of the projects' directories.

      I looked into the polling log and it looks like the way that the plugin is getting the commit log and checking the paths may be causing the issue. Here is the command that the job runs in the polling log:

      > /usr/bin/git log --full-history --no-abbrev --format=raw -M -m --raw ff0ec94a066cee23c13faa16b38b247911cd5ee7..f47993565f701a4f28174c4ea81f35d0e1103590

      If I run this command manually, I can see that the returned paths for each commit also includes paths from other commits. I'm not sure why this is, but from the 'git' documentation, it looks like the '-m' parameter will do a full diff on both parents of a merge commit. If I leave off the '-m' parameter, or if I include '--first-parent', the list of paths in the commits only shows the actual affected files made by the commit which triggered the job. Perhaps this option needs to be included in the job config?

      Attachments

        Issue Links

          Activity

            eric_tucker Eric Tucker added a comment -

            Any updates? Noticed this bug myself as well.

            eric_tucker Eric Tucker added a comment - Any updates? Noticed this bug myself as well.
            doronshai Doron Shai added a comment -

            Also here.... this issue is a real problem

            doronshai Doron Shai added a comment - Also here.... this issue is a real problem

            I made a pull request to address the git-client portion of this issue:
            https://github.com/jenkinsci/git-client-plugin/pull/162

            git-plugin would need to be updated to use the new method if it's approved/merged.

            elsabbahyahmed Ahmed Elsabbahy added a comment - I made a pull request to address the git-client portion of this issue: https://github.com/jenkinsci/git-client-plugin/pull/162 git-plugin would need to be updated to use the new method if it's approved/merged.
            fnaum Federico Naum added a comment -

            Hi, I'm experiencing exactly the same issue. (and as doronshai mentioned it is a real problem)
            This problem is quite exacerbated when people merge master to on old branch

            At a first glance removing the `-m` on the `git log` line seems to return the results that I'll be expecting. But I'll try to use that PR that uses `git diff --name-only old...new`
            .....
            I'd had a look at the pull request and the comments with the reasoning it was not yet integrated.
            If I can get this working I'll try to check why the JGit vs git behaviors are different.

            fnaum Federico Naum added a comment - Hi, I'm experiencing exactly the same issue. (and as doronshai mentioned it is a real problem) This problem is quite exacerbated when people merge master to on old branch At a first glance removing the `-m` on the `git log` line seems to return the results that I'll be expecting. But I'll try to use that PR that uses `git diff --name-only old...new` ..... I'd had a look at the pull request and the comments with the reasoning it was not yet integrated. If I can get this working I'll try to check why the JGit vs git behaviors are different.
            pjohnsonic509 Phil Johnson added a comment -

            I can verify that this excluded/included region issue with "merge commits" is still an issue.

            Jenkins ver. 1.642
            git-scm plugin 2.4.2

            pjohnsonic509 Phil Johnson added a comment - I can verify that this excluded/included region issue with "merge commits" is still an issue. Jenkins ver. 1.642 git-scm plugin 2.4.2
            darenjsmith Daren Smith added a comment -

            Seeing the same issue as described on merge commits.

            darenjsmith Daren Smith added a comment - Seeing the same issue as described on merge commits.
            souciance Souciance Eqdam Rashti added a comment - - edited

            We are seeing the same issue here.

            We have set included and excluded regions but merge commits trigger all jobs which causes some jobs to fail and we have to manually trigger them.

            Our environment:
            jenkins 1.642.4
            git 2.4.4
            git-client 1.19.6

            souciance Souciance Eqdam Rashti added a comment - - edited We are seeing the same issue here. We have set included and excluded regions but merge commits trigger all jobs which causes some jobs to fail and we have to manually trigger them. Our environment: jenkins 1.642.4 git 2.4.4 git-client 1.19.6
            lledhub lledhub dell added a comment -

            Same issue here, in a setting in which GitFlow is used. The Jenkins job is configured to poll the developer branch: Polling ignores commits in certain paths - 'Included Regions' - /sources/.*
            Two feature branches are made:
            1. commit to a file in /sources/
            2. commit to a file in /documentation/
            For both feature branches, when being merged back to developer, the job is triggered. This should only be one build trigger, i.e. only the commit+merge that affects files in the sources directory.

            Git client plugin
            1.21.0

            Git plugin
            2.5.3

            Jenkins
            1.651.1

            lledhub lledhub dell added a comment - Same issue here, in a setting in which GitFlow is used. The Jenkins job is configured to poll the developer branch: Polling ignores commits in certain paths - 'Included Regions' - /sources/.* Two feature branches are made: 1. commit to a file in /sources/ 2. commit to a file in /documentation/ For both feature branches, when being merged back to developer, the job is triggered. This should only be one build trigger, i.e. only the commit+merge that affects files in the sources directory. Git client plugin 1.21.0 Git plugin 2.5.3 Jenkins 1.651.1
            kevinspeak Kevin Reed added a comment -

            Same issue, this is a big deal for large multi app repos

            kevinspeak Kevin Reed added a comment - Same issue, this is a big deal for large multi app repos
            markewaite Mark Waite added a comment - - edited

            I've placed a series of verification jobs into my lts-with-plugins docker image that duplicates the problem. I've resolved the PR162 conflicts in and pushed it as a branch in my development repository.

            The changes in PR162 are the git client plugin changes. The necessary changes in the git plugin have not been made. Changes in both will be required. The new API in git client will need to be called from git plugin (just as elsabbahyahmed said in his earlier comment).

            markewaite Mark Waite added a comment - - edited I've placed a series of verification jobs into my lts-with-plugins docker image that duplicates the problem. I've resolved the PR162 conflicts in and pushed it as a branch in my development repository . The changes in PR162 are the git client plugin changes. The necessary changes in the git plugin have not been made. Changes in both will be required. The new API in git client will need to be called from git plugin (just as elsabbahyahmed said in his earlier comment).
            pjohnsonic509 Phil Johnson added a comment -

             

            pjohnsonic509 Phil Johnson added a comment -  
            tmar Tomasz M added a comment -

            We have the same problem. Our temporary workaround is to set "Merge strategies" to "Rebase and merge" on Bitbucket Server. After rebase current git log command seems to work properly.

             

            Maybe, as a simple fix, you could allow to customize git log command in configuration?

             

            tmar Tomasz M added a comment - We have the same problem. Our temporary workaround is to set "Merge strategies" to "Rebase and merge" on Bitbucket Server. After rebase current git log command seems to work properly.   Maybe, as a simple fix, you could allow to customize git log command in configuration?  
            kshultz Karl Shultz added a comment -

            tmar (and others) - here's a question about the issues you're experiencing with included and excluded regions. Are you, by chance, starting your included regions with a leading /, as lledhub alluded to in this comment from September 9, 2016?

            I've written eight new tests that look at included/excluded areas specifically with merge commits. It took me a really, really long time to see it, but eventually I realized that the last failing test was failing because it had committed directory/filename to the git repository, but the included/excluded region pattern was looking for /directory/filename. The difference was that leading slash.

            kshultz Karl Shultz added a comment - tmar (and others) - here's a question about the issues you're experiencing with included and excluded regions. Are you, by chance, starting your included regions with a leading / , as lledhub alluded to in this comment from September 9, 2016? I've written eight new tests that look at included/excluded areas specifically with merge commits. It took me a really, really long time to see it, but eventually I realized that the last failing test was failing because it had committed directory/filename to the git repository, but the included/excluded region pattern was looking for /directory/filename . The difference was that leading slash.
            kshultz Karl Shultz added a comment - - edited

            The eight new tests have been merged to master. They include, in the form of "top level directory of the repo" and "inside a subdirectory" tests, the following:

            • Merge commits inside of excluded regions get ignored (file, directory)
            • Merge commits inside of included regions get processed as new changes (file, directory)
            • Merge commits outside of included regions get ignored (file, directory)
            • Merge commits outside of excluded regions get processed (file, directory)
            kshultz Karl Shultz added a comment - - edited The eight new tests have been merged to master . They include, in the form of "top level directory of the repo" and "inside a subdirectory" tests, the following: Merge commits inside of excluded regions get ignored ( file , directory ) Merge commits inside of included regions get processed as new changes ( file , directory ) Merge commits outside of included regions get ignored ( file , directory ) Merge commits outside of excluded regions get processed ( file , directory )
            tmar Tomasz M added a comment -

            kshultz - here are the settings we use:

            Branch Specifier (blank for 'any') :origin/develop
            Included Regions ^report-service/.*
            Excluded Regions [empty]  
            tmar Tomasz M added a comment - kshultz - here are the settings we use: Branch Specifier (blank for 'any') :origin/develop Included Regions ^report-service/.* Excluded Regions [empty]  

            We have the same issue: https://issues.jenkins.io/browse/JENKINS-65932
            Indeed the command with '-m'
            ```
            git log --full-history --no-abbrev --format=raw --raw -M -m 9aa631d36264e7d21e60881265572ec099ebc2b1..4fa8bc728c0a83d8defd2fb8a786505c1fbca7be
            ```
            contains invalid commit history included foreign changes from merge request

            How to disable '-m' flag in polling?

            anatolys Anatoly Shirokov added a comment - We have the same issue: https://issues.jenkins.io/browse/JENKINS-65932 Indeed the command with '-m' ``` git log --full-history --no-abbrev --format=raw --raw -M -m 9aa631d36264e7d21e60881265572ec099ebc2b1..4fa8bc728c0a83d8defd2fb8a786505c1fbca7be ``` contains invalid commit history included foreign changes from merge request How to disable '-m' flag in polling?
            anatolys Anatoly Shirokov added a comment - It seems to me here is fix https://github.com/jenkinsci/git-client-plugin/pull/750

            People

              Unassigned Unassigned
              tduskin Trey Duskin
              Votes:
              27 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

                Created:
                Updated: