Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12545

Jenkins should build all tracked branches that have changed when triggered by Github

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      I have several projects configured to build multiple branches and set to build when changes are pushed to Github. For example, several projects have origin/master and origin/development registered. It seems to build the correct branch when I push to a single branch on Github, but it fails to behave as expected in the following situations:

      1. If I push to two branches at once, only one of them gets built. It should build both of them. This probably has something to do with the changelog for 1.1.12 at https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin that talks about picking branches.

      2. If I push to one branch, then push to a second branch while the first is being built, the second one never gets built.

      This bug seems to force me to set up a separate job for each branch I want to build, which works poorly with the Github webhook (it build all of the branches every time). I can't switch to polling (I think) because my slave systems are volatile and Jenkins seems to lose track of which ref was built last when the slave it was built on goes away.

        Attachments

          Issue Links

            Activity

            Hide
            cebe Carsten Brandt added a comment -
            Show
            cebe Carsten Brandt added a comment - might be expected behavior when they are dependend: http://jenkins.361315.n4.nabble.com/Building-multiple-branches-with-Jenkins-td3547909.html
            Hide
            cebe Carsten Brandt added a comment -

            anyway it would be nice if it was possible to force build for every branch

            Show
            cebe Carsten Brandt added a comment - anyway it would be nice if it was possible to force build for every branch
            Hide
            smazurov Stepan Mazurov added a comment -

            Hi, this appears to continue being an issue, are there any solutions? Im personally not a fan of the {[multi-branch}} plugin

            Show
            smazurov Stepan Mazurov added a comment - Hi, this appears to continue being an issue, are there any solutions? Im personally not a fan of the {[multi-branch}} plugin
            Hide
            integer Kanstantsin Shautsou added a comment -

            Added git plugin, because GH plugin only calls polling for every incoming commit.
            We plan switch from poll call to notifyCommit, but i don't think that it may fix this issue.
            Comments from git-plugin maintainer are welcome.

            Show
            integer Kanstantsin Shautsou added a comment - Added git plugin, because GH plugin only calls polling for every incoming commit. We plan switch from poll call to notifyCommit, but i don't think that it may fix this issue. Comments from git-plugin maintainer are welcome.
            Hide
            markewaite Mark Waite added a comment - - edited

            I'm unable to duplicate the scenario using only the git plugin. I don't have a GitHub webhook configured, so I can't test with the GitHub webhook case specifically. Can you help me understand where I made a mistake in my steps?

            1. Create a bare git repository (/var/lib/git/jenkins/bugs/JENKINS-12545.git)
            2. Add a post-receive hook to bare git repository to notify my Jenkins on each commit
            3. Clone that bare repository, create a master branch, and push the master branch to the bare repo
            4. Create 10 branches (branch-0 through branch-9) and push them to the bare repo
            5. Create a job monitoring the bare repo, with a shell build step "sleep 3; git log -n 1"
            6. Configure the job to poll SCM but with no time defined (listen to notify commit messages, but don't poll the repo on a schedule)
            7. Commit changes to all branches and push them at the same time (git push --all), confirm that all pushed branches build
            8. While those builds are running, commit changes to a few more branches, confirm those branches are built

            I used git plugin 2.4.0 and git client plugin 1.18.0 for those steps

            Show
            markewaite Mark Waite added a comment - - edited I'm unable to duplicate the scenario using only the git plugin. I don't have a GitHub webhook configured, so I can't test with the GitHub webhook case specifically. Can you help me understand where I made a mistake in my steps? 1. Create a bare git repository (/var/lib/git/jenkins/bugs/ JENKINS-12545 .git) 2. Add a post-receive hook to bare git repository to notify my Jenkins on each commit 3. Clone that bare repository, create a master branch, and push the master branch to the bare repo 4. Create 10 branches (branch-0 through branch-9) and push them to the bare repo 5. Create a job monitoring the bare repo, with a shell build step "sleep 3; git log -n 1" 6. Configure the job to poll SCM but with no time defined (listen to notify commit messages, but don't poll the repo on a schedule) 7. Commit changes to all branches and push them at the same time (git push --all), confirm that all pushed branches build 8. While those builds are running, commit changes to a few more branches, confirm those branches are built I used git plugin 2.4.0 and git client plugin 1.18.0 for those steps
            Hide
            integer Kanstantsin Shautsou added a comment - - edited

            Mark Waite github executes polling in threadPoolForRemoting, so scenario looks like:
            1) 2hooks incoming into jenkins
            2) github-plugin sends for every hook `git scm .poll() and if it true - schedules build` into threadPool.
            Seems second poll after first returns false because first polling saved full state of BuildData or first poll didn't trigger two builds?
            MM... seems we really need migrate to commitNotify() with sending branch/hashes into it so it may definitely say that single branch changed.

            Show
            integer Kanstantsin Shautsou added a comment - - edited Mark Waite github executes polling in threadPoolForRemoting, so scenario looks like: 1) 2hooks incoming into jenkins 2) github-plugin sends for every hook `git scm .poll() and if it true - schedules build` into threadPool. Seems second poll after first returns false because first polling saved full state of BuildData or first poll didn't trigger two builds? MM... seems we really need migrate to commitNotify() with sending branch/hashes into it so it may definitely say that single branch changed.
            Hide
            markewaite Mark Waite added a comment -

            Kanstantsin Shautsou I don't know the details of the scm.poll() implementation enough to tell you why the second poll returns false.

            There is a (to me) surprising current behavior in the git plugin which might be relevant here. When a job is running that uses the git plugin, if a poll is needed, then that poll starts (as visible from the UI) but blocks waiting for the current running job to complete. I'm not sure if that is due to synchronization or something else, but it is a behavior that I have consistently observed. Do you think that behavior might have some relevance to this bug?

            Show
            markewaite Mark Waite added a comment - Kanstantsin Shautsou I don't know the details of the scm.poll() implementation enough to tell you why the second poll returns false. There is a (to me) surprising current behavior in the git plugin which might be relevant here. When a job is running that uses the git plugin, if a poll is needed, then that poll starts (as visible from the UI) but blocks waiting for the current running job to complete. I'm not sure if that is due to synchronization or something else, but it is a behavior that I have consistently observed. Do you think that behavior might have some relevance to this bug?
            Hide
            integer Kanstantsin Shautsou added a comment -

            Stepan Mazurov Leon Miller-Out Carsten Brandt could you:

            1. update to latest git/github plugins
            2. open github-plugin sources in IDE
            3. run debug and put breakpoints here https://github.com/jenkinsci/github-plugin/blob/master/src/main/java/com/cloudbees/jenkins/GitHubPushTrigger.java#L69-L72
            4. create new clean repo (how Mark Waite described)
            5. then push into two branches and compare.

            Mark Waite seems i mistaken, remoting thread pool is wrapped in github descriptor.

            Show
            integer Kanstantsin Shautsou added a comment - Stepan Mazurov Leon Miller-Out Carsten Brandt could you: update to latest git/github plugins open github-plugin sources in IDE run debug and put breakpoints here https://github.com/jenkinsci/github-plugin/blob/master/src/main/java/com/cloudbees/jenkins/GitHubPushTrigger.java#L69-L72 create new clean repo (how Mark Waite described) then push into two branches and compare. Mark Waite seems i mistaken, remoting thread pool is wrapped in github descriptor.
            Hide
            smazurov Stepan Mazurov added a comment - - edited

            Unfortunately, I do not have ability to run locally with a debugger. I will, however, share as much info as possible.

            My github hook log when i pushed develop and master at the same time:

            Using strategy: Default
            [poll] Last Built Revision: Revision [sha1] (origin/develop)
             > git rev-parse --is-inside-work-tree # timeout=10
            Fetching changes from the remote Git repositories
             > git config remote.origin.url git@[github url] # timeout=10
            Pruning obsolete local branches
            Fetching upstream changes from [github url]
             > git --version # timeout=10
            using GIT_SSH to set credentials A Deploy user
             > git -c core.askpass=true fetch --tags --progress [github url] +refs/heads/*:refs/remotes/origin/* --prune
            Polling for changes in
            Seen branch in repository origin/develop
            Seen branch in repository origin/master
            Seen 2 remote branches
             > git log --full-history --no-abbrev --format=raw -M -m --raw [older sha1]..[sha 1] # timeout=10
            Done. Took 3.1 sec
            Changes found
            

            Only develop built.

            I then switched the trigger from Build when a change is pushed to GitHub to Poll SCM, and it still only built 1 branch.

            Here are the logs:

            Using strategy: Default
            [poll] Last Built Revision: Revision [sha1] (origin/develop)
             > git rev-parse --is-inside-work-tree # timeout=10
            Fetching changes from the remote Git repositories
             > git config remote.origin.url [github project] # timeout=10
            Pruning obsolete local branches
            Fetching upstream changes from [github project]
             > git --version # timeout=10
            using GIT_SSH to set credentials A Deploy user
             > git -c core.askpass=true fetch --tags --progress [github project] +refs/heads/*:refs/remotes/origin/* --prune
            Polling for changes in
            Seen branch in repository origin/develop
            Seen branch in repository origin/master
            Seen 2 remote branches
            Done. Took 1.1 sec
            No changes
            

            github-plugin version: 1.11.3 Edit: and 1.12.0, functionality unchanged
            git-plugin version: 2.4.0
            git-client version: 1.18.0

            Show
            smazurov Stepan Mazurov added a comment - - edited Unfortunately, I do not have ability to run locally with a debugger. I will, however, share as much info as possible. My github hook log when i pushed develop and master at the same time: Using strategy: Default [poll] Last Built Revision: Revision [sha1] (origin/develop) > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repositories > git config remote.origin.url git@[github url] # timeout=10 Pruning obsolete local branches Fetching upstream changes from [github url] > git --version # timeout=10 using GIT_SSH to set credentials A Deploy user > git -c core.askpass= true fetch --tags --progress [github url] +refs/heads/*:refs/remotes/origin/* --prune Polling for changes in Seen branch in repository origin/develop Seen branch in repository origin/master Seen 2 remote branches > git log --full-history --no-abbrev --format=raw -M -m --raw [older sha1]..[sha 1] # timeout=10 Done. Took 3.1 sec Changes found Only develop built. I then switched the trigger from Build when a change is pushed to GitHub to Poll SCM , and it still only built 1 branch. Here are the logs: Using strategy: Default [poll] Last Built Revision: Revision [sha1] (origin/develop) > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repositories > git config remote.origin.url [github project] # timeout=10 Pruning obsolete local branches Fetching upstream changes from [github project] > git --version # timeout=10 using GIT_SSH to set credentials A Deploy user > git -c core.askpass= true fetch --tags --progress [github project] +refs/heads/*:refs/remotes/origin/* --prune Polling for changes in Seen branch in repository origin/develop Seen branch in repository origin/master Seen 2 remote branches Done. Took 1.1 sec No changes github-plugin version: 1.11.3 Edit: and 1.12.0 , functionality unchanged git-plugin version: 2.4.0 git-client version: 1.18.0
            Hide
            smazurov Stepan Mazurov added a comment -

            Mark Waite I effectively did same repro steps as you, with 2 changes:

            1. I did not have any post-receive hooks set up, simply SCM poll every minute
            2. The "origin" repo was on github

            Does it work correctly for you if you remove post-receive hook and simply poll?

            Kanstantsin Shautsou is there anything else I could do to help fix this? Its quite a big issue that I would love to resolve. Effectively all git-flow type workflows are affected.

            Show
            smazurov Stepan Mazurov added a comment - Mark Waite I effectively did same repro steps as you, with 2 changes: 1. I did not have any post-receive hooks set up, simply SCM poll every minute 2. The "origin" repo was on github Does it work correctly for you if you remove post-receive hook and simply poll? Kanstantsin Shautsou is there anything else I could do to help fix this? Its quite a big issue that I would love to resolve. Effectively all git-flow type workflows are affected.
            Hide
            markewaite Mark Waite added a comment -

            It continues to behave as expected for me if I remove the post-receive hook and simply poll. When I push to multiple branches, multiple jobs are triggered by polling. I assume that there must be something inside the git plugin which decides that multiple jobs need to be scheduled on a single poll result.

            Show
            markewaite Mark Waite added a comment - It continues to behave as expected for me if I remove the post-receive hook and simply poll. When I push to multiple branches, multiple jobs are triggered by polling. I assume that there must be something inside the git plugin which decides that multiple jobs need to be scheduled on a single poll result.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              sbleon Leon Miller-Out
              Votes:
              4 Vote for this issue
              Watchers:
              10 Start watching this issue

                Dates

                Created:
                Updated: