Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-12545

Jenkins should build all tracked branches that have changed when triggered by Github

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • Jenkins 1.448, Github-api plugin 1.16, Github plugin 1.0, Git plugin 1.1.15
      Ubuntu 9.04

      I have several projects configured to build multiple branches and set to build when changes are pushed to Github. For example, several projects have origin/master and origin/development registered. It seems to build the correct branch when I push to a single branch on Github, but it fails to behave as expected in the following situations:

      1. If I push to two branches at once, only one of them gets built. It should build both of them. This probably has something to do with the changelog for 1.1.12 at https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin that talks about picking branches.

      2. If I push to one branch, then push to a second branch while the first is being built, the second one never gets built.

      This bug seems to force me to set up a separate job for each branch I want to build, which works poorly with the Github webhook (it build all of the branches every time). I can't switch to polling (I think) because my slave systems are volatile and Jenkins seems to lose track of which ref was built last when the slave it was built on goes away.

          [JENKINS-12545] Jenkins should build all tracked branches that have changed when triggered by Github

          anyway it would be nice if it was possible to force build for every branch

          Carsten Brandt added a comment - anyway it would be nice if it was possible to force build for every branch

          Hi, this appears to continue being an issue, are there any solutions? Im personally not a fan of the {[multi-branch}} plugin

          Stepan Mazurov added a comment - Hi, this appears to continue being an issue, are there any solutions? Im personally not a fan of the {[multi-branch}} plugin

          Added git plugin, because GH plugin only calls polling for every incoming commit.
          We plan switch from poll call to notifyCommit, but i don't think that it may fix this issue.
          Comments from git-plugin maintainer are welcome.

          Kanstantsin Shautsou added a comment - Added git plugin, because GH plugin only calls polling for every incoming commit. We plan switch from poll call to notifyCommit, but i don't think that it may fix this issue. Comments from git-plugin maintainer are welcome.

          Mark Waite added a comment - - edited

          I'm unable to duplicate the scenario using only the git plugin. I don't have a GitHub webhook configured, so I can't test with the GitHub webhook case specifically. Can you help me understand where I made a mistake in my steps?

          1. Create a bare git repository (/var/lib/git/jenkins/bugs/JENKINS-12545.git)
          2. Add a post-receive hook to bare git repository to notify my Jenkins on each commit
          3. Clone that bare repository, create a master branch, and push the master branch to the bare repo
          4. Create 10 branches (branch-0 through branch-9) and push them to the bare repo
          5. Create a job monitoring the bare repo, with a shell build step "sleep 3; git log -n 1"
          6. Configure the job to poll SCM but with no time defined (listen to notify commit messages, but don't poll the repo on a schedule)
          7. Commit changes to all branches and push them at the same time (git push --all), confirm that all pushed branches build
          8. While those builds are running, commit changes to a few more branches, confirm those branches are built

          I used git plugin 2.4.0 and git client plugin 1.18.0 for those steps

          Mark Waite added a comment - - edited I'm unable to duplicate the scenario using only the git plugin. I don't have a GitHub webhook configured, so I can't test with the GitHub webhook case specifically. Can you help me understand where I made a mistake in my steps? 1. Create a bare git repository (/var/lib/git/jenkins/bugs/ JENKINS-12545 .git) 2. Add a post-receive hook to bare git repository to notify my Jenkins on each commit 3. Clone that bare repository, create a master branch, and push the master branch to the bare repo 4. Create 10 branches (branch-0 through branch-9) and push them to the bare repo 5. Create a job monitoring the bare repo, with a shell build step "sleep 3; git log -n 1" 6. Configure the job to poll SCM but with no time defined (listen to notify commit messages, but don't poll the repo on a schedule) 7. Commit changes to all branches and push them at the same time (git push --all), confirm that all pushed branches build 8. While those builds are running, commit changes to a few more branches, confirm those branches are built I used git plugin 2.4.0 and git client plugin 1.18.0 for those steps

          Kanstantsin Shautsou added a comment - - edited

          markewaite github executes polling in threadPoolForRemoting, so scenario looks like:
          1) 2hooks incoming into jenkins
          2) github-plugin sends for every hook `git scm .poll() and if it true - schedules build` into threadPool.
          Seems second poll after first returns false because first polling saved full state of BuildData or first poll didn't trigger two builds?
          MM... seems we really need migrate to commitNotify() with sending branch/hashes into it so it may definitely say that single branch changed.

          Kanstantsin Shautsou added a comment - - edited markewaite github executes polling in threadPoolForRemoting, so scenario looks like: 1) 2hooks incoming into jenkins 2) github-plugin sends for every hook `git scm .poll() and if it true - schedules build` into threadPool. Seems second poll after first returns false because first polling saved full state of BuildData or first poll didn't trigger two builds? MM... seems we really need migrate to commitNotify() with sending branch/hashes into it so it may definitely say that single branch changed.

          Mark Waite added a comment -

          integer I don't know the details of the scm.poll() implementation enough to tell you why the second poll returns false.

          There is a (to me) surprising current behavior in the git plugin which might be relevant here. When a job is running that uses the git plugin, if a poll is needed, then that poll starts (as visible from the UI) but blocks waiting for the current running job to complete. I'm not sure if that is due to synchronization or something else, but it is a behavior that I have consistently observed. Do you think that behavior might have some relevance to this bug?

          Mark Waite added a comment - integer I don't know the details of the scm.poll() implementation enough to tell you why the second poll returns false. There is a (to me) surprising current behavior in the git plugin which might be relevant here. When a job is running that uses the git plugin, if a poll is needed, then that poll starts (as visible from the UI) but blocks waiting for the current running job to complete. I'm not sure if that is due to synchronization or something else, but it is a behavior that I have consistently observed. Do you think that behavior might have some relevance to this bug?

          smazurov sbleon cebe could you:

          1. update to latest git/github plugins
          2. open github-plugin sources in IDE
          3. run debug and put breakpoints here https://github.com/jenkinsci/github-plugin/blob/master/src/main/java/com/cloudbees/jenkins/GitHubPushTrigger.java#L69-L72
          4. create new clean repo (how markewaite described)
          5. then push into two branches and compare.

          markewaite seems i mistaken, remoting thread pool is wrapped in github descriptor.

          Kanstantsin Shautsou added a comment - smazurov sbleon cebe could you: update to latest git/github plugins open github-plugin sources in IDE run debug and put breakpoints here https://github.com/jenkinsci/github-plugin/blob/master/src/main/java/com/cloudbees/jenkins/GitHubPushTrigger.java#L69-L72 create new clean repo (how markewaite described) then push into two branches and compare. markewaite seems i mistaken, remoting thread pool is wrapped in github descriptor.

          Stepan Mazurov added a comment - - edited

          Unfortunately, I do not have ability to run locally with a debugger. I will, however, share as much info as possible.

          My github hook log when i pushed develop and master at the same time:

          Using strategy: Default
          [poll] Last Built Revision: Revision [sha1] (origin/develop)
           > git rev-parse --is-inside-work-tree # timeout=10
          Fetching changes from the remote Git repositories
           > git config remote.origin.url git@[github url] # timeout=10
          Pruning obsolete local branches
          Fetching upstream changes from [github url]
           > git --version # timeout=10
          using GIT_SSH to set credentials A Deploy user
           > git -c core.askpass=true fetch --tags --progress [github url] +refs/heads/*:refs/remotes/origin/* --prune
          Polling for changes in
          Seen branch in repository origin/develop
          Seen branch in repository origin/master
          Seen 2 remote branches
           > git log --full-history --no-abbrev --format=raw -M -m --raw [older sha1]..[sha 1] # timeout=10
          Done. Took 3.1 sec
          Changes found
          

          Only develop built.

          I then switched the trigger from Build when a change is pushed to GitHub to Poll SCM, and it still only built 1 branch.

          Here are the logs:

          Using strategy: Default
          [poll] Last Built Revision: Revision [sha1] (origin/develop)
           > git rev-parse --is-inside-work-tree # timeout=10
          Fetching changes from the remote Git repositories
           > git config remote.origin.url [github project] # timeout=10
          Pruning obsolete local branches
          Fetching upstream changes from [github project]
           > git --version # timeout=10
          using GIT_SSH to set credentials A Deploy user
           > git -c core.askpass=true fetch --tags --progress [github project] +refs/heads/*:refs/remotes/origin/* --prune
          Polling for changes in
          Seen branch in repository origin/develop
          Seen branch in repository origin/master
          Seen 2 remote branches
          Done. Took 1.1 sec
          No changes
          

          github-plugin version: 1.11.3 Edit: and 1.12.0, functionality unchanged
          git-plugin version: 2.4.0
          git-client version: 1.18.0

          Stepan Mazurov added a comment - - edited Unfortunately, I do not have ability to run locally with a debugger. I will, however, share as much info as possible. My github hook log when i pushed develop and master at the same time: Using strategy: Default [poll] Last Built Revision: Revision [sha1] (origin/develop) > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repositories > git config remote.origin.url git@[github url] # timeout=10 Pruning obsolete local branches Fetching upstream changes from [github url] > git --version # timeout=10 using GIT_SSH to set credentials A Deploy user > git -c core.askpass= true fetch --tags --progress [github url] +refs/heads/*:refs/remotes/origin/* --prune Polling for changes in Seen branch in repository origin/develop Seen branch in repository origin/master Seen 2 remote branches > git log --full-history --no-abbrev --format=raw -M -m --raw [older sha1]..[sha 1] # timeout=10 Done. Took 3.1 sec Changes found Only develop built. I then switched the trigger from Build when a change is pushed to GitHub to Poll SCM , and it still only built 1 branch. Here are the logs: Using strategy: Default [poll] Last Built Revision: Revision [sha1] (origin/develop) > git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repositories > git config remote.origin.url [github project] # timeout=10 Pruning obsolete local branches Fetching upstream changes from [github project] > git --version # timeout=10 using GIT_SSH to set credentials A Deploy user > git -c core.askpass= true fetch --tags --progress [github project] +refs/heads/*:refs/remotes/origin/* --prune Polling for changes in Seen branch in repository origin/develop Seen branch in repository origin/master Seen 2 remote branches Done. Took 1.1 sec No changes github-plugin version: 1.11.3 Edit: and 1.12.0 , functionality unchanged git-plugin version: 2.4.0 git-client version: 1.18.0

          markewaite I effectively did same repro steps as you, with 2 changes:

          1. I did not have any post-receive hooks set up, simply SCM poll every minute
          2. The "origin" repo was on github

          Does it work correctly for you if you remove post-receive hook and simply poll?

          integer is there anything else I could do to help fix this? Its quite a big issue that I would love to resolve. Effectively all git-flow type workflows are affected.

          Stepan Mazurov added a comment - markewaite I effectively did same repro steps as you, with 2 changes: 1. I did not have any post-receive hooks set up, simply SCM poll every minute 2. The "origin" repo was on github Does it work correctly for you if you remove post-receive hook and simply poll? integer is there anything else I could do to help fix this? Its quite a big issue that I would love to resolve. Effectively all git-flow type workflows are affected.

          Mark Waite added a comment -

          It continues to behave as expected for me if I remove the post-receive hook and simply poll. When I push to multiple branches, multiple jobs are triggered by polling. I assume that there must be something inside the git plugin which decides that multiple jobs need to be scheduled on a single poll result.

          Mark Waite added a comment - It continues to behave as expected for me if I remove the post-receive hook and simply poll. When I push to multiple branches, multiple jobs are triggered by polling. I assume that there must be something inside the git plugin which decides that multiple jobs need to be scheduled on a single poll result.

            Unassigned Unassigned
            sbleon Leon Miller-Out
            Votes:
            4 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated: