Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23415

2 Jobs w/same repo. Job triggers twice when first triggered job is Git SCM is still downloading

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • CentOS 5, JDK 1.7

      Problem:
      A job keeps triggering when it should not.

      Background:
      2 separate jobs are configured to be triggered on the same Github repository depending on branch that pushed to.

      When pushing to the 1st branch, "origin/test/branch1", the 1st job is triggered. Good.
      When pushing to the 2nd branch, "origin/test/branch2", the 2nd job triggers. Good.
      But the 1st job is triggered again. Bad. There's only been 2 pushes.

      The repository is very large, so the git clone takes a few minutes for both jobs.

      The problem consistently occurs when pushing to the 1st branch, THEN immediately pushing to the 2nd branch. The 1st job is triggered and git cloning. And the 2nd job is triggered and queued. BUT another 1st job is triggered and queued.

      Seems to happen when it's still downloading during the clone.

      That extra triggered job is causing us grief, because it's used for deployments.

      This is reproducible consistently with this script:

      ----------------------------
      #!/bin/bash
      pushd mydirectory
      git checkout develop
      echo ${1}d >> d.txt
      git add .
      git commit -m "${1}d"
      git push

      git checkout test/branch1
      git pull --no-edit upstream develop
      git push

      git checkout test/branch2
      git pull --no-edit upstream test/branch1
      git push
      popd
      ----------------------------

      And run it as:

      1. ./script 1
      2. ./script 2
        etc...

      Jenkins v1.558
      Git Client v1.9.1
      Git plugin v2.2.1
      GitHub API v1.54
      Github OAuth v0.14
      GitHub Pull Request Builder 1.12
      GitHub plugin v1.8

          [JENKINS-23415] 2 Jobs w/same repo. Job triggers twice when first triggered job is Git SCM is still downloading

          Frank Lee added a comment -

          Actually, I'd much prefer having the job with the regex work (e.g. queue up builds).

          Frank Lee added a comment - Actually, I'd much prefer having the job with the regex work (e.g. queue up builds).

          Mark Waite added a comment -

          To have the job queue multiple builds, you might try the "allow concurrent builds" check box, in hopes that would not optimize away the second request to build.

          I can't test the github push mechanism because I'm not willing to expose my test Jenkins server on the public internet, and I'm not willing to configure and expose a reverse proxy on the public internet.

          I can only guess that the key difference is between the GitHub push notifications that you're using and the simple git hook based notifications I'm using. I removed the GitHub components from the bug report because I assumed it was generic to the git client plugin. You'll probably need to put them back, or confirm that you can see the problem without requiring a GitHub push notification.

          Mark Waite added a comment - To have the job queue multiple builds, you might try the "allow concurrent builds" check box, in hopes that would not optimize away the second request to build. I can't test the github push mechanism because I'm not willing to expose my test Jenkins server on the public internet, and I'm not willing to configure and expose a reverse proxy on the public internet. I can only guess that the key difference is between the GitHub push notifications that you're using and the simple git hook based notifications I'm using. I removed the GitHub components from the bug report because I assumed it was generic to the git client plugin. You'll probably need to put them back, or confirm that you can see the problem without requiring a GitHub push notification.

          Frank Lee added a comment -

          To have the job queue multiple builds, you might try the "allow concurrent builds" check box, in hopes that would not optimize away the second request to build.

          That's a interesting idea. Unfortunately in this case, the job isn't made to run concurrently, else the concurrent builds will step on each other.

          I removed the GitHub components from the bug report because I assumed it was generic to the git client plugin. You'll probably need to put them back, or confirm that you can see the problem without requiring a GitHub push notification.

          Since it's going through the GitHub component because "Build when a change is pushed to GitHub" is checked and I'm not sure how to reproduce this problem without having GitHub push notification setup, I've added them back.

          Frank Lee added a comment - To have the job queue multiple builds, you might try the "allow concurrent builds" check box, in hopes that would not optimize away the second request to build. That's a interesting idea. Unfortunately in this case, the job isn't made to run concurrently, else the concurrent builds will step on each other. I removed the GitHub components from the bug report because I assumed it was generic to the git client plugin. You'll probably need to put them back, or confirm that you can see the problem without requiring a GitHub push notification. Since it's going through the GitHub component because "Build when a change is pushed to GitHub" is checked and I'm not sure how to reproduce this problem without having GitHub push notification setup, I've added them back.

          Frank Lee added a comment -

          I edited the original post to add one more plugin: "GitHub plugin v1.8".
          I'm guessing the GitHub payloads are triggered with that plugin.
          If there's anything else I can do to move this issue forward, let me know.

          Frank Lee added a comment - I edited the original post to add one more plugin: "GitHub plugin v1.8". I'm guessing the GitHub payloads are triggered with that plugin. If there's anything else I can do to move this issue forward, let me know.

          Frank Lee added a comment -

          I've applied the updates:

          Git Client v1.9.0 -> v1.9.1
          GitHub API v1.44  -> v1.54
          

          I've updated the origin post to reflect the current versions. I'm still seeing the same problem of the first job triggering a second time on the last push

          git checkout test/branch2
          git pull --no-edit upstream test/branch1
          git push
          

          Security is tighten down with a reverse proxy in front of Jenkins. Only Github IP addresses are allowed in as listed in this link: https://help.github.com/articles/what-ip-addresses-does-github-use-that-i-should-whitelist
          Further, there's bot account that is limited to read and build using the "Project-based Matrix Authorization Strategy".

          Frank Lee added a comment - I've applied the updates: Git Client v1.9.0 -> v1.9.1 GitHub API v1.44 -> v1.54 I've updated the origin post to reflect the current versions. I'm still seeing the same problem of the first job triggering a second time on the last push git checkout test/branch2 git pull --no-edit upstream test/branch1 git push Security is tighten down with a reverse proxy in front of Jenkins. Only Github IP addresses are allowed in as listed in this link: https://help.github.com/articles/what-ip-addresses-does-github-use-that-i-should-whitelist Further, there's bot account that is limited to read and build using the "Project-based Matrix Authorization Strategy".

          Xavier Shay added a comment -

          I am seeing something very similar to this, I suspect it might be the same issue.

          We have one large repo with multiple builds configured off it using different branches.

          We use /git/notifyCommit endpoint to trigger builds.

          Pushing one of the branches causes a double build.

          I haven't narrowed down a repro yet, but will add more here as I figure it out.

          Xavier Shay added a comment - I am seeing something very similar to this, I suspect it might be the same issue. We have one large repo with multiple builds configured off it using different branches. We use /git/notifyCommit endpoint to trigger builds. Pushing one of the branches causes a double build. I haven't narrowed down a repro yet, but will add more here as I figure it out.

          Mark Waite added a comment -

          If you're using fast remote polling (the default), and if your "Branches to build" setting matches more than one branch in the repository, then you might consider upgrading to git-client-plugin 1.10.0 and modifying your "Branches to build" to use the "refs/heads/<branchName>" syntax rather than the "origin/master" syntax. The git-client-plugin 1.10.0 release adds a way to unambiguously specify a branch name, while still allowing the earlier syntax and its related ambiguity.

          Refer to Alexander Link's summary for more details.

          Mark Waite added a comment - If you're using fast remote polling (the default), and if your "Branches to build" setting matches more than one branch in the repository, then you might consider upgrading to git-client-plugin 1.10.0 and modifying your "Branches to build" to use the "refs/heads/<branchName>" syntax rather than the "origin/master" syntax. The git-client-plugin 1.10.0 release adds a way to unambiguously specify a branch name, while still allowing the earlier syntax and its related ambiguity. Refer to Alexander Link's summary for more details.

          Seems to be still happening on Jenkins ver. 1.609.3 - and I am running git-client-plugin 1.19.0 and specify the branches as "ref/heads/<branchName>", and I use polling mechanism to trigger the build of a private repository hosted on GitHub (as GitHub push notifications do not work for some reason). Seems to be really caused by the fact that the repository is a really large one ( >850 Mb) and we need to wipe out the workspace prior to the new checkout - so it is essentially a new git clone every time the job is triggered.

          Roman Revyakin added a comment - Seems to be still happening on Jenkins ver. 1.609.3 - and I am running git-client-plugin 1.19.0 and specify the branches as "ref/heads/<branchName>", and I use polling mechanism to trigger the build of a private repository hosted on GitHub (as GitHub push notifications do not work for some reason). Seems to be really caused by the fact that the repository is a really large one ( >850 Mb) and we need to wipe out the workspace prior to the new checkout - so it is essentially a new git clone every time the job is triggered.

          Mark Waite added a comment -

          revrom you might consider adding the optional behavior for a reference repository, then place a bare copy of your 850 MB repository on each of the slaves in a known location. The clone will then create pointers to the reference repository, rather than downloading a fresh copy of the repository each time.

          Mark Waite added a comment - revrom you might consider adding the optional behavior for a reference repository, then place a bare copy of your 850 MB repository on each of the slaves in a known location. The clone will then create pointers to the reference repository, rather than downloading a fresh copy of the repository each time.

          Thanks for the hint markewaite, I've enabled the reference repository and it works like charm.

          Roman Revyakin added a comment - Thanks for the hint markewaite , I've enabled the reference repository and it works like charm.

            Unassigned Unassigned
            tokyowizard Frank Lee
            Votes:
            5 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: