Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23415

2 Jobs w/same repo. Job triggers twice when first triggered job is Git SCM is still downloading

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • CentOS 5, JDK 1.7

      Problem:
      A job keeps triggering when it should not.

      Background:
      2 separate jobs are configured to be triggered on the same Github repository depending on branch that pushed to.

      When pushing to the 1st branch, "origin/test/branch1", the 1st job is triggered. Good.
      When pushing to the 2nd branch, "origin/test/branch2", the 2nd job triggers. Good.
      But the 1st job is triggered again. Bad. There's only been 2 pushes.

      The repository is very large, so the git clone takes a few minutes for both jobs.

      The problem consistently occurs when pushing to the 1st branch, THEN immediately pushing to the 2nd branch. The 1st job is triggered and git cloning. And the 2nd job is triggered and queued. BUT another 1st job is triggered and queued.

      Seems to happen when it's still downloading during the clone.

      That extra triggered job is causing us grief, because it's used for deployments.

      This is reproducible consistently with this script:

      ----------------------------
      #!/bin/bash
      pushd mydirectory
      git checkout develop
      echo ${1}d >> d.txt
      git add .
      git commit -m "${1}d"
      git push

      git checkout test/branch1
      git pull --no-edit upstream develop
      git push

      git checkout test/branch2
      git pull --no-edit upstream test/branch1
      git push
      popd
      ----------------------------

      And run it as:

      1. ./script 1
      2. ./script 2
        etc...

      Jenkins v1.558
      Git Client v1.9.1
      Git plugin v2.2.1
      GitHub API v1.54
      Github OAuth v0.14
      GitHub Pull Request Builder 1.12
      GitHub plugin v1.8

          [JENKINS-23415] 2 Jobs w/same repo. Job triggers twice when first triggered job is Git SCM is still downloading

          Mark Waite added a comment -

          I attempted to duplicate the bug and was unsuccessful. I'll need more information before further investigation is performed.

          1. I created an initial repository with the following structure of commits:
            * 3bcea79 (upstream/test/main, test/main) Adapt README for test/main branch
            | * b3b7c59 (upstream/test/branch2, test/branch2) Adapt README for test/branch2
            |/
            | * 4f038d3 (upstream/test/branch1, test/branch1) Adapt README for the test/branch1 branch
            |/
            | * 5ccca9d (HEAD, upstream/develop, develop) Updated README for the develop branch
            |/
            * ea8c87b (upstream/master, master) Add README
            
          2. I created a script in the parent directory as you listed.
          3. I created a Jenkins job monitoring upstream/test/branch1, polling every 2 minutes, confirmed it was checking out correct branch
          4. I created a Jenkins job monitoring upstream/test/branch2, polling every 2 minutes, confirmed it was checking out correct branch
          5. I ran the script with argument 1
          6. I confirmed the Jenkins job on test/branch1 ran and the Jenkins job on test/branch2 did not run

          After that, I tried further by defining a hook in the git repo which caused the Jenkins jobs to poll each time there was a commit to the repo. The jobs when polled as a result of the hook also behaved as expected. The job monitoring test/branch1 ran each time there was a commit to test/branch1. The job monitoring test/branch2 did not run when there was a commit to test/branch1.

          Can you provide more details of your configuration (job definitions, etc.) and confirm that the same problem can be duplicated on another Jenkins server?

          Can you update your git client plugin from 1.9.0 to the currently released 1.9.1, in case there is some relevant difference between the version you're using and the version I'm using?

          Mark Waite added a comment - I attempted to duplicate the bug and was unsuccessful. I'll need more information before further investigation is performed. I created an initial repository with the following structure of commits: * 3bcea79 (upstream/test/main, test/main) Adapt README for test/main branch | * b3b7c59 (upstream/test/branch2, test/branch2) Adapt README for test/branch2 |/ | * 4f038d3 (upstream/test/branch1, test/branch1) Adapt README for the test/branch1 branch |/ | * 5ccca9d (HEAD, upstream/develop, develop) Updated README for the develop branch |/ * ea8c87b (upstream/master, master) Add README I created a script in the parent directory as you listed. I created a Jenkins job monitoring upstream/test/branch1, polling every 2 minutes, confirmed it was checking out correct branch I created a Jenkins job monitoring upstream/test/branch2, polling every 2 minutes, confirmed it was checking out correct branch I ran the script with argument 1 I confirmed the Jenkins job on test/branch1 ran and the Jenkins job on test/branch2 did not run After that, I tried further by defining a hook in the git repo which caused the Jenkins jobs to poll each time there was a commit to the repo. The jobs when polled as a result of the hook also behaved as expected. The job monitoring test/branch1 ran each time there was a commit to test/branch1. The job monitoring test/branch2 did not run when there was a commit to test/branch1. Can you provide more details of your configuration (job definitions, etc.) and confirm that the same problem can be duplicated on another Jenkins server? Can you update your git client plugin from 1.9.0 to the currently released 1.9.1, in case there is some relevant difference between the version you're using and the version I'm using?

          Mark Waite added a comment - - edited

          Jira reported that you had changed the definition of the script in your bug report. I took your revised script and tried to duplicate the problem. I still can't duplicate the problem.

          When I execute

           ./script 1
           ./script 2
          

          a single instance of job 1 executes, and a single instance of job 2 executes.

          Can you upload the config.xml file from your jobs/(job-name)/ directory so that I can compare your job definition with mine?

          Mark Waite added a comment - - edited Jira reported that you had changed the definition of the script in your bug report. I took your revised script and tried to duplicate the problem. I still can't duplicate the problem. When I execute ./script 1 ./script 2 a single instance of job 1 executes, and a single instance of job 2 executes. Can you upload the config.xml file from your jobs/(job-name)/ directory so that I can compare your job definition with mine?

          Frank Lee added a comment -

          Thank you for the quick reply!

          Sorry, the branches in my original post with the last "git checkout" and "git pull" was wrong. I've corrected the branches above. It's pulling the "develop" branch into "test/branch1". Then pulling "test/branch1" into "test/branch2".

          Could you try that to see if it's reproducible?

          Not sure if it makes a difference, but instead of polling, I'm using githhub-webhook on the Github repo. So pushes to the Github repo triggers Github to send payloads to the Jenkins server.

          Details on the configuration of the job for branch1 are:

          1. "Discard Old Builds" is checked
          2. "Max # of builds to keep" is set to 50
          3. "Enable project-based security" is set.
          4. "GitHub project" is set with the URL of the github repo.
          5. "HipChat Notifications" is set with the Project Room.
          6. "Restrict where this project can be run" is set with "Label Expression" set to "master".
          7. "Source Code Management" is set with "Git" selected. In that section the following are set:
            • "Repository URL" has the ssh URL from Github.
            • "Credentials" is set to none.
            • "Branch Specifier" is set to "origin/test/branch1"
            • "Additional Behaviours" has "Advanced sub-modules behaviours" set with "Disable submodules processing" checked.
          8. "Build Triggers" has "Build when a change is pushed to GitHub" checked.
          9. "Build Environment" has "Delete workspace before build starts" and "Color ANSI Console Output" both checked.
          10. "Build" has "Execute shell" with a shell script to do stuff for branch1.
          11. The "Post-build Actions" has HipChat and email notification.

          Details for the job for branch2 are the exact same, except for:

          • "Branch Specifier" is set to "origin/test/branch2"

          To add, originally, we wanted only 1 job by using regex as follows:

          • "Branch Specifier" is set to "origin/test/**"

          And in the "Execute shell" have "if GIT_BRANCH=branch1 then do... if GIT_BRANCH=branch2 then do ...", which is more maintainable than maintaining multiple Jenkins jobs.

          But with the 1 job with the regex, we're having the opposite problem. Instead of the job triggers an extra build, it's ignores subsequent triggers when the first build is running. So for example: when we push 2 or more times immediately while the job is running from the first push that triggered it, consecutive jobs won't queue up. The jenkins.log does show Jenkins receiving the webhook and poking the job, but the job just doesn't trigger. This is a separate problem, but I mention it because it may have some relationship to the original problem.

          I'll try duplicating it in a test environment. And try testing 1.9.1.

          Frank Lee added a comment - Thank you for the quick reply! Sorry, the branches in my original post with the last "git checkout" and "git pull" was wrong. I've corrected the branches above. It's pulling the "develop" branch into "test/branch1". Then pulling "test/branch1" into "test/branch2". Could you try that to see if it's reproducible? Not sure if it makes a difference, but instead of polling, I'm using githhub-webhook on the Github repo. So pushes to the Github repo triggers Github to send payloads to the Jenkins server. Details on the configuration of the job for branch1 are: "Discard Old Builds" is checked "Max # of builds to keep" is set to 50 "Enable project-based security" is set. "GitHub project" is set with the URL of the github repo. "HipChat Notifications" is set with the Project Room. "Restrict where this project can be run" is set with "Label Expression" set to "master". "Source Code Management" is set with "Git" selected. In that section the following are set: "Repository URL" has the ssh URL from Github. "Credentials" is set to none. "Branch Specifier" is set to "origin/test/branch1" "Additional Behaviours" has "Advanced sub-modules behaviours" set with "Disable submodules processing" checked. "Build Triggers" has "Build when a change is pushed to GitHub" checked. "Build Environment" has "Delete workspace before build starts" and "Color ANSI Console Output" both checked. "Build" has "Execute shell" with a shell script to do stuff for branch1. The "Post-build Actions" has HipChat and email notification. Details for the job for branch2 are the exact same, except for: "Branch Specifier" is set to "origin/test/branch2" To add, originally, we wanted only 1 job by using regex as follows: "Branch Specifier" is set to "origin/test/**" And in the "Execute shell" have "if GIT_BRANCH=branch1 then do... if GIT_BRANCH=branch2 then do ...", which is more maintainable than maintaining multiple Jenkins jobs. But with the 1 job with the regex, we're having the opposite problem. Instead of the job triggers an extra build, it's ignores subsequent triggers when the first build is running. So for example: when we push 2 or more times immediately while the job is running from the first push that triggered it, consecutive jobs won't queue up. The jenkins.log does show Jenkins receiving the webhook and poking the job, but the job just doesn't trigger. This is a separate problem, but I mention it because it may have some relationship to the original problem. I'll try duplicating it in a test environment. And try testing 1.9.1.

          Frank Lee added a comment -

          Actually, I'd much prefer having the job with the regex work (e.g. queue up builds).

          Frank Lee added a comment - Actually, I'd much prefer having the job with the regex work (e.g. queue up builds).

          Mark Waite added a comment -

          To have the job queue multiple builds, you might try the "allow concurrent builds" check box, in hopes that would not optimize away the second request to build.

          I can't test the github push mechanism because I'm not willing to expose my test Jenkins server on the public internet, and I'm not willing to configure and expose a reverse proxy on the public internet.

          I can only guess that the key difference is between the GitHub push notifications that you're using and the simple git hook based notifications I'm using. I removed the GitHub components from the bug report because I assumed it was generic to the git client plugin. You'll probably need to put them back, or confirm that you can see the problem without requiring a GitHub push notification.

          Mark Waite added a comment - To have the job queue multiple builds, you might try the "allow concurrent builds" check box, in hopes that would not optimize away the second request to build. I can't test the github push mechanism because I'm not willing to expose my test Jenkins server on the public internet, and I'm not willing to configure and expose a reverse proxy on the public internet. I can only guess that the key difference is between the GitHub push notifications that you're using and the simple git hook based notifications I'm using. I removed the GitHub components from the bug report because I assumed it was generic to the git client plugin. You'll probably need to put them back, or confirm that you can see the problem without requiring a GitHub push notification.

          Frank Lee added a comment -

          To have the job queue multiple builds, you might try the "allow concurrent builds" check box, in hopes that would not optimize away the second request to build.

          That's a interesting idea. Unfortunately in this case, the job isn't made to run concurrently, else the concurrent builds will step on each other.

          I removed the GitHub components from the bug report because I assumed it was generic to the git client plugin. You'll probably need to put them back, or confirm that you can see the problem without requiring a GitHub push notification.

          Since it's going through the GitHub component because "Build when a change is pushed to GitHub" is checked and I'm not sure how to reproduce this problem without having GitHub push notification setup, I've added them back.

          Frank Lee added a comment - To have the job queue multiple builds, you might try the "allow concurrent builds" check box, in hopes that would not optimize away the second request to build. That's a interesting idea. Unfortunately in this case, the job isn't made to run concurrently, else the concurrent builds will step on each other. I removed the GitHub components from the bug report because I assumed it was generic to the git client plugin. You'll probably need to put them back, or confirm that you can see the problem without requiring a GitHub push notification. Since it's going through the GitHub component because "Build when a change is pushed to GitHub" is checked and I'm not sure how to reproduce this problem without having GitHub push notification setup, I've added them back.

          Frank Lee added a comment -

          I edited the original post to add one more plugin: "GitHub plugin v1.8".
          I'm guessing the GitHub payloads are triggered with that plugin.
          If there's anything else I can do to move this issue forward, let me know.

          Frank Lee added a comment - I edited the original post to add one more plugin: "GitHub plugin v1.8". I'm guessing the GitHub payloads are triggered with that plugin. If there's anything else I can do to move this issue forward, let me know.

          Frank Lee added a comment -

          I've applied the updates:

          Git Client v1.9.0 -> v1.9.1
          GitHub API v1.44  -> v1.54
          

          I've updated the origin post to reflect the current versions. I'm still seeing the same problem of the first job triggering a second time on the last push

          git checkout test/branch2
          git pull --no-edit upstream test/branch1
          git push
          

          Security is tighten down with a reverse proxy in front of Jenkins. Only Github IP addresses are allowed in as listed in this link: https://help.github.com/articles/what-ip-addresses-does-github-use-that-i-should-whitelist
          Further, there's bot account that is limited to read and build using the "Project-based Matrix Authorization Strategy".

          Frank Lee added a comment - I've applied the updates: Git Client v1.9.0 -> v1.9.1 GitHub API v1.44 -> v1.54 I've updated the origin post to reflect the current versions. I'm still seeing the same problem of the first job triggering a second time on the last push git checkout test/branch2 git pull --no-edit upstream test/branch1 git push Security is tighten down with a reverse proxy in front of Jenkins. Only Github IP addresses are allowed in as listed in this link: https://help.github.com/articles/what-ip-addresses-does-github-use-that-i-should-whitelist Further, there's bot account that is limited to read and build using the "Project-based Matrix Authorization Strategy".

          Xavier Shay added a comment -

          I am seeing something very similar to this, I suspect it might be the same issue.

          We have one large repo with multiple builds configured off it using different branches.

          We use /git/notifyCommit endpoint to trigger builds.

          Pushing one of the branches causes a double build.

          I haven't narrowed down a repro yet, but will add more here as I figure it out.

          Xavier Shay added a comment - I am seeing something very similar to this, I suspect it might be the same issue. We have one large repo with multiple builds configured off it using different branches. We use /git/notifyCommit endpoint to trigger builds. Pushing one of the branches causes a double build. I haven't narrowed down a repro yet, but will add more here as I figure it out.

          Mark Waite added a comment -

          If you're using fast remote polling (the default), and if your "Branches to build" setting matches more than one branch in the repository, then you might consider upgrading to git-client-plugin 1.10.0 and modifying your "Branches to build" to use the "refs/heads/<branchName>" syntax rather than the "origin/master" syntax. The git-client-plugin 1.10.0 release adds a way to unambiguously specify a branch name, while still allowing the earlier syntax and its related ambiguity.

          Refer to Alexander Link's summary for more details.

          Mark Waite added a comment - If you're using fast remote polling (the default), and if your "Branches to build" setting matches more than one branch in the repository, then you might consider upgrading to git-client-plugin 1.10.0 and modifying your "Branches to build" to use the "refs/heads/<branchName>" syntax rather than the "origin/master" syntax. The git-client-plugin 1.10.0 release adds a way to unambiguously specify a branch name, while still allowing the earlier syntax and its related ambiguity. Refer to Alexander Link's summary for more details.

          Seems to be still happening on Jenkins ver. 1.609.3 - and I am running git-client-plugin 1.19.0 and specify the branches as "ref/heads/<branchName>", and I use polling mechanism to trigger the build of a private repository hosted on GitHub (as GitHub push notifications do not work for some reason). Seems to be really caused by the fact that the repository is a really large one ( >850 Mb) and we need to wipe out the workspace prior to the new checkout - so it is essentially a new git clone every time the job is triggered.

          Roman Revyakin added a comment - Seems to be still happening on Jenkins ver. 1.609.3 - and I am running git-client-plugin 1.19.0 and specify the branches as "ref/heads/<branchName>", and I use polling mechanism to trigger the build of a private repository hosted on GitHub (as GitHub push notifications do not work for some reason). Seems to be really caused by the fact that the repository is a really large one ( >850 Mb) and we need to wipe out the workspace prior to the new checkout - so it is essentially a new git clone every time the job is triggered.

          Mark Waite added a comment -

          revrom you might consider adding the optional behavior for a reference repository, then place a bare copy of your 850 MB repository on each of the slaves in a known location. The clone will then create pointers to the reference repository, rather than downloading a fresh copy of the repository each time.

          Mark Waite added a comment - revrom you might consider adding the optional behavior for a reference repository, then place a bare copy of your 850 MB repository on each of the slaves in a known location. The clone will then create pointers to the reference repository, rather than downloading a fresh copy of the repository each time.

          Thanks for the hint markewaite, I've enabled the reference repository and it works like charm.

          Roman Revyakin added a comment - Thanks for the hint markewaite , I've enabled the reference repository and it works like charm.

            Unassigned Unassigned
            tokyowizard Frank Lee
            Votes:
            5 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: