Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-23415

2 Jobs w/same repo. Job triggers twice when first triggered job is Git SCM is still downloading

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • git-plugin
    • CentOS 5, JDK 1.7

      Problem:
      A job keeps triggering when it should not.

      Background:
      2 separate jobs are configured to be triggered on the same Github repository depending on branch that pushed to.

      When pushing to the 1st branch, "origin/test/branch1", the 1st job is triggered. Good.
      When pushing to the 2nd branch, "origin/test/branch2", the 2nd job triggers. Good.
      But the 1st job is triggered again. Bad. There's only been 2 pushes.

      The repository is very large, so the git clone takes a few minutes for both jobs.

      The problem consistently occurs when pushing to the 1st branch, THEN immediately pushing to the 2nd branch. The 1st job is triggered and git cloning. And the 2nd job is triggered and queued. BUT another 1st job is triggered and queued.

      Seems to happen when it's still downloading during the clone.

      That extra triggered job is causing us grief, because it's used for deployments.

      This is reproducible consistently with this script:

      ----------------------------
      #!/bin/bash
      pushd mydirectory
      git checkout develop
      echo ${1}d >> d.txt
      git add .
      git commit -m "${1}d"
      git push

      git checkout test/branch1
      git pull --no-edit upstream develop
      git push

      git checkout test/branch2
      git pull --no-edit upstream test/branch1
      git push
      popd
      ----------------------------

      And run it as:

      1. ./script 1
      2. ./script 2
        etc...

      Jenkins v1.558
      Git Client v1.9.1
      Git plugin v2.2.1
      GitHub API v1.54
      Github OAuth v0.14
      GitHub Pull Request Builder 1.12
      GitHub plugin v1.8

          [JENKINS-23415] 2 Jobs w/same repo. Job triggers twice when first triggered job is Git SCM is still downloading

          Frank Lee created issue -

          Mark Waite added a comment -

          I attempted to duplicate the bug and was unsuccessful. I'll need more information before further investigation is performed.

          1. I created an initial repository with the following structure of commits:
            * 3bcea79 (upstream/test/main, test/main) Adapt README for test/main branch
            | * b3b7c59 (upstream/test/branch2, test/branch2) Adapt README for test/branch2
            |/
            | * 4f038d3 (upstream/test/branch1, test/branch1) Adapt README for the test/branch1 branch
            |/
            | * 5ccca9d (HEAD, upstream/develop, develop) Updated README for the develop branch
            |/
            * ea8c87b (upstream/master, master) Add README
            
          2. I created a script in the parent directory as you listed.
          3. I created a Jenkins job monitoring upstream/test/branch1, polling every 2 minutes, confirmed it was checking out correct branch
          4. I created a Jenkins job monitoring upstream/test/branch2, polling every 2 minutes, confirmed it was checking out correct branch
          5. I ran the script with argument 1
          6. I confirmed the Jenkins job on test/branch1 ran and the Jenkins job on test/branch2 did not run

          After that, I tried further by defining a hook in the git repo which caused the Jenkins jobs to poll each time there was a commit to the repo. The jobs when polled as a result of the hook also behaved as expected. The job monitoring test/branch1 ran each time there was a commit to test/branch1. The job monitoring test/branch2 did not run when there was a commit to test/branch1.

          Can you provide more details of your configuration (job definitions, etc.) and confirm that the same problem can be duplicated on another Jenkins server?

          Can you update your git client plugin from 1.9.0 to the currently released 1.9.1, in case there is some relevant difference between the version you're using and the version I'm using?

          Mark Waite added a comment - I attempted to duplicate the bug and was unsuccessful. I'll need more information before further investigation is performed. I created an initial repository with the following structure of commits: * 3bcea79 (upstream/test/main, test/main) Adapt README for test/main branch | * b3b7c59 (upstream/test/branch2, test/branch2) Adapt README for test/branch2 |/ | * 4f038d3 (upstream/test/branch1, test/branch1) Adapt README for the test/branch1 branch |/ | * 5ccca9d (HEAD, upstream/develop, develop) Updated README for the develop branch |/ * ea8c87b (upstream/master, master) Add README I created a script in the parent directory as you listed. I created a Jenkins job monitoring upstream/test/branch1, polling every 2 minutes, confirmed it was checking out correct branch I created a Jenkins job monitoring upstream/test/branch2, polling every 2 minutes, confirmed it was checking out correct branch I ran the script with argument 1 I confirmed the Jenkins job on test/branch1 ran and the Jenkins job on test/branch2 did not run After that, I tried further by defining a hook in the git repo which caused the Jenkins jobs to poll each time there was a commit to the repo. The jobs when polled as a result of the hook also behaved as expected. The job monitoring test/branch1 ran each time there was a commit to test/branch1. The job monitoring test/branch2 did not run when there was a commit to test/branch1. Can you provide more details of your configuration (job definitions, etc.) and confirm that the same problem can be duplicated on another Jenkins server? Can you update your git client plugin from 1.9.0 to the currently released 1.9.1, in case there is some relevant difference between the version you're using and the version I'm using?
          Frank Lee made changes -
          Description Original: Problem:
          A job keeps triggering when it should not.

          Background:
          2 separate jobs are configured to be triggered on the same Github repository depending on branch that pushed to.

          When pushing to the 1st branch, "origin/test/branch1", the 1st job is triggered. Good.
          When pushing to the 2nd branch, "origin/test/branch2", the 2nd job triggers. Good.
          But the 1st job is triggered again. Bad. There's only been 2 pushes.

          The repository is very large, so the git clone takes a few minutes for both jobs.

          The problem consistently occurs when pushing to the 1st branch, THEN immediately pushing to the 2nd branch. The 1st job is triggered and git cloning. And the 2nd job is triggered and queued. BUT another 1st job is triggered and queued.

          Seems to happen when it's still downloading during the clone.

          That extra triggered job is causing us grief, because it's used for deployments.

          This is reproducible consistently with this script:


          ----------------------------
          #!/bin/bash
          pushd mydirectory
            git checkout develop
            echo ${1}d >> d.txt
            git add .
            git commit -m "${1}d"
            git push

          git checkout test/branch1
          git pull --no-edit upstream develop
          git push

          git checkout test/main
          git pull --no-edit upstream test/branch2
          git push
          popd
          ----------------------------

          And run it as:

          # ./script 1
          # ./script 2
          etc...


          Jenkins v1.558
          Git Client v1.9.0
          Git plugin v2.2.1
          GitHub API v1.44
          Github OAuth v0.14
          GitHub Pull Request Builder 1.12
          New: Problem:
          A job keeps triggering when it should not.

          Background:
          2 separate jobs are configured to be triggered on the same Github repository depending on branch that pushed to.

          When pushing to the 1st branch, "origin/test/branch1", the 1st job is triggered. Good.
          When pushing to the 2nd branch, "origin/test/branch2", the 2nd job triggers. Good.
          But the 1st job is triggered again. Bad. There's only been 2 pushes.

          The repository is very large, so the git clone takes a few minutes for both jobs.

          The problem consistently occurs when pushing to the 1st branch, THEN immediately pushing to the 2nd branch. The 1st job is triggered and git cloning. And the 2nd job is triggered and queued. BUT another 1st job is triggered and queued.

          Seems to happen when it's still downloading during the clone.

          That extra triggered job is causing us grief, because it's used for deployments.

          This is reproducible consistently with this script:


          ----------------------------
          #!/bin/bash
          pushd mydirectory
            git checkout develop
            echo ${1}d >> d.txt
            git add .
            git commit -m "${1}d"
            git push

          git checkout test/branch1
          git pull --no-edit upstream develop
          git push

          git checkout test/branch2
          git pull --no-edit upstream test/branch1
          git push
          popd
          ----------------------------

          And run it as:

          # ./script 1
          # ./script 2
          etc...


          Jenkins v1.558
          Git Client v1.9.0
          Git plugin v2.2.1
          GitHub API v1.44
          Github OAuth v0.14
          GitHub Pull Request Builder 1.12
          Mark Waite made changes -
          Component/s Original: github [ 15896 ]
          Component/s Original: github-oauth [ 15900 ]
          Component/s Original: github-api [ 17496 ]
          Labels Original: git jenkins scm New: git scm

          Mark Waite added a comment - - edited

          Jira reported that you had changed the definition of the script in your bug report. I took your revised script and tried to duplicate the problem. I still can't duplicate the problem.

          When I execute

           ./script 1
           ./script 2
          

          a single instance of job 1 executes, and a single instance of job 2 executes.

          Can you upload the config.xml file from your jobs/(job-name)/ directory so that I can compare your job definition with mine?

          Mark Waite added a comment - - edited Jira reported that you had changed the definition of the script in your bug report. I took your revised script and tried to duplicate the problem. I still can't duplicate the problem. When I execute ./script 1 ./script 2 a single instance of job 1 executes, and a single instance of job 2 executes. Can you upload the config.xml file from your jobs/(job-name)/ directory so that I can compare your job definition with mine?

          Frank Lee added a comment -

          Thank you for the quick reply!

          Sorry, the branches in my original post with the last "git checkout" and "git pull" was wrong. I've corrected the branches above. It's pulling the "develop" branch into "test/branch1". Then pulling "test/branch1" into "test/branch2".

          Could you try that to see if it's reproducible?

          Not sure if it makes a difference, but instead of polling, I'm using githhub-webhook on the Github repo. So pushes to the Github repo triggers Github to send payloads to the Jenkins server.

          Details on the configuration of the job for branch1 are:

          1. "Discard Old Builds" is checked
          2. "Max # of builds to keep" is set to 50
          3. "Enable project-based security" is set.
          4. "GitHub project" is set with the URL of the github repo.
          5. "HipChat Notifications" is set with the Project Room.
          6. "Restrict where this project can be run" is set with "Label Expression" set to "master".
          7. "Source Code Management" is set with "Git" selected. In that section the following are set:
            • "Repository URL" has the ssh URL from Github.
            • "Credentials" is set to none.
            • "Branch Specifier" is set to "origin/test/branch1"
            • "Additional Behaviours" has "Advanced sub-modules behaviours" set with "Disable submodules processing" checked.
          8. "Build Triggers" has "Build when a change is pushed to GitHub" checked.
          9. "Build Environment" has "Delete workspace before build starts" and "Color ANSI Console Output" both checked.
          10. "Build" has "Execute shell" with a shell script to do stuff for branch1.
          11. The "Post-build Actions" has HipChat and email notification.

          Details for the job for branch2 are the exact same, except for:

          • "Branch Specifier" is set to "origin/test/branch2"

          To add, originally, we wanted only 1 job by using regex as follows:

          • "Branch Specifier" is set to "origin/test/**"

          And in the "Execute shell" have "if GIT_BRANCH=branch1 then do... if GIT_BRANCH=branch2 then do ...", which is more maintainable than maintaining multiple Jenkins jobs.

          But with the 1 job with the regex, we're having the opposite problem. Instead of the job triggers an extra build, it's ignores subsequent triggers when the first build is running. So for example: when we push 2 or more times immediately while the job is running from the first push that triggered it, consecutive jobs won't queue up. The jenkins.log does show Jenkins receiving the webhook and poking the job, but the job just doesn't trigger. This is a separate problem, but I mention it because it may have some relationship to the original problem.

          I'll try duplicating it in a test environment. And try testing 1.9.1.

          Frank Lee added a comment - Thank you for the quick reply! Sorry, the branches in my original post with the last "git checkout" and "git pull" was wrong. I've corrected the branches above. It's pulling the "develop" branch into "test/branch1". Then pulling "test/branch1" into "test/branch2". Could you try that to see if it's reproducible? Not sure if it makes a difference, but instead of polling, I'm using githhub-webhook on the Github repo. So pushes to the Github repo triggers Github to send payloads to the Jenkins server. Details on the configuration of the job for branch1 are: "Discard Old Builds" is checked "Max # of builds to keep" is set to 50 "Enable project-based security" is set. "GitHub project" is set with the URL of the github repo. "HipChat Notifications" is set with the Project Room. "Restrict where this project can be run" is set with "Label Expression" set to "master". "Source Code Management" is set with "Git" selected. In that section the following are set: "Repository URL" has the ssh URL from Github. "Credentials" is set to none. "Branch Specifier" is set to "origin/test/branch1" "Additional Behaviours" has "Advanced sub-modules behaviours" set with "Disable submodules processing" checked. "Build Triggers" has "Build when a change is pushed to GitHub" checked. "Build Environment" has "Delete workspace before build starts" and "Color ANSI Console Output" both checked. "Build" has "Execute shell" with a shell script to do stuff for branch1. The "Post-build Actions" has HipChat and email notification. Details for the job for branch2 are the exact same, except for: "Branch Specifier" is set to "origin/test/branch2" To add, originally, we wanted only 1 job by using regex as follows: "Branch Specifier" is set to "origin/test/**" And in the "Execute shell" have "if GIT_BRANCH=branch1 then do... if GIT_BRANCH=branch2 then do ...", which is more maintainable than maintaining multiple Jenkins jobs. But with the 1 job with the regex, we're having the opposite problem. Instead of the job triggers an extra build, it's ignores subsequent triggers when the first build is running. So for example: when we push 2 or more times immediately while the job is running from the first push that triggered it, consecutive jobs won't queue up. The jenkins.log does show Jenkins receiving the webhook and poking the job, but the job just doesn't trigger. This is a separate problem, but I mention it because it may have some relationship to the original problem. I'll try duplicating it in a test environment. And try testing 1.9.1.

          Frank Lee added a comment -

          Actually, I'd much prefer having the job with the regex work (e.g. queue up builds).

          Frank Lee added a comment - Actually, I'd much prefer having the job with the regex work (e.g. queue up builds).

          Mark Waite added a comment -

          To have the job queue multiple builds, you might try the "allow concurrent builds" check box, in hopes that would not optimize away the second request to build.

          I can't test the github push mechanism because I'm not willing to expose my test Jenkins server on the public internet, and I'm not willing to configure and expose a reverse proxy on the public internet.

          I can only guess that the key difference is between the GitHub push notifications that you're using and the simple git hook based notifications I'm using. I removed the GitHub components from the bug report because I assumed it was generic to the git client plugin. You'll probably need to put them back, or confirm that you can see the problem without requiring a GitHub push notification.

          Mark Waite added a comment - To have the job queue multiple builds, you might try the "allow concurrent builds" check box, in hopes that would not optimize away the second request to build. I can't test the github push mechanism because I'm not willing to expose my test Jenkins server on the public internet, and I'm not willing to configure and expose a reverse proxy on the public internet. I can only guess that the key difference is between the GitHub push notifications that you're using and the simple git hook based notifications I'm using. I removed the GitHub components from the bug report because I assumed it was generic to the git client plugin. You'll probably need to put them back, or confirm that you can see the problem without requiring a GitHub push notification.
          Frank Lee made changes -
          Labels Original: git scm New: git github scm

          Frank Lee added a comment -

          To have the job queue multiple builds, you might try the "allow concurrent builds" check box, in hopes that would not optimize away the second request to build.

          That's a interesting idea. Unfortunately in this case, the job isn't made to run concurrently, else the concurrent builds will step on each other.

          I removed the GitHub components from the bug report because I assumed it was generic to the git client plugin. You'll probably need to put them back, or confirm that you can see the problem without requiring a GitHub push notification.

          Since it's going through the GitHub component because "Build when a change is pushed to GitHub" is checked and I'm not sure how to reproduce this problem without having GitHub push notification setup, I've added them back.

          Frank Lee added a comment - To have the job queue multiple builds, you might try the "allow concurrent builds" check box, in hopes that would not optimize away the second request to build. That's a interesting idea. Unfortunately in this case, the job isn't made to run concurrently, else the concurrent builds will step on each other. I removed the GitHub components from the bug report because I assumed it was generic to the git client plugin. You'll probably need to put them back, or confirm that you can see the problem without requiring a GitHub push notification. Since it's going through the GitHub component because "Build when a change is pushed to GitHub" is checked and I'm not sure how to reproduce this problem without having GitHub push notification setup, I've added them back.

            Unassigned Unassigned
            tokyowizard Frank Lee
            Votes:
            5 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated: