Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48996

jenkins.branch.MultiBranchProject disregards quick delete and recreation via branch webhooks.

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • branch-api-plugin
    • None

      Bug:

      When you delete a branch and create it again within seconds (2 seconds) on your remote Bitbucket server, jenkins is not done processing the remove event when the create event comes, preventing it from starting a new job.

      Steps to reproduce:
      create a branch called eg. test/sal and let it run until finish.

      After that, redo the last commit, and delete the remote branch and push again

       git commit -a --amend -C HEAD && date && git push origin :test/sal ; sleep 2 && git push origin test/sal && date

      result:
      Sometimes (every second time, but varying) the trigger is not executed.

      Bad run logs:

      ==> logs/jenkins.branch.MultiBranchProject.log <==
      [Tue Jan 16 08:44:43 UTC 2018] Received com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 REMOVED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:44:43 UTC 2018
      Found match against pipeline-ntdrv
      [Tue Jan 16 08:44:51 UTC 2018] Finished processing com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 REMOVED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:44:43 UTC 2018. Matched 1.
      
      ==> logs/jenkins.branch.MultiBranchProject.log <==
      [Tue Jan 16 08:44:48 UTC 2018] Received com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 CREATED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:44:48 UTC 2018
      [Tue Jan 16 08:44:48 UTC 2018] Finished processing com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 CREATED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:44:48 UTC 2018. Matched 0.
      
      

      Good

      ==> logs/jenkins.branch.MultiBranchProject.log <==
      [Tue Jan 16 08:46:35 UTC 2018] Received com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 REMOVED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:46:35 UTC 2018
      Found match against pipeline-ntdrv
      [Tue Jan 16 08:46:43 UTC 2018] Finished processing com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 REMOVED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:46:35 UTC 2018. Matched 1.
      
      ==> logs/jenkins.branch.MultiBranchProject.log <==
      [Tue Jan 16 08:46:39 UTC 2018] Received com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 CREATED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:46:39 UTC 2018
      Found match against pipeline-ntdrv (resurrect branch test/sal-cd382)
      [Tue Jan 16 08:46:48 UTC 2018] Finished processing com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 CREATED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:46:39 UTC 2018. Matched 1.
      
      

      Notice the timestamps for "Received/Finished".
      In the bad example, REMOVED event processing does not finish until after the CREATED event is started and finished. In the good example, the REMOVED event does finish after CREATED starts, but before it finishes. This does indeed allow for a scenario where the processing of the CREATED event races with the REMOVED event.

       

          [JENKINS-48996] jenkins.branch.MultiBranchProject disregards quick delete and recreation via branch webhooks.

          Sofus Albertsen created issue -
          Sofus Albertsen made changes -
          Description Original: Bug:

          When you delete a branch and create it again within seconds (2 seconds) on your remote Bitbucket server, jenkins is not done processing the remove event when the create event comes, preventing it from starting a new job.
           
          Steps to reproduce:
          create a branch called eg. test/sal and let it run until finish.

          After that, redo the last commit, and delete the remote branch and push again
          {noformat}
           git commit -a --amend -C HEAD && date && git push origin :test/sal ; sleep 2 && git push origin test/sal && date{noformat}

          result:
          Sometimes (every second time, but varying) the trigger is not executed.

          Bad run logs:

          {noformat}
          ==> logs/jenkins.branch.MultiBranchProject.log <==
          [Tue Jan 16 08:44:43 UTC 2018] Received com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 REMOVED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:44:43 UTC 2018
          Found match against pipeline-ntdrv
          [Tue Jan 16 08:44:51 UTC 2018] Finished processing com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 REMOVED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:44:43 UTC 2018. Matched 1.

          ==> logs/jenkins.branch.MultiBranchProject.log <==
          [Tue Jan 16 08:44:48 UTC 2018] Received com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 CREATED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:44:48 UTC 2018
          [Tue Jan 16 08:44:48 UTC 2018] Finished processing com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 CREATED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:44:48 UTC 2018. Matched 0.

          {noformat}


          Good


          {noformat}
          ==> logs/jenkins.branch.MultiBranchProject.log <==
          [Tue Jan 16 08:46:35 UTC 2018] Received com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 REMOVED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:46:35 UTC 2018
          Found match against pipeline-ntdrv
          [Tue Jan 16 08:46:43 UTC 2018] Finished processing com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 REMOVED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:46:35 UTC 2018. Matched 1.

          ==> logs/jenkins.branch.MultiBranchProject.log <==
          [Tue Jan 16 08:46:39 UTC 2018] Received com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 CREATED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:46:39 UTC 2018
          Found match against pipeline-ntdrv (resurrect branch test/sal-cd382)
          [Tue Jan 16 08:46:48 UTC 2018] Finished processing com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 CREATED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:46:39 UTC 2018. Matched 1.

          {noformat}

          Notice the timestamps for "Received/Finished".
          In the bad example, REMOVED event processing does not finish until after the CREATED event is started and finished. In the good example, the REMOVED event does finish after CREATED starts, but before it finishes. This does indeed allow for a scenario where the processing of the CREATED event races with the REMOVED event.

          I would argue that the race is not in Jenkins, but would instead interpret the logs as if the cloudbees Bitbucket plugin misuse an asynchronous API, blindly calling the API on every event it receives rather than serializing them on the "Finish" callback/event to guarantee their order.
          New: Bug:

          When you delete a branch and create it again within seconds (2 seconds) on your remote Bitbucket server, jenkins is not done processing the remove event when the create event comes, preventing it from starting a new job.

          Steps to reproduce:
           create a branch called eg. test/sal and let it run until finish.

          After that, redo the last commit, and delete the remote branch and push again
          {noformat}
           git commit -a --amend -C HEAD && date && git push origin :test/sal ; sleep 2 && git push origin test/sal && date{noformat}
          result:
           Sometimes (every second time, but varying) the trigger is not executed.

          Bad run logs:
          {noformat}
          ==> logs/jenkins.branch.MultiBranchProject.log <==
          [Tue Jan 16 08:44:43 UTC 2018] Received com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 REMOVED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:44:43 UTC 2018
          Found match against pipeline-ntdrv
          [Tue Jan 16 08:44:51 UTC 2018] Finished processing com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 REMOVED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:44:43 UTC 2018. Matched 1.

          ==> logs/jenkins.branch.MultiBranchProject.log <==
          [Tue Jan 16 08:44:48 UTC 2018] Received com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 CREATED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:44:48 UTC 2018
          [Tue Jan 16 08:44:48 UTC 2018] Finished processing com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 CREATED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:44:48 UTC 2018. Matched 0.

          {noformat}
          Good
          {noformat}
          ==> logs/jenkins.branch.MultiBranchProject.log <==
          [Tue Jan 16 08:46:35 UTC 2018] Received com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 REMOVED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:46:35 UTC 2018
          Found match against pipeline-ntdrv
          [Tue Jan 16 08:46:43 UTC 2018] Finished processing com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 REMOVED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:46:35 UTC 2018. Matched 1.

          ==> logs/jenkins.branch.MultiBranchProject.log <==
          [Tue Jan 16 08:46:39 UTC 2018] Received com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 CREATED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:46:39 UTC 2018
          Found match against pipeline-ntdrv (resurrect branch test/sal-cd382)
          [Tue Jan 16 08:46:48 UTC 2018] Finished processing com.cloudbees.jenkins.plugins.bitbucket.hooks.PushHookProcessor$1 CREATED event from 10.240.2.28 ⇒ https://jenkins.acme.com:8080/dsw/bitbucket-scmsource-hook/notify with timestamp Tue Jan 16 08:46:39 UTC 2018. Matched 1.

          {noformat}
          Notice the timestamps for "Received/Finished".
           In the bad example, REMOVED event processing does not finish until after the CREATED event is started and finished. In the good example, the REMOVED event does finish after CREATED starts, but before it finishes. This does indeed allow for a scenario where the processing of the CREATED event races with the REMOVED event.

           
          Stephen Connolly made changes -
          Assignee Original: Stephen Connolly [ stephenconnolly ]

            Unassigned Unassigned
            sofusalbertsen Sofus Albertsen
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: