Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-66789

Multibranch pipelines with non-matching triggers disabling matching jobs on push

    • 3.0.2

      After upgrading the Bitbucket Server Integration plugin to 3.0.0 our Mutlibranch Pipeline jobs failed to work. It was like someone disabled all jobs manually, but the "Disable Multibranch Pipeline" button didn't do anything. If we went into the configuration and saved it without any changes, the jobs were active again. After some successful scan/run there were in "disabled" state again. We tried to downgrade our Jenkins engine first from 2.303.1 to 2.289.3 but the problem was still there. After we downgraded the Bitbucket Server Integration plugin from 3.0.0 to 2.1.3 everything was perfect again.

       

      Here are the relevant event logs when this happened:

      Multibranch Pipeline Events
      [Wed Sep 29 22:00:38 CEST 2021] Received com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookConsumer$BitbucketSCMHeadEvent UPDATED event from our_repository with timestamp Wed Sep 29 22:00:38 CEST 2021
      [Wed Sep 29 22:00:38 CEST 2021] com.atlassian.bitbucket.jenkins.internal.trigger.BitbucketWebhookConsumer$BitbucketSCMHeadEvent UPDATED event from m_sw_main with timestamp Wed Sep 29 22:00:38 CEST 2021 processed in 44 ms

       

      And then:

      Started on Sep 29, 2021, 10:14:00 PM
      Build disabled
      Done. Took 0 ms
      No changes

      Replication instructions

      • Create a new Multibranch pipeline project pointing to a remote repository. Ensure that no triggers have been enabled
      • Create a second Multibranch pipeline project pointing to the same repository. Enable the push trigger for this project
      • Push a change to a branch on the repository that both projects have successfully scanned.

      Expected behaviour
      The second job builds as normal, and no behaviour is observed on the first

      Actual behaviour
      The branch the change was made to is disabled on the first project

      Workaround
      There is no workaround for this issue

          [JENKINS-66789] Multibranch pipelines with non-matching triggers disabling matching jobs on push

          Martin Henschke added a comment - - edited

          Hi Gergely, thanks for your response. This appears to be related to the PR trigger feature we added in 3.0.0, but I've run a few local tests and am not replicating the issue you describe.

          Could you provide more details on your Jenkins environment, specifically

          • Any other plugins you might be using
          • Any triggers you have enabled on your job
          • Any other information about the jobs. Is it happening for every multibranch pipeline, and for every branch? If you create a test repo with a very basic jenkinsfile, is the errors still occurring?

          Martin Henschke added a comment - - edited Hi Gergely, thanks for your response. This appears to be related to the PR trigger feature we added in 3.0.0, but I've run a few local tests and am not replicating the issue you describe. Could you provide more details on your Jenkins environment, specifically Any other plugins you might be using Any triggers you have enabled on your job Any other information about the jobs. Is it happening for every multibranch pipeline, and for every branch? If you create a test repo with a very basic jenkinsfile, is the errors still occurring?

          Hi,

          I've listed the installed plugins, please see the attachment..plugin_list.txt

          The jobs are triggered when any change has made on any branch. So basically after every merged PR the jobs are triggered and the problem occured on every branch. I can't test this with a basic Jenkinsfile because we have downgraded the plugin and now it's working fine so we don't want to break it again. This is a production environment you know.

          Gergely Pilisi added a comment - Hi, I've listed the installed plugins, please see the attachment.. plugin_list.txt The jobs are triggered when any change has made on any branch. So basically after every merged PR the jobs are triggered and the problem occured on every branch. I can't test this with a basic Jenkinsfile because we have downgraded the plugin and now it's working fine so we don't want to break it again. This is a production environment you know.

          Hi,
          We also have the same issue, after updating to 3.0.1.
          In two different pipeline, one job was disabled.
          After a "Scan Multibranch Pipeline Now" is was fixed both times. In both jobs te last build before it was disabled was a failure. Maybe there is a connection?

          Marco van Munster added a comment - Hi, We also have the same issue, after updating to 3.0.1. In two different pipeline, one job was disabled. After a "Scan Multibranch Pipeline Now" is was fixed both times. In both jobs te last build before it was disabled was a failure. Maybe there is a connection?

          Thanks for the comment Marco.

          I've managed to perform a local replication with similar symptoms and need to verify the problem I've found matches the one you're seeing. On the job that is having branches disabled, what Bitbucket triggers do you have enabled? Do you have any other jobs in Jenkins that have Bitbucket triggers enabled as well, and if so which ones?

          For context I think I've replicated the issue, and I think it's actually two bugs- the first is Jenkins never actually filters out irrelevant SCMs when processing head events in `MultiBranchProject`, so many unnecessary retrieves are being performed- this seems to be an issue in all versions of the plugin. The second was adding trigger checking in the retrieve method in 3.0.0, which only performs the git operation if the triggers are present- this has the effect that on projects with inapplicable triggers (`BitbucketWebhookMultibranchTrigger.isApplicableForEventType`), the observer believes there are no branches on the repo, and marks any heads in the event as dead.
          I'll update the description with replication instructions and increase the severity. If this behaviour doesn't match what you're seeing, let me know and I'll keep investigating.

          Martin Henschke added a comment - Thanks for the comment Marco. I've managed to perform a local replication with similar symptoms and need to verify the problem I've found matches the one you're seeing. On the job that is having branches disabled, what Bitbucket triggers do you have enabled? Do you have any other jobs in Jenkins that have Bitbucket triggers enabled as well, and if so which ones? For context I think I've replicated the issue, and I think it's actually two bugs- the first is Jenkins never actually filters out irrelevant SCMs when processing head events in `MultiBranchProject`, so many unnecessary retrieves are being performed- this seems to be an issue in all versions of the plugin. The second was adding trigger checking in the retrieve method in 3.0.0, which only performs the git operation if the triggers are present- this has the effect that on projects with inapplicable triggers (`BitbucketWebhookMultibranchTrigger.isApplicableForEventType`), the observer believes there are no branches on the repo, and marks any heads in the event as dead. I'll update the description with replication instructions and increase the severity. If this behaviour doesn't match what you're seeing, let me know and I'll keep investigating.

          Yes, later yesterday I noticed that option was disabled in pipeline, but there was a trigger present in Jenkins. But i was still validating if that solved the problem.

          Marco van Munster added a comment - Yes, later yesterday I noticed that option was disabled in pipeline, but there was a trigger present in Jenkins. But i was still validating if that solved the problem.

            mhenschke_atlassian Martin Henschke
            gpilisi Gergely Pilisi
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: