Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-57588

Branch API 2.5.0 rebuilds all multibranch jobs on first scan after upgrade

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: branch-api-plugin
    • Labels:
      None
    • Environment:
    • Similar Issues:
    • Released As:
      2.5.1

      Description

      I updated to branch API plugin 2.5.0 today. When I scan a multibranch Pipeline (Git branch source or GitHub branch source) the first time after the upgrade to branch API plugin 2.5.0, all jobs in that multibranch Pipeline are started. That means (in my case) that I now have more than 500 jobs in the queue waiting to run.

      The "Scan repository log" of the multibranch job shows:

      Started by user Mark Waite
      [Tue May 21 13:37:54 MDT 2019] Starting branch indexing...
      13:37:55 Connecting to https://api.github.com using MarkEWaite/****** (github personal access token for home.markwaite.net)
      Examining MarkEWaite/jenkins-bugs-private
      
        Checking branches...
      
        Getting remote branches...
      
          Checking branch master
      
        Getting remote pull requests...
            ‘Jenkinsfile’ found
          Met criteria
      Changes detected: master (null → 49101d07a954d083cae4bb0a06aaa081c6ae0fc8)
      Scheduled build for branch: master
      
          Checking branch JENKINS-06203
            ‘Jenkinsfile’ found
          Met criteria
      Changes detected: JENKINS-06203 (null → 6de0c64bb4cc6eb12964479bf37f5bb202f52aa4)
      Scheduled build for branch: JENKINS-06203
      

      Each repository reports that changes are detected in the transition from null to a SHA-1. However, I know that almost every one of the branches in that repository have received no changes since before the update.

      Once the initial scan has completed and the resulting rebuild of all branches is complete, then the "Scan repository log" starts no job and the change detection message says:

      No changes detected: master (still at 49101d07a954d083cae4bb0a06aaa081c6ae0fc8)
      

        Attachments

          Issue Links

            Activity

            markewaite Mark Waite created issue -
            markewaite Mark Waite made changes -
            Field Original Value New Value
            Issue Type Task [ 3 ] Bug [ 1 ]
            markewaite Mark Waite made changes -
            Environment Jenkins 1.76 (selected for next LTS baseline)
            Git plugin 3.10.0
            Git client plugin 2.7.7
            Branch API plugin 2.5.0
            Jenkins 2.176 (selected for next LTS baseline)
            Git plugin 3.10.0
            Git client plugin 2.7.7
            Branch API plugin 2.5.0
            markewaite Mark Waite made changes -
            Environment Jenkins 2.176 (selected for next LTS baseline)
            Git plugin 3.10.0
            Git client plugin 2.7.7
            Branch API plugin 2.5.0
            Jenkins 2.164.3 (latest LTS)
            Git plugin 3.10.0
            Git client plugin 2.7.7
            Branch API plugin 2.5.0

            Jenkins 2.176 (selected for next LTS baseline)
            Git plugin 4.0.0-beta10 pre-release
            Git client plugin 3.0.0-beta10 pre-release
            Branch API plugin 2.5.0
            markewaite Mark Waite made changes -
            Description I updated to branch API plugin 2.5.0 today. When I scan a multibranch Pipeline (Git branch source or GitHub branch source) the first time after the upgrade to branch API plugin 2.5.0, all jobs in that multibranch Pipeline are started. That means (in my case) that I now have more than 500 jobs in the queue waiting to run.

            The "Scan repository log" of the multibranch job shows:

            {noformat}
            Started by user Mark Waite
            [Tue May 21 13:37:54 MDT 2019] Starting branch indexing...
            13:37:55 Connecting to https://api.github.com using MarkEWaite/****** (github personal access token for home.markwaite.net)
            Examining MarkEWaite/jenkins-bugs-private

              Checking branches...

              Getting remote branches...

                Checking branch master

              Getting remote pull requests...
                  ‘Jenkinsfile’ found
                Met criteria
            Changes detected: master (null → 49101d07a954d083cae4bb0a06aaa081c6ae0fc8)
            Scheduled build for branch: master

                Checking branch JENKINS-06203
                  ‘Jenkinsfile’ found
                Met criteria
            Changes detected: JENKINS-06203 (null → 6de0c64bb4cc6eb12964479bf37f5bb202f52aa4)
            Scheduled build for branch: JENKINS-06203
            {noformat}

            Each repository reports that changes are detected in the transition from {{null}} to a SHA-1. However, I know that almost every one of the branches in that repository have received no changes since before the update.
            I updated to branch API plugin 2.5.0 today. When I scan a multibranch Pipeline (Git branch source or GitHub branch source) the first time after the upgrade to branch API plugin 2.5.0, all jobs in that multibranch Pipeline are started. That means (in my case) that I now have more than 500 jobs in the queue waiting to run.

            The "Scan repository log" of the multibranch job shows:

            {noformat}
            Started by user Mark Waite
            [Tue May 21 13:37:54 MDT 2019] Starting branch indexing...
            13:37:55 Connecting to https://api.github.com using MarkEWaite/****** (github personal access token for home.markwaite.net)
            Examining MarkEWaite/jenkins-bugs-private

              Checking branches...

              Getting remote branches...

                Checking branch master

              Getting remote pull requests...
                  ‘Jenkinsfile’ found
                Met criteria
            Changes detected: master (null → 49101d07a954d083cae4bb0a06aaa081c6ae0fc8)
            Scheduled build for branch: master

                Checking branch JENKINS-06203
                  ‘Jenkinsfile’ found
                Met criteria
            Changes detected: JENKINS-06203 (null → 6de0c64bb4cc6eb12964479bf37f5bb202f52aa4)
            Scheduled build for branch: JENKINS-06203
            {noformat}

            Each repository reports that changes are detected in the transition from {{null}} to a SHA-1. However, I know that almost every one of the branches in that repository have received no changes since before the update.

            Once the initial scan has completed and the resulting rebuild of all branches is complete, then the "Scan repository log" starts no job and the change detection message says:

            {noformat}
            No changes detected: master (still at 49101d07a954d083cae4bb0a06aaa081c6ae0fc8)
            {noformat}
            markewaite Mark Waite made changes -
            Summary Branch API 2.5.0 rebuilds all jobs in my multibranch folder Branch API 2.5.0 rebuilds all jobs in multibranch folder on first scan after upgrade
            Hide
            markewaite Mark Waite added a comment -

            Refer to pull request 152 for an initial idea that might prevent the build storm.

            It seems that line 2082 in the diff might not be handling the initial startup case where a job already exists but the lastSeenRevision file has not been written.

            Show
            markewaite Mark Waite added a comment - Refer to pull request 152 for an initial idea that might prevent the build storm. It seems that line 2082 in the diff might not be handling the initial startup case where a job already exists but the lastSeenRevision file has not been written.
            Hide
            bitwiseman Liam Newman added a comment -

            Mark Waite
            Since you were able to repro this do you have an instance where you can try this incremental?
            https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/branch-api/2.5.1-rc752.20712eff7952/

            If trying to repro this on an existing instance that was upgraded, you will of course need to downgrade and then delete the lastSeen files first.

            Show
            bitwiseman Liam Newman added a comment - Mark Waite Since you were able to repro this do you have an instance where you can try this incremental? https://repo.jenkins-ci.org/incrementals/org/jenkins-ci/plugins/branch-api/2.5.1-rc752.20712eff7952/ If trying to repro this on an existing instance that was upgraded, you will of course need to downgrade and then delete the lastSeen files first.
            Hide
            markewaite Mark Waite added a comment - - edited

            That incremental build resolved the issue. Thanks!

            Fixed in branch API release 2.5.1

            Show
            markewaite Mark Waite added a comment - - edited That incremental build resolved the issue. Thanks! Fixed in branch API release 2.5.1
            markewaite Mark Waite made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            markewaite Mark Waite made changes -
            Summary Branch API 2.5.0 rebuilds all jobs in multibranch folder on first scan after upgrade Branch API 2.5.0 rebuilds all multibranch jobs on first scan after upgrade
            bitwiseman Liam Newman made changes -
            Released As 2.5.1
            Resolution Fixed [ 1 ]
            Status In Progress [ 3 ] Resolved [ 5 ]
            bitwiseman Liam Newman made changes -
            Assignee Liam Newman [ bitwiseman ]
            bitwiseman Liam Newman made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            bitwiseman Liam Newman made changes -
            Link This issue is related to JENKINS-57618 [ JENKINS-57618 ]
            bitwiseman Liam Newman made changes -
            Remote Link This issue links to "PR-152 (Web Link)" [ 23008 ]

              People

              Assignee:
              bitwiseman Liam Newman
              Reporter:
              markewaite Mark Waite
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: