Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61431

Automatically Scan Bitbucket Server Project for Repositories

      Hi,

      this plugin creates reliable Webhooks in the repositories hosted in a Bitbucket Server Instance, but one has to manually create at least a multibranch project for this to work. However, it would be nice, if there were an option for an automatic scan for each Repo in a Project, like in the old Plugin from CloudBee (which does not create a Webhook reliable (only like once in ~40 retrys) and also searches a new Maintainer, so I wouldn't bet on this horse anymore).

          [JENKINS-61431] Automatically Scan Bitbucket Server Project for Repositories

          Paul Schubert added a comment -

          +1

          This would be a major feature for us and probably many others. We are currently using https://github.com/jenkinsci/bitbucket-branch-source-plugin which has some bugs and is, as stated by Matthias, up for adoption, so we would like to switch to the official plugin by Atlassian. The only feature blocking us right now is this one, scanning Bitbucket-Projects for repositories with Jenkinsfiles and creating Jenkins multi branch pipeline projects for them (like it does now already, but currently only for one specific repo).

          Paul Schubert added a comment - +1 This would be a major feature for us and probably many others. We are currently using https://github.com/jenkinsci/bitbucket-branch-source-plugin which has some bugs and is, as stated by Matthias, up for adoption, so we would like to switch to the official plugin by Atlassian. The only feature blocking us right now is this one, scanning Bitbucket-Projects for repositories with Jenkinsfiles and creating Jenkins multi branch pipeline projects for them (like it does now already, but currently only for one specific repo).

          +1

          Padraig Lennon added a comment - +1

          Martin Karing added a comment -

          +1

          Martin Karing added a comment - +1

          +1

          I currently have something like when the using the bitbucket-branch-source plugin to create multi-branch for all repos in FOO Bitbucket project. Being able to replicate that with the atlassian-bitbucket-server-integration-plugin would be great.

          organizations {
           bitbucket {
             serverUrl("https://bitbucket.example.com:8443")
             repoOwner("FOO")
             credentialsId("bitbucket_password_credential")
             traits {
               bitbucketBranchDiscovery{strategyId(0)}
               bitbucketPullRequestDiscovery{ strategyId(1)}
             }
           }
          }
          
          buildStrategies {
           buildRegularBranches()
           buildChangeRequests{ ignoreTargetOnlyChanges(false) ignoreUntrustedChanges(false) } }
          
          projectFactories {
           workflowMultiBranchProjectFactory{ scriptPath 'Jenkinsfile' scriptPath 'debian/Jenkinsfile' }
          }
          
          triggers { periodicFolderTrigger{ interval('5m') } }
          orphanedItemStrategy { discardOldItems { } }
          
          }
          
          

          Nicholas Brown added a comment - I currently have something like when the using the bitbucket-branch-source plugin to create multi-branch for all repos in FOO Bitbucket project. Being able to replicate that with the atlassian-bitbucket-server-integration-plugin would be great. organizations { bitbucket { serverUrl( "https: //bitbucket.example.com:8443" ) repoOwner( "FOO" ) credentialsId( "bitbucket_password_credential" ) traits { bitbucketBranchDiscovery{strategyId(0)} bitbucketPullRequestDiscovery{ strategyId(1)} } } } buildStrategies { buildRegularBranches() buildChangeRequests{ ignoreTargetOnlyChanges( false ) ignoreUntrustedChanges( false ) } } projectFactories { workflowMultiBranchProjectFactory{ scriptPath 'Jenkinsfile' scriptPath 'debian/Jenkinsfile' } } triggers { periodicFolderTrigger{ interval( '5m' ) } } orphanedItemStrategy { discardOldItems { } } }

          Martin Henschke added a comment - - edited

          Hi Matthias, and everyone who has commented on this issue- thank you for your input and suggestions! We are investigating an implementation of this at the moment.
          For now, we are still exploring what a technical design would look like, but we have a number of questions about the kind of configuration options you will need on such a project. Specifically:

          • Do you store your Jenkinsfile in a consistent location for every repository in a project? Would you expect the plugin to be able to "find" your Jenkinsfile anywhere in the project, or would the restriction of a consistent path be appropriate?
          • When scanning a project, would you want every repository with a Jenkinsfile to be matched, or just a subset? If so, what kind of criteria would you want to include/exclude repositories?
          • What config do you want to be automatically set on each project? How much would need to be individualized for each project?
          • Do you rely on many other plugins, apart from the Bitbucket Jenkins Integration plugin in your multibranch projects? If so, do you need/expect this configuration to be carried over to child projects? Which plugins do you use?
          • And more generally, would you be using this as a migration tool (so you can quickly generate a lot of Jenkinsfiles), for automatically handling job creation on an ongoing basis, or both?

          If you have time to answer these questions, it would be a great help. You can leave comments on this ticket, or by email. Thank you again!

          Martin Henschke added a comment - - edited Hi Matthias, and everyone who has commented on this issue- thank you for your input and suggestions! We are investigating an implementation of this at the moment. For now, we are still exploring what a technical design would look like, but we have a number of questions about the kind of configuration options you will need on such a project. Specifically: Do you store your Jenkinsfile in a consistent location for every repository in a project? Would you expect the plugin to be able to "find" your Jenkinsfile anywhere in the project, or would the restriction of a consistent path be appropriate? When scanning a project, would you want every repository with a Jenkinsfile to be matched, or just a subset? If so, what kind of criteria would you want to include/exclude repositories? What config do you want to be automatically set on each project? How much would need to be individualized for each project? Do you rely on many other plugins, apart from the Bitbucket Jenkins Integration plugin in your multibranch projects? If so, do you need/expect this configuration to be carried over to child projects? Which plugins do you use? And more generally, would you be using this as a migration tool (so you can quickly generate a lot of Jenkinsfiles), for automatically handling job creation on an ongoing basis, or both? If you have time to answer these questions, it would be a great help. You can leave comments on this ticket, or by email. Thank you again!

          Paul Schubert added a comment -

          Hi mhenschke_atlassian ,

          many thanks for asking us before starting development on this topic. Awesome!
          I'll try to provide you short answers. Please don't hesitate do ask further questions.

          1. Do you store your Jenkinsfile in a consistent location for every repository in a project? Would you expect the plugin to be able to "find" your Jenkinsfile anywhere in the project, or would the restriction of a consistent path be appropriate?

          We have the Jenkinsfile in the root of each repository. So we don't expect the Jenkins plugin to find it anywhere. 

          1. When scanning a project, would you want every repository with a Jenkinsfile to be matched, or just a subset? If so, what kind of criteria would you want to include/exclude repositories?

          We do would like to have the posiibility to exclude a repository. So within a project we expext per default every repository containing a Jenkinsfile on root level to be matched, but we would like to be able to exclude specific ones (ideally "configured" inside of the Jenkinsfile or repository).

          1. What config do you want to be automatically set on each project? How much would need to be individualized for each project?

          In this issue we expected to create a Jenkins project with your plugin for each Bitbucket project. So, if you consider providing a "scan" not (only) on Bitbucket project level but on Bitbucket in general, than we only need to select/include/exclude specific Bitbucket projects, nothing more.

          1. Do you rely on many other plugins, apart from the Bitbucket Jenkins Integration plugin in your multibranch projects? If so, do you need/expect this configuration to be carried over to child projects? Which plugins do you use?

          Well we don't really understand this question, but we do not (expect to) use other plugins beside the "standard" plugins. (Of course we need to be able to configure stuff currently configureable like "Bitbucket Server instance", "Clone from" or the "Behaviours" on the top level.

          1. And more generally, would you be using this as a migration tool (so you can quickly generate a lot of Jenkinsfiles), for automatically handling job creation on an ongoing basis, or both?

          We would be using it "for automatically handling job creation on an ongoing basis".

          Many thanks in advance for considering and hopefully takling this issue/feature. If you need any further information, let us know.

          Cheers, Paul

          Paul Schubert added a comment - Hi mhenschke_atlassian , many thanks for asking us before starting development on this topic. Awesome! I'll try to provide you short answers. Please don't hesitate do ask further questions. Do you store your Jenkinsfile in a consistent location for every repository in a project? Would you expect the plugin to be able to "find" your Jenkinsfile anywhere in the project, or would the restriction of a consistent path be appropriate? We have the Jenkinsfile in the root of each repository. So we don't expect the Jenkins plugin to find it anywhere.  When scanning a project, would you want every repository with a Jenkinsfile to be matched, or just a subset? If so, what kind of criteria would you want to include/exclude repositories? We do would like to have the posiibility to exclude a repository. So within a project we expext per default every repository containing a Jenkinsfile on root level to be matched, but we would like to be able to exclude specific ones (ideally "configured" inside of the Jenkinsfile or repository). What config do you want to be automatically set on each project? How much would need to be individualized for each project? In this issue we expected to create a Jenkins project with your plugin for each Bitbucket project. So, if you consider providing a "scan" not (only) on Bitbucket project level but on Bitbucket in general, than we only need to select/include/exclude specific Bitbucket projects, nothing more. Do you rely on many other plugins, apart from the Bitbucket Jenkins Integration plugin in your multibranch projects? If so, do you need/expect this configuration to be carried over to child projects? Which plugins do you use? Well we don't really understand this question, but we do not (expect to) use other plugins beside the "standard" plugins. (Of course we need to be able to configure stuff currently configureable like "Bitbucket Server instance", "Clone from" or the "Behaviours" on the top level. And more generally, would you be using this as a migration tool (so you can quickly generate a lot of Jenkinsfiles), for automatically handling job creation on an ongoing basis, or both? We would be using it "for automatically handling job creation on an ongoing basis". Many thanks in advance for considering and hopefully takling this issue/feature. If you need any further information, let us know. Cheers, Paul

          Pat Bee added a comment -

          mhenschke_atlassian
          Answering each of your questions for the environment I'm currently working in.

          • We have the jenkinsfile in the root of the repo.  (I will note that the other plug in allows you to specify the name and path to the jenkinsfile that you want.)
          • If we don't want a repo to match we tend to rename the jenkinsfile or put it in a subdirectory.  (Ex. "jenkinsfile.xx" )
          • In our application the most common alternate configuration is how often to run the build if not triggered.  Some repos are built daily even if there are no commits.
          • We use ALOT of plugins, but in the context of your question I'm not really sure what is in play at that level.
          • We have 272 projects and 4070 repositories, this is really to allow use to not have to manually set up thousands of repos. Or to avoid having to write some sort of code to automate the set up of all of the repositories.

          Pat Bee added a comment - mhenschke_atlassian Answering each of your questions for the environment I'm currently working in. We have the jenkinsfile in the root of the repo.  (I will note that the other plug in allows you to specify the name and path to the jenkinsfile that you want.) If we don't want a repo to match we tend to rename the jenkinsfile or put it in a subdirectory.  (Ex. "jenkinsfile.xx" ) In our application the most common alternate configuration is how often to run the build if not triggered.  Some repos are built daily even if there are no commits. We use ALOT of plugins, but in the context of your question I'm not really sure what is in play at that level. We have 272 projects and 4070 repositories, this is really to allow use to not have to manually set up thousands of repos. Or to avoid having to write some sort of code to automate the set up of all of the repositories.

          Kevin Huber added a comment -

          Hey mhenschke_atlassian,
          why don't you just adopt https://plugins.jenkins.io/cloudbees-bitbucket-branch-source/ ?
          The plugin looks for a new maintainer and is widely used.
          All the problems you are trying to solve, are already solved by this plugin.

          Kevin Huber added a comment - Hey mhenschke_atlassian , why don't you just adopt https://plugins.jenkins.io/cloudbees-bitbucket-branch-source/ ? The plugin looks for a new maintainer and is widely used. All the problems you are trying to solve, are already solved by this plugin.

          If you extend jenkins.scm.api.SCMNavigator (javadoc), then jenkins.branch.OrganizationFolder (javadoc) can use it. That's how the Bitbucket Branch Source and GitHub Branch Source plugins provide this feature.

          Kalle Niemitalo added a comment - If you extend jenkins.scm.api.SCMNavigator ( javadoc ), then jenkins.branch.OrganizationFolder ( javadoc ) can use it. That's how the Bitbucket Branch Source and GitHub Branch Source plugins provide this feature.

          Pasha Sh. added a comment -

          mhenschke_atlassian Are there any updates around this feature request or maybe a ballpark ETA? Please consider bumping up its priority since it's been pushing users to chose unmaintained cloudbees-bitbucket-branch-source.

          Pasha Sh. added a comment - mhenschke_atlassian  Are there any updates around this feature request or maybe a ballpark ETA? Please consider bumping up its priority since it's been pushing users to chose unmaintained  cloudbees-bitbucket-branch-source.

          Hi pashcan and all the other responders on this ticket, thank you for your comments and sorry I wasn't able to get back to you sooner.

          We've heard your feedback. I can't provide a ballpark ETA at this time, but I hope to target a future release with an implementation of this feature.

          Martin Henschke added a comment - Hi pashcan and all the other responders on this ticket, thank you for your comments and sorry I wasn't able to get back to you sooner. We've heard your feedback. I can't provide a ballpark ETA at this time, but I hope to target a future release with an implementation of this feature.

          Pasha Sh. added a comment -

          Much appreciate your response nonetheless 

          Pasha Sh. added a comment - Much appreciate your response nonetheless 

          Bump, we've had a request for this plugin and aren't able to use it because we need this functionality. Would love to see more features implemented in this plugin as it looks like it has potential to replace multiple other plugins we're using between Jenkins and Bitbucket and it would be great to have a plugin that is supported by Atlassian. Looking forward to an update on the eta!

          Caralee Jackson added a comment - Bump, we've had a request for this plugin and aren't able to use it because we need this functionality. Would love to see more features implemented in this plugin as it looks like it has potential to replace multiple other plugins we're using between Jenkins and Bitbucket and it would be great to have a plugin that is supported by Atlassian. Looking forward to an update on the eta!

          Disappointing that this is still an open issue after years of being open. It's a much needed feature. Ideally we could even scan our entire Bitbucket instance for repositories (i.e. including all projects)

          Jonny van Beek added a comment - Disappointing that this is still an open issue after years of being open. It's a much needed feature. Ideally we could even scan our entire Bitbucket instance for repositories (i.e. including all projects)

          Joe Hewitt added a comment -

          The Cloudbees plugin provides the project level setup for each project (100+ projects) which paired with role matrix plugin allows us to give each team (team = project) self-service capability. From initial setup perspective, we configure the Bitbucket project, Jenkins multi-branch project and permissions for each team when a new project is requested. From that point forward, there is no need for Jenkins administrator to be involved with the teams day-to-day work of adding repos or configuring their build trigger requirements. 

          Without the ability to setup a project that will auto-detect repos/branches, there is a requirement for much more central involvement in regular upkeep of projects in Jenkins. 

          We do have the atlassian-bitbucket-plugin installed to our Jenkins environment as well to take advantage of some of the capabilities is offers like Build integration and rerun from Bitbucket UI. However this becomes confusing as there is overlap in functionality. Once the Atlassian plugin supports project scanning, we can consolidate back down to a single plugin to integrate Bitbucket and Jenkins. 

          Joe Hewitt added a comment - The Cloudbees plugin provides the project level setup for each project (100+ projects) which paired with role matrix plugin allows us to give each team (team = project) self-service capability. From initial setup perspective, we configure the Bitbucket project, Jenkins multi-branch project and permissions for each team when a new project is requested. From that point forward, there is no need for Jenkins administrator to be involved with the teams day-to-day work of adding repos or configuring their build trigger requirements.  Without the ability to setup a project that will auto-detect repos/branches, there is a requirement for much more central involvement in regular upkeep of projects in Jenkins.  We do have the atlassian-bitbucket-plugin installed to our Jenkins environment as well to take advantage of some of the capabilities is offers like Build integration and rerun from Bitbucket UI. However this becomes confusing as there is overlap in functionality. Once the Atlassian plugin supports project scanning, we can consolidate back down to a single plugin to integrate Bitbucket and Jenkins. 

          Jared Kauppila added a comment - I was curious if there will be any movement on this with the `Automate at scale with project-level webhooks` functionality delivered in Bitbucket Data Center 8.8 ?

            mhenschke_atlassian Martin Henschke
            garionion Matthias Detsch
            Votes:
            28 Vote for this issue
            Watchers:
            35 Start watching this issue

              Created:
              Updated: