Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48960

Org folders repository folder never runs scans on repository

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • branch-api-plugin
    • None
    • Jenkins 2.101, Folders Plugin 6.3, GitHub Branch Source Plugin 2.3.2

      Recently updated Jenkins and all available plugins, and the Repository folder within our Organization folder stopped scanning completely. 

       

      I went so far as to delete our entire organization structure and re-create it. The organization does scan correctly on changes. The only time the repository level folder scans automatically is upon its initial creation from the creation of the organization. 

       

      Also makes little sense that the repository folder has a non-configurable interval of 1 day, while the organization folder was setup with an interval of 5 minutes. This is how the repo's trigger is defined in its config.xml:

      <spec>H H * * *</spec>
      <interval>86400000</interval>
      

      I manually changed that to match the Org folders config trigger, e.g. 

      <spec>H/12 * * * *</spec>
      <interval>300000</interval>

      and then gave Jenkins a restart. After ~5 minutes, sure enough, it's now scanning correctly. 

       

      So why on earth is the interval timer being set to such a long duration, and so dramatically different than the org's interval timer? 

          [JENKINS-48960] Org folders repository folder never runs scans on repository

          Thomas K added a comment -

          I believe I'm experiencing the same problem. Just to clarify, you have to add this custom interval to each and every repository in the organization folder, correct? Not really a particularly good way to handle things.

          Does anybody know which plugin/plugin version is responsible for this? I'd like to try to downgrade it until this issue is fixed.

          Thomas K added a comment - I believe I'm experiencing the same problem. Just to clarify, you have to add this custom interval to each and every repository in the organization folder, correct? Not really a particularly good way to handle things. Does anybody know which plugin/plugin version is responsible for this? I'd like to try to downgrade it until this issue is fixed.

          thomaskasene we only have one repository at the moment that contains Jenkinsfile's but yes, I'd assume so. Thankfully at least it's only a one time change unless of course you delete the organization and recreate it completely. 

          At the time that it happened I tried downgrading several of the plugins that seemed related, but continued to see the same behavior. I couldn't narrow it down, and so started digging for a workaround instead. 

           

          James Hardwick added a comment - thomaskasene we only have one repository at the moment that contains Jenkinsfile's but yes, I'd assume so. Thankfully at least it's only a one time change unless of course you delete the organization and recreate it completely.  At the time that it happened I tried downgrading several of the plugins that seemed related, but continued to see the same behavior. I couldn't narrow it down, and so started digging for a workaround instead.   

          Thomas K added a comment -

          Hmm yes - I'm fortunate enough to struggle with dozens of repositories per organization folder, so manually updating config.xml for each one isn't feasible; it defeats any reason I had for using the plugin in the first place, as webhooks isn't an option for us just yet.

          Thomas K added a comment - Hmm yes - I'm fortunate enough to struggle with dozens of repositories per organization folder, so manually updating config.xml for each one isn't feasible; it defeats any reason I had for using the plugin in the first place, as webhooks isn't an option for us just yet.

          Thomas K added a comment -

          Hmm - I tried your workaround, and changed it to:

          <com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger plugin="cloudbees-folder@6.3">
              <spec>* * * * *</spec>
              <interval>60000</interval>
          </com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger>

          in an attempt to make it scan every minute or so. It didn't take long for it to revert to this, however:

          <com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger plugin="cloudbees-folder@6.3">
              <spec>H H * * *</spec>
              <interval>86400000</interval>
          </com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger>

          So it doesn't seem like it's a possible workaround for me after all. Did you encounter this problem, hardwickj?

          Thomas K added a comment - Hmm - I tried your workaround, and changed it to: <com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger plugin="cloudbees-folder@6.3"> <spec>* * * * *</spec> <interval>60000</interval> </com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger> in an attempt to make it scan every minute or so. It didn't take long for it to revert to this, however: <com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger plugin="cloudbees-folder@6.3"> <spec>H H * * *</spec> <interval>86400000</interval> </com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger> So it doesn't seem like it's a possible workaround for me after all. Did you encounter this problem, hardwickj ?

          thomaskasene mine is still set. Something is odd between the spec and interval relationship. The first time I tried it, I changed only the interval and it reset like you described. When I copied the spec and interval from the parent org folder, it then remained and worked as expected. 

          James Hardwick added a comment - thomaskasene mine is still set. Something is odd between the spec and interval relationship. The first time I tried it, I changed only the interval and it reset like you described. When I copied the spec and interval from the parent org folder, it then remained and worked as expected. 

          Does it scan after a day or does it just not scan at all?

          And what happens if you set the org folder scan to faster than a day?

          I am wondering if https://github.com/jenkinsci/cloudbees-folder-plugin/blob/master/src/main/java/com/cloudbees/hudson/plugins/folder/computed/PeriodicFolderTrigger.java#L201 is the root cause.

          Basically, every time a PeriodicFolderTrigger is instantiated, it gets its lastTriggered reset to 0.

          If the Org Folder scan is updating the PeriodicFolderTrigger instance, that would set lastTriggered back to 0... which will mean that it will be initialized to a random time in the next day... chances are that will not be before Org Folder wipes the periodic folder trigger again... resetting the whole process.

           

          To confirm my analysis you would need to set the Org Folder to a longer period like 2 days or weekly... then you should see the daily scans and then once the org folder scan kicks in, the 24h interval of any one repo's daily scan should end up being delayed by some random amount before resuming the 24h interval again.

           

          If you can confirm the above, we should be able to find a way to preserve the lastTriggered between runs and prevent the issue.

           

          P.S. Can you file a separate JIRA to allow configuring the periodic trigger of the multibranch projects in an org folder, Thanks in advance

          Stephen Connolly added a comment - Does it scan after a day or does it just not scan at all? And what happens if you set the org folder scan to faster than a day? I am wondering if https://github.com/jenkinsci/cloudbees-folder-plugin/blob/master/src/main/java/com/cloudbees/hudson/plugins/folder/computed/PeriodicFolderTrigger.java#L201  is the root cause. Basically, every time a PeriodicFolderTrigger is instantiated, it gets its lastTriggered reset to 0. If the Org Folder scan is updating the PeriodicFolderTrigger instance, that would set lastTriggered back to 0... which will mean that it will be initialized to a random time in the next day... chances are that will not be before Org Folder wipes the periodic folder trigger again... resetting the whole process.   To confirm my analysis you would need to set the Org Folder to a longer period like 2 days or weekly... then you should see the daily scans and then once the org folder scan kicks in, the 24h interval of any one repo's daily scan should end up being delayed by some random amount before resuming the 24h interval again.   If you can confirm the above, we should be able to find a way to preserve the lastTriggered between runs and prevent the issue.   P.S. Can you file a separate JIRA to allow configuring the periodic trigger of the multibranch projects in an org folder, Thanks in advance

          https://github.com/jenkinsci/branch-api-plugin/blob/dcc0d23a0870058c6138362d3aa017c0d56effc2/src/main/java/jenkins/branch/OrganizationFolder.java#L1429 is where the 1d default is coming from BTW. I am not seeing any code path that would revert it after the repository project has been created

          Stephen Connolly added a comment - https://github.com/jenkinsci/branch-api-plugin/blob/dcc0d23a0870058c6138362d3aa017c0d56effc2/src/main/java/jenkins/branch/OrganizationFolder.java#L1429  is where the 1d default is coming from BTW. I am not seeing any code path that would revert it after the repository project has been created

          Having the same issue (with bitbucket branch source instead of github).
          I also tried to set it through one of the jobs with

          pipelineTriggers([[$class: 'PeriodicFolderTrigger', interval: '5m']]),

          But didn't work either.

          Patrick Ruckstuhl added a comment - Having the same issue (with bitbucket branch source instead of github). I also tried to set it through one of the jobs with pipelineTriggers([ [$class: 'PeriodicFolderTrigger', interval: '5m'] ]), But didn't work either.

          Found an easier workaround than to manually edit all the configs.

          Using the script console, this will change all the triggers to 5 minutes for the folders inside Organizations

          for (f in Jenkins.instance.getAllItems(jenkins.branch.MultiBranchProject.class)) {
            if (f.parent instanceof jenkins.branch.OrganizationFolder) {
          	  f.addTrigger(new com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger("5m"));
            }
          }
          

          Patrick Ruckstuhl added a comment - Found an easier workaround than to manually edit all the configs. Using the script console, this will change all the triggers to 5 minutes for the folders inside Organizations for (f in Jenkins.instance.getAllItems(jenkins.branch.MultiBranchProject.class)) { if (f.parent instanceof jenkins.branch.OrganizationFolder) { f.addTrigger( new com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger( "5m" )); } }

          Andy Airey added a comment -

          tario That workaround does not work for me, also using Bitbucket Branch Source plugin.

          Is there any progress on solving the core of the issue?
          This is kind of blocking.

          Andy Airey added a comment - tario That workaround does not work for me, also using Bitbucket Branch Source plugin. Is there any progress on solving the core of the issue? This is kind of blocking.

          Tristan Cole added a comment -

          aairey I am in the process of setting up a Bitbucket Team Project with the Bitbucket Branch Source plugin and I ran into this issue. I tried tario's fix and it worked for me. Otherwise, you could access the Jenkins server and edit the multibranch job configs manually (read: via script) and reload configuration from disk.

          Tristan Cole added a comment - aairey I am in the process of setting up a Bitbucket Team Project with the Bitbucket Branch Source plugin and I ran into this issue. I tried tario 's fix and it worked for me. Otherwise, you could access the Jenkins server and edit the multibranch job configs manually (read: via script) and reload configuration from disk.

          Jim Klimov added a comment -

          Hello all, I posted a similar issue some time ago (now linked it as a duplicate of this one and closed); while waiting for a better solution, I can offer this workaround: to add a job to regularly rescan MBPs that are inside folders : https://pastebin.com/xKM7CStM

          It still is sub-optimal and has its quirks, but at least seems to work.

          Beware to not set it to too-often, or you could hit Github REST API rate limiting for example.

          Jim Klimov added a comment - Hello all, I posted a similar issue some time ago (now linked it as a duplicate of this one and closed); while waiting for a better solution, I can offer this workaround: to add a job to regularly rescan MBPs that are inside folders : https://pastebin.com/xKM7CStM It still is sub-optimal and has its quirks, but at least seems to work. Beware to not set it to too-often, or you could hit Github REST API rate limiting for example.

          Andy Airey added a comment - - edited

          Tried tario workaround again and got it working this time.

          Was trying jimklimov's one first but that didn't seem to find any jobs for me.
          Also, it requires the GroovyPostBuild plugin or it will not be able to use manager.

          Andy Airey added a comment - - edited Tried tario workaround again and got it working this time. Was trying jimklimov 's one first but that didn't seem to find any jobs for me. Also, it requires the GroovyPostBuild plugin or it will not be able to use manager .

          tario solution works. Would be good to be able to alter from UI. Our polling didn't seem to work and I could not figure out why until I noticed the project frequency was not what I expected. 

          cathleen turner added a comment - tario solution works. Would be good to be able to alter from UI. Our polling didn't seem to work and I could not figure out why until I noticed the project frequency was not what I expected. 

          Still waiting for somebody to try

          To confirm my analysis you would need to set the Org Folder to a longer period like 2 days or weekly... then you should see the daily scans and then once the org folder scan kicks in, the 24h interval of any one repo's daily scan should end up being delayed by some random amount before resuming the 24h interval again.

          So that we can confirm my analysis... with confirmation we can fix...

          Stephen Connolly added a comment - Still waiting for somebody to try To confirm my analysis you would need to set the Org Folder to a longer period like 2 days or weekly... then you should see the daily scans and then once the org folder scan kicks in, the 24h interval of any one repo's daily scan should end up being delayed by some random amount before resuming the 24h interval again. So that we can confirm my analysis... with confirmation we can fix...

          stephenconnolly if it's any help, this is how I recall it initially failing when I reset my Org folder:

          • On a Friday afternoon, recreated the entire Org with a 5 minute scan interval. 
          • On initial scan, all repo's and all branches per repo are correctly scanned. I thought this meant it was fixed at the time. 
          • Monday morning (~2.5 days later) I see that while the org level scans were working, the repo level scans had not, and if I recall correctly, we had activity over that weekend. 

          I know this isn't exactly what you're asking for, but it may help until I (or someone else) can setup a test repo with your parameters. We can't afford to make such changes to our active development environment. 

          James Hardwick added a comment - stephenconnolly if it's any help, this is how I recall it initially failing when I reset my Org folder: On a Friday afternoon, recreated the entire Org with a 5 minute scan interval.  On initial scan, all repo's and all branches per repo are correctly scanned. I thought this meant it was fixed at the time.  Monday morning (~2.5 days later) I see that while the org level scans were working, the repo level scans had not, and if I recall correctly, we had activity over that weekend.  I know this isn't exactly what you're asking for, but it may help until I (or someone else) can setup a test repo with your parameters. We can't afford to make such changes to our active development environment. 

          Patrick Artounian added a comment - - edited

          Yeah this is still an issue, basically a blocker for my team

          Patrick Artounian added a comment - - edited Yeah this is still an issue, basically a blocker for my team

          Craig Dunford added a comment -

          This is an issue for me. Patrick's workaround works for me, but is fairly cumbersome. What needs to be done to get this looked at and fixed. It is a fairly highly voted issue.

          Craig Dunford added a comment - This is an issue for me. Patrick's workaround works for me, but is fairly cumbersome. What needs to be done to get this looked at and fixed. It is a fairly highly voted issue.

          James Hardwick added a comment - - edited

          If it's any help, I had to setup a fresh Jenkins server for a new project. Again using GitHub Organization plugin to scan a repository for all available branches containing a Jenkinsfile. The repository scan worked the first time as expected, and then immediately started reproducing the original issue as described here. So, pretty easy to setup and reproduce as far as I can tell.  

          I ended up just creating a job that contains tario's console script but instead as a pipeline script, so when the scan interval occasionally resets for whatever reason, I could just run the job and it's fixed. A la...

          node {
            for (f in Jenkins.instance.getAllItems(jenkins.branch.MultiBranchProject.class)) {
              if (f.parent instanceof jenkins.branch.OrganizationFolder) {
                f.addTrigger(new com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger("5m"));
              }
            }
          }

          stephenconnolly I'm looking back and trying to understand your original request but not sure it makes sense. Can you break that down with a bit more detail as to just what exactly should be happening? 

          James Hardwick added a comment - - edited If it's any help, I had to setup a fresh Jenkins server for a new project. Again using GitHub Organization plugin to scan a repository for all available branches containing a Jenkinsfile. The repository scan worked the first time as expected, and then immediately started reproducing the original issue as described here. So, pretty easy to setup and reproduce as far as I can tell.   I ended up just creating a job that contains tario 's console script but instead as a pipeline script, so when the scan interval occasionally resets for whatever reason, I could just run the job and it's fixed. A la... node { for (f in Jenkins.instance.getAllItems(jenkins.branch.MultiBranchProject.class)) { if (f.parent instanceof jenkins.branch.OrganizationFolder) { f.addTrigger( new com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger( "5m" )); } } } stephenconnolly I'm looking back and trying to understand your original request but not sure it makes sense. Can you break that down with a bit more detail as to just what exactly should be happening? 

          There are two periodic scans "configured".

          One is user configurable, namely the one at the org folder level. The other is not currently configurable, namely the one at the branch folder level.

          Now the way this is designed is on the basis that you will set up webhooks because "polling must die" to quote a blog post from KK ages back. So the periodic scanning is more to cover two cases:

          1. To allow controlled pruning of orphaned items. For technical reasons, the code was actually written originally to allow multiple sources in the same org folder. So you could have one source be a github organization and another be a bitbucket team. Then if you moved a repository from the bitbucket team to the github organization the multibranch project would still stay there. This was to mirror the fact that a multibranch project can have multiple sources. However this was too confusing, so we locked the org folders to one source only. In any case, because there originally could be (and for a multibranch project there still can be) multiple things that can claim ownership of a child name, we cannot decide if an event that a name has been deleted by one source will actually result in the deletion of the child. We have to ask all the candidate sources in the priority order: "is this yours?" until we get someone answering "yes"... So the orphaned item strategy only runs with a full scan. This also allows the orphaned item strategy to correctly decide the order in which orphans should be removed.

          2. Events may be dropped or lost. We should find those changes eventually. The scan fills that gap.

          If you have webhooks set up correctly then you only need scanning say once per day or less frequently.

          All the cron triggers store a random offset as a transient field. This random offset is used to decide when in the `H` portion of a trigger rule to actually pick. The periodic folder trigger converts the times into rules like H * * * * (for hourly) or H H * * * (for daily) so this transient field is important that it remain consistent.

          What I think is happening is that the org folder scan is resetting the transient field.

          So if your org folder is scanning every 5 minutes, you will have a 5 in 60 chance that the transient gets reset to a value in the next 5 minutes. Of course we have two H's for the daily scan that the multibranch has. So basically if my theory that the transient is getting reset is correct then there is a 1 in 288 chance that the transient will have a value resulting in a scan during the next 5 minutes... and then the transient gets reset again. Each 5 minute interval gas a 287 in 288 chance of not scanning... so we are naïvely looking at the probability of a multibranch project not scanning being (287/288)^288 = 0.36... it's actually worse, because that is actually the probability of it checking whether a scan is due... and if the transient field for H is being cleared then so too will this one: https://github.com/jenkinsci/cloudbees-folder-plugin/blob/23367538c5c6b405a3ba63d8033ca459aa2f65a0/src/main/java/com/cloudbees/hudson/plugins/folder/computed/PeriodicFolderTrigger.java#L71 and so if we win the 63% lottery then one of those will run through to initialize lastTriggered to a random value https://github.com/jenkinsci/cloudbees-folder-plugin/blob/23367538c5c6b405a3ba63d8033ca459aa2f65a0/src/main/java/com/cloudbees/hudson/plugins/folder/computed/PeriodicFolderTrigger.java#L211-L217

          So if my theory is correct then you would expect:

          • When the org folder is set to scan at a period much shorter than 24h the multibranch will rarely scan.
          • When the org folder is set to scan at a period much longer than 24h the multibranch will scan mostly daily except for the 24h after the org folder scans.

          NOTE: the org folder scan is NOT supposed to trigger the child multibranch scans as that would be sympathetic harmonisation and basically result in everything scanning at the same time. The aim is to level out all scanning across the entire period.

          NOTE: I would like the child multibranch scanning period to be configurable, but the UI for that was not agreed so I couldn't get approval to work on that in my paid time... and my wife vetoed me working on it in my spare time :shrug:

          Stephen Connolly added a comment - There are two periodic scans "configured". One is user configurable, namely the one at the org folder level. The other is not currently configurable, namely the one at the branch folder level. Now the way this is designed is on the basis that you will set up webhooks because "polling must die" to quote a blog post from KK ages back. So the periodic scanning is more to cover two cases: 1. To allow controlled pruning of orphaned items. For technical reasons, the code was actually written originally to allow multiple sources in the same org folder. So you could have one source be a github organization and another be a bitbucket team. Then if you moved a repository from the bitbucket team to the github organization the multibranch project would still stay there. This was to mirror the fact that a multibranch project can have multiple sources. However this was too confusing, so we locked the org folders to one source only. In any case, because there originally could be (and for a multibranch project there still can be) multiple things that can claim ownership of a child name, we cannot decide if an event that a name has been deleted by one source will actually result in the deletion of the child. We have to ask all the candidate sources in the priority order: "is this yours?" until we get someone answering "yes"... So the orphaned item strategy only runs with a full scan. This also allows the orphaned item strategy to correctly decide the order in which orphans should be removed. 2. Events may be dropped or lost. We should find those changes eventually . The scan fills that gap. If you have webhooks set up correctly then you only need scanning say once per day or less frequently. All the cron triggers store a random offset as a transient field. This random offset is used to decide when in the `H` portion of a trigger rule to actually pick. The periodic folder trigger converts the times into rules like H * * * * (for hourly) or H H * * * (for daily) so this transient field is important that it remain consistent. What I think is happening is that the org folder scan is resetting the transient field. So if your org folder is scanning every 5 minutes, you will have a 5 in 60 chance that the transient gets reset to a value in the next 5 minutes. Of course we have two H's for the daily scan that the multibranch has. So basically if my theory that the transient is getting reset is correct then there is a 1 in 288 chance that the transient will have a value resulting in a scan during the next 5 minutes... and then the transient gets reset again. Each 5 minute interval gas a 287 in 288 chance of not scanning... so we are naïvely looking at the probability of a multibranch project not scanning being (287/288)^288 = 0.36... it's actually worse, because that is actually the probability of it checking whether a scan is due... and if the transient field for H is being cleared then so too will this one: https://github.com/jenkinsci/cloudbees-folder-plugin/blob/23367538c5c6b405a3ba63d8033ca459aa2f65a0/src/main/java/com/cloudbees/hudson/plugins/folder/computed/PeriodicFolderTrigger.java#L71 and so if we win the 63% lottery then one of those will run through to initialize lastTriggered to a random value https://github.com/jenkinsci/cloudbees-folder-plugin/blob/23367538c5c6b405a3ba63d8033ca459aa2f65a0/src/main/java/com/cloudbees/hudson/plugins/folder/computed/PeriodicFolderTrigger.java#L211-L217 So if my theory is correct then you would expect: When the org folder is set to scan at a period much shorter than 24h the multibranch will rarely scan. When the org folder is set to scan at a period much longer than 24h the multibranch will scan mostly daily except for the 24h after the org folder scans. NOTE: the org folder scan is NOT supposed to trigger the child multibranch scans as that would be sympathetic harmonisation and basically result in everything scanning at the same time. The aim is to level out all scanning across the entire period. NOTE: I would like the child multibranch scanning period to be configurable, but the UI for that was not agreed so I couldn't get approval to work on that in my paid time... and my wife vetoed me working on it in my spare time :shrug:

          Andy Airey added a comment -

          There's a third case for the need of periodic scanning of underlying multibranch projects; when webhooks don't work (JENKINS-47891).

          Andy Airey added a comment - There's a third case for the need of periodic scanning of underlying multibranch projects; when webhooks don't work ( JENKINS-47891 ).

          aairey then fix your webhooks

          Stephen Connolly added a comment - aairey then fix your webhooks

          Andy Airey added a comment -

          If it were that simple ...
          I guess you did not read the linked issue

          Andy Airey added a comment - If it were that simple ... I guess you did not read the linked issue

          Jim Klimov added a comment -

          Then there's also a problem of the corporate networks. Our build farm may poll github, but won't be permitted to get webhooks or any other messages initiated from the internet, directly or through tunnels.

          Hmm... a case for email-hooks... rss-hooks... out of scope here though.

          Jim Klimov added a comment - Then there's also a problem of the corporate networks. Our build farm may poll github, but won't be permitted to get webhooks or any other messages initiated from the internet, directly or through tunnels. Hmm... a case for email-hooks... rss-hooks... out of scope here though.

          jimklimov so the correct solution to that would be something like the GitHub SNS plugin (if it was updated to replicate events correctly... last time I checked it wasn't )

          aairey The point is that those are not issues for the branch-api plugin itself, they are issues with the SCM sources... or the SCMs themselves... or your environment. Polling is exceedingly expensive in terms of overhead to check and also in terms of delay of notification. Especially when you have an entire organization of repositories to scan. For something like GitHub or Bitbucket Cloud, because of API limits, polling may not even be in any way practicable.

          If you are really stuck, then there are other paths you can take, for example the Apache Software Foundation uses its own custom org folder and event forwarding... https://github.com/stephenc/asf-gitpubsub-jenkins-plugin (also see the video series where I show how this was developed https://www.youtube.com/watch?v=W8BRtJmq_2Y&list=PLvBBnHmZuNQLXcL6CBZPKbX9OstMNIhTV ... note that the video is real time development of the plugin. The only code I wrote was during the video recordings)

          Stephen Connolly added a comment - jimklimov so the correct solution to that would be something like the GitHub SNS plugin (if it was updated to replicate events correctly... last time I checked it wasn't ) aairey The point is that those are not issues for the branch-api plugin itself, they are issues with the SCM sources... or the SCMs themselves... or your environment. Polling is exceedingly expensive in terms of overhead to check and also in terms of delay of notification. Especially when you have an entire organization of repositories to scan. For something like GitHub or Bitbucket Cloud, because of API limits, polling may not even be in any way practicable. If you are really stuck, then there are other paths you can take, for example the Apache Software Foundation uses its own custom org folder and event forwarding... https://github.com/stephenc/asf-gitpubsub-jenkins-plugin (also see the video series where I show how this was developed https://www.youtube.com/watch?v=W8BRtJmq_2Y&list=PLvBBnHmZuNQLXcL6CBZPKbX9OstMNIhTV ... note that the video is real time development of the plugin. The only code I wrote was during the video recordings)

          Andy Airey added a comment -

          Sure stephenconnolly, it's just that you were listing a limited set of use-cases. Wanted to share mine as well since you were trying to create the bigger picture here (which is appreciated).

          I agree with all your points, I wish already for more than a year now that I could use webhooks, but I simply cannot because it does not exist or is not worked on. (These are good reasons to change CI system btw ...).

          I cannot install my own custom plugins on Bitbucket server because I am not in control of that, corporate is and they don't allow us to. They only support the native webhooks.
          So yeah, you can argue that this is a problem within my corporate atmosphere - but I am fairly certain there are more businesses out there using Jenkins that are in a similar situation (be it with firewalls or what have you).

          I understand that you cannot support each and every environment but it comes down to "polling is dead"  - we all wish it were, but it isn't.

          Andy Airey added a comment - Sure stephenconnolly , it's just that you were listing a limited set of use-cases. Wanted to share mine as well since you were trying to create the bigger picture here (which is appreciated). I agree with all your points, I wish already for more than a year now that I could use webhooks, but I simply cannot because it does not exist or is not worked on. (These are good reasons to change CI system btw ...). I cannot install my own custom plugins on Bitbucket server because I am not in control of that, corporate is and they don't allow us to. They only support the native webhooks. So yeah, you can argue that this is a problem within my corporate atmosphere - but I am fairly certain there are more businesses out there using Jenkins that are in a similar situation (be it with firewalls or what have you). I understand that you cannot support each and every environment but it comes down to "polling is dead"  - we all wish it were, but it isn't.

          Yea, we're in the same boat of keeping our Jenkins instance firewall'd off from the outside world, hence the polling. It's not a big deal for us since the Organization being polled has a fairly limited set of repos. 

          Every company I've ever been with has used Jenkins this way, and usually has had very legitimate reasons for doing so. I think it'd be a mistake to have a default worldview of "webhooks everywhere". Just my $.02. 

          James Hardwick added a comment - Yea, we're in the same boat of keeping our Jenkins instance firewall'd off from the outside world, hence the polling. It's not a big deal for us since the Organization being polled has a fairly limited set of repos.  Every company I've ever been with has used Jenkins this way, and usually has had very legitimate reasons for doing so. I think it'd be a mistake to have a default worldview of "webhooks everywhere". Just my $.02. 

          Bryce Schober added a comment -

          Count me as +1 org with a firewalled Jenkins instance that can't currently accept webhooks.

          On that point, does the Jenkins community have a guide for best-security-practices forwarding proxy setup for specific webhooks that orgs could follow?

          Bryce Schober added a comment - Count me as +1 org with a firewalled Jenkins instance that can't currently accept webhooks. On that point, does the Jenkins community have a guide for best-security-practices forwarding proxy setup for specific webhooks that orgs could follow?

          Another +1 org that can't use webhooks.

          Any progress here?

          Anthony Benedetti added a comment - Another +1 org that can't use webhooks. Any progress here?

          Kevin Wren added a comment -

          +1, I also cant use webhooks due to firewalls (and dont have admin access to my jenkins instance). I'm assuming this isnt being worked on after 7 months of low activity?

          As anybody found a workaround in in the repo JenkinsFile?

          Kevin Wren added a comment - +1, I also cant use webhooks due to firewalls (and dont have admin access to my jenkins instance). I'm assuming this isnt being worked on after 7 months of low activity? As anybody found a workaround in in the repo JenkinsFile?

          Jim Borden added a comment -

          Another +1 for firewalls!  Our only remedy to poll and the whole arbitrary 1 day poll on a job quite a ridiculous way to set things up at that.  It took me days to figure out that there were two separate polls going on and left me wondering why I had to constantly click "scan organization now" to get the results I wanted.  

          Jim Borden added a comment - Another +1 for firewalls!  Our only remedy to poll and the whole arbitrary 1 day poll on a job quite a ridiculous way to set things up at that.  It took me days to figure out that there were two separate polls going on and left me wondering why I had to constantly click "scan organization now" to get the results I wanted.  

          Devin Nusbaum added a comment - - edited

          Branch API plugin 2.4.0 added the ability to configure the scan interval for the multibranch projects inside of organization folders (JENKINS-56917), see the "Child Scan Triggers" section of the organization folder configuration screen. Did that address this issue?

          Devin Nusbaum added a comment - - edited Branch API plugin 2.4.0 added the ability to configure the scan interval for the multibranch projects inside of organization folders ( JENKINS-56917 ), see the "Child Scan Triggers" section of the organization folder configuration screen. Did that address this issue?

            Unassigned Unassigned
            hardwickj James Hardwick
            Votes:
            32 Vote for this issue
            Watchers:
            42 Start watching this issue

              Created:
              Updated: