Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-63344

Job DSL rebuilds all jobs when a folder is created with views inside

      After branch-api version 2.5.5 was released, this change was introduced https://issues.jenkins-ci.org/browse/JENKINS-39682.

       

      Now every-time a XML for a job changes it triggers a re-build of the job. There are a few problems with this. First, I use Bitbucket cloud which has API limits and I have hundreds of jobs in Jenkins. If they all keep triggering scans it breaks everything as the API limits can't are exceeded.

       

      I have made it in a way that the job XMLs themselves very rarely change. They will only change once in a few months if we decide to add a new trait etc... So that part is ok. But, all those hundreds of jobs leave inside folders. 

       

      If the XML of folder change, all jobs inside will trigger scans as well. I don't see a reason for this. I believe it would be beneficial to try to avoid triggering re-scans for child items if the parent changes.

       

      My folders themselves don't rarely change as well. The problem is that we have views inside the folders, and those are making it trigger the child jobs on EVERY seed job run.

       

      Consider this code:

      folder("folderName") {
        views {
          listView("viewName") {
            jobs {
              names(["folderName/test1", "folderName/test2"].toArray(new String[0]))
            }
            recurse()
            columns {
              status()
              weather()
              name()
              lastSuccess()
              lastFailure()
              lastDuration()
              buildButton()
            }
          }
        }
      }
      

      This somehow makes it trigger on every run. The XML of the folder does not actually change, but it trigger scans on every child item anyway.

      Also, considering that a view CAN change often, triggering the child items for this is not a good idea.

      I was hoping this re-scanning could be avoided all together, but if has to stay, at least that folders could behave better and not trigger the child items.

          [JENKINS-63344] Job DSL rebuilds all jobs when a folder is created with views inside

          Liam Newman added a comment -

          michelzanini

          Folder configuration can include data the changes the behavior of the jobs inside that folder.  It might not be possible to know for sure until the job runs. 

           

          This issue was caused by a change to branch-api. One option would be to add a setting (off by default) that uses the old behavior.  I'm not a huge fan of adding more settings, but it is an option.  

          This somehow makes it trigger on every run. The XML of the folder does not actually change, but it trigger scans on every child item anyway.

          That is some pretty bad behavior.  I agree that changing a view should not trigger scans on child items.

          Can you cause this to happen outside of jobDSL?  Manually change a view and it causes scans? 

           

          Liam Newman added a comment - michelzanini Folder configuration can include data the changes the behavior of the jobs inside that folder.  It might not be possible to know for sure until the job runs.    This issue was caused by a change to branch-api. One option would be to add a setting (off by default) that uses the old behavior.  I'm not a huge fan of adding more settings, but it is an option.   This somehow makes it trigger on every run. The XML of the folder does not actually change, but it trigger scans on every child item anyway. That is some pretty bad behavior.  I agree that changing a view should not trigger scans on child items. Can you cause this to happen outside of jobDSL?  Manually change a view and it causes scans?   

          Karol Lassak added a comment -

          michelzanini As I explained in previous ticket we had exactly same issue..

          To debug it I added even log line to show differences (I forgot to create PR to upstream, It would probably help few people to debug it)https://github.com/jenkinsci/job-dsl-plugin/compare/master...ingwarsw:show_diffs?expand=1

          For us issues was "portet_id" that changed with every build..

          But I see you dont have any portrets..

          But maybe some ids changes somewhere there? (It was impossible in our case to find it at XML level)

          In the end of the day we ended up manually configuring everything and not relay on functions provided by DSL..

           

                  dsl.folder(name) {
                      views {
                          dashboardView('Jobs statistics') {
                              recurse()
                              jobs {
                                  regex(viewsToShowRegexp)
                              }
          
                              configure { view ->
                                  view / 'leftPortlets' / 'hudson.plugins.view.dashboard.stats.StatBuilds' {
                                      name 'Build statistics'
                                  }
                                  
                                  view / 'rightPortlets' / 'hudson.plugins.view.dashboard.stats.StatJobs' {
                                      name 'Job statistics'
                                  }
          
                                  view / 'bottomPortlets' / 'hudson.plugins.view.dashboard.core.UnstableJobsPortlet' {
                                      name 'Unstable Jobs'
                                      showOnlyFailedJobs false
                                      recurse false
                                  }
                              }
                          }
                    }
                }
          

           

           

           

          Karol Lassak added a comment - michelzanini As I explained in previous ticket we had exactly same issue.. To debug it I added even log line to show differences (I forgot to create PR to upstream, It would probably help few people to debug it) https://github.com/jenkinsci/job-dsl-plugin/compare/master...ingwarsw:show_diffs?expand=1 For us issues was "portet_id" that changed with every build.. But I see you dont have any portrets.. But maybe some ids changes somewhere there? (It was impossible in our case to find it at XML level) In the end of the day we ended up manually configuring everything and not relay on functions provided by DSL..   dsl.folder(name) { views { dashboardView( 'Jobs statistics' ) { recurse() jobs { regex(viewsToShowRegexp) } configure { view -> view / 'leftPortlets' / 'hudson.plugins.view.dashboard.stats.StatBuilds' { name 'Build statistics' } view / 'rightPortlets' / 'hudson.plugins.view.dashboard.stats.StatJobs' { name 'Job statistics' } view / 'bottomPortlets' / 'hudson.plugins.view.dashboard.core.UnstableJobsPortlet' { name 'Unstable Jobs' showOnlyFailedJobs false recurse false } } } } }      

          Michel Zanini added a comment -

          Hi ingwar,

           

          The XML for the folder does NOT change for me. I compared before and after many times over.

          The code I use is in the description of this ticket. There's no ID or anything else.

           

          I believe what could the problem, could be that when views are inside folders, Job DSL might do two file writes, one to create the folder XML itself, another to add the view inside the XML. So in the end, the XML is always the same, but because there are multiple writes been done, it is seen as a change.

           

          Thanks.

           

          Michel Zanini added a comment - Hi ingwar ,   The XML for the folder does NOT change for me. I compared before and after many times over. The code I use is in the description of this ticket. There's no ID or anything else.   I believe what could the problem, could be that when views are inside folders, Job DSL might do two file writes, one to create the folder XML itself, another to add the view inside the XML. So in the end, the XML is always the same, but because there are multiple writes been done, it is seen as a change.   Thanks.  

          Michel Zanini added a comment -

          Hi bitwiseman,

           

           One option would be to add a setting (off by default) that uses the old behavior.  I'm not a huge fan of adding more settings, but it is an option.  

          This would be extremely helpful for us. A flag to disable what has been done at https://github.com/jenkinsci/branch-api-plugin/pull/158/files would save us.

          The only real change would be to add the AND statement with the flag check to the if Items.currentlyUpdatingByXml().

          Then the flag can be enabled by default, and I could use it to disable this behavior. That would be really good.

           


           

          Can you cause this to happen outside of jobDSL?  Manually change a view and it causes scans? 

          I can replicate via manually changes to the UI as follows:

          1. Check the jobs and see the last time they were scanned - see how they are not at the same time so they are ok.

           

          2. Add a new view inside a particular folder:

           

          3. Run the JOB DSL again and look at the scanning times once more - see how they have just been scanned all together:

           

           

          This happens all the time for ALL jobs for us. We have over 500 jobs! It crashes Bitbucket API limit call. Once we gave up using the views inside the folders it did improve.

          But we did notice that from time to time when upgrading Jenkins and it's plugins this can happen again, even though nothing changed - expect new plugin updates!

          A flag to disable this would make everything so much more stable!

           

          Michel Zanini added a comment - Hi bitwiseman ,    One option would be to add a setting (off by default) that uses the old behavior.  I'm not a huge fan of adding more settings, but it is an option.   This would be extremely helpful for us. A flag to disable what has been done at https://github.com/jenkinsci/branch-api-plugin/pull/158/files  would save us. The only real change would be to add the AND statement with the flag check to the if Items.currentlyUpdatingByXml(). Then the flag can be enabled by default, and I could use it to disable this behavior. That would be really good.     Can you cause this to happen outside of jobDSL?  Manually change a view and it causes scans?  I can replicate via manually changes to the UI as follows: Check the jobs and see the last time they were scanned - see how they are not at the same time so they are ok.   2. Add a new view inside a particular folder:   3. Run the JOB DSL again and look at the scanning times once more - see how they have just been scanned all together:     This happens all the time for ALL jobs for us. We have over 500 jobs! It crashes Bitbucket API limit call. Once we gave up using the views inside the folders it did improve. But we did notice that from time to time when upgrading Jenkins and it's plugins this can happen again, even though nothing changed - expect new plugin updates! A flag to disable this would make everything so much more stable!  

          Michel Zanini added a comment -

          If you agree with adding the flag I can try creating the pull request myself, if you don't get the time to do it.

          Thanks.

          Michel Zanini added a comment - If you agree with adding the flag I can try creating the pull request myself, if you don't get the time to do it. Thanks.

          Michel Zanini added a comment -

          This issue has just happened once more today. We have not changed anything! Just upgraded the Jenkins versions and plugins to the latest, and the seed job just went crazy updating everything for no reason. It reached the API limit and then it can take hours to fix itself and Jenkins is unusable for that time period.

           

          I am going to lock and revert the branch-api plugin back to 2.5.4 we can't afford this issue.

          Michel Zanini added a comment - This issue has just happened once more today. We have not changed anything! Just upgraded the Jenkins versions and plugins to the latest, and the seed job just went crazy updating everything for no reason. It reached the API limit and then it can take hours to fix itself and Jenkins is unusable for that time period.   I am going to lock and revert the branch-api plugin back to 2.5.4 we can't afford this issue.

          Carroll Chiou added a comment -

          michelzanini https://github.com/jenkinsci/branch-api-plugin/pull/249 has been merged, but I don't think that this issue should be closed as the underlying issues still remain.

          Carroll Chiou added a comment - michelzanini https://github.com/jenkinsci/branch-api-plugin/pull/249 has been merged, but I don't think that this issue should be closed as the underlying issues still remain.

            jamietanna Jamie Tanna
            michelzanini Michel Zanini
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: