Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-64253

Deliver Pipeline Load Time increases dramatically as more than 200 jobs are added to the pipeline

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • Jenkins 2.249.2 LTS

      We would like the performance of the Delivery Pipeline to be able to scale dashboards with 500 jobs and load in about 30 seconds. 

       

      About 188 jobs loads in 12 seconds, so 15.6 jobs/second

      About 412 jobs loads in 104 seconds, so 3.9 jobs/second

      Not sure why doubling the number of jobs causes the load time to be 8.6 times slower.

      We love the delivery pipeline with its excellent user interface and features.  However, we are trying to expand our current dashboards to include more environments which increases the number of jobs from around 200 to 450.  Older versions of Jenkins would error with "Error communicating to Server!" errors.  Newer versions of Jenkins do allow the dashboard to eventually complete.

      Attached is a screen recording where one of our dashboards takes over 90 seconds to load. 

       

      We find that Firefox offers multi-CPU rendering to load faster than other browsers, attached screen recordings where taken on Firefox with a 2.4 GHz Intel Core i5 MacBook Pro (13-inch, 2019, Four Thunderbolt 3 ports).

          [JENKINS-64253] Deliver Pipeline Load Time increases dramatically as more than 200 jobs are added to the pipeline

          Ryan Naylor added a comment - - edited

          Work around is to increase the timeout:20000 to timeout:300000 in the plugin file.  This can be done via editing the file in the plugins folder pipe.js...

          https://github.com/Diabol/delivery-pipeline-plugin/blob/master/src/main/webapp/pipe.js 

          Ryan Naylor added a comment - - edited Work around is to increase the timeout:20000 to timeout:300000 in the plugin file.  This can be done via editing the file in the plugins folder pipe.js... https://github.com/Diabol/delivery-pipeline-plugin/blob/master/src/main/webapp/pipe.js  

          Ryan Naylor added a comment -

          patbos Even with the timeout set to 5 minutes, we see this occurring in our main pipeline.  I have job history set to 50 builds to try to keep the plugin working.   This is preventing us from adding additional jobs to our pipeline. 

          I updated to the latest plugins and 2.289.2 LTS Jenkins last night, but this did not improve performance.  I was hopeful because other plugins' release notes indicate performance fixes...

          My team (3 developers) and I are going through the source code on github to see if we can find a performance fix.   

           

          We noticed that the cache: false in the pipe.js call, trying to test out enabling cache..  Also looking for any other timeouts and/or retry logic.

           

          We would love to zoom with you to get a better understanding of this plugin.

          Ryan Naylor added a comment - patbos  Even with the timeout set to 5 minutes, we see this occurring in our main pipeline.  I have job history set to 50 builds to try to keep the plugin working.   This is preventing us from adding additional jobs to our pipeline.  I updated to the latest plugins and 2.289.2 LTS Jenkins last night, but this did not improve performance.  I was hopeful because other plugins' release notes indicate performance fixes... My team (3 developers) and I are going through the source code on github to see if we can find a performance fix.      We noticed that the cache: false in the pipe.js call, trying to test out enabling cache..  Also looking for any other timeouts and/or retry logic.   We would love to zoom with you to get a better understanding of this plugin.

          Ryan Naylor added a comment -

          patbos one other note, the pipeline seems to load faster for users that are directly added to Jenkins as administrators instead of Active Directory.   

          Not sure if their is a way to optimize authentication in the plugin?  

           

          Even with the user added as a Admin the pipeline takes significant time to load...

           

          Ryan Naylor added a comment - patbos  one other note, the pipeline seems to load faster for users that are directly added to Jenkins as administrators instead of Active Directory.    Not sure if their is a way to optimize authentication in the plugin?     Even with the user added as a Admin the pipeline takes significant time to load...  

          David Westd added a comment - - edited

          I also have a vested interest in getting this to load faster. We use this at work extensively and we need better page load times for our users, our developers and production support team.

          Pages sometimes take 50 or more seconds to load, which is a substantial amount of time. Many users report the pages never load for them do to timeout issues.

          That is a shame since this plugin makes for a user experience that is otherwise unparalleled. We don't know any other, better way to manage our releases and control what artifacts get deployed where.

          Generally speaking this begins to be a problem when we hit 3 stages in our pipeline and have large amounts of environments in one stage and many tasks in the stages.

          We are currently using profiling tools to see if we can find the exact bottleneck. 

          David Westd added a comment - - edited I also have a vested interest in getting this to load faster. We use this at work extensively and we need better page load times for our users, our developers and production support team. Pages sometimes take 50 or more seconds to load, which is a substantial amount of time. Many users report the pages never load for them do to timeout issues. That is a shame since this plugin makes for a user experience that is otherwise unparalleled. We don't know any other, better way to manage our releases and control what artifacts get deployed where. Generally speaking this begins to be a problem when we hit 3 stages in our pipeline and have large amounts of environments in one stage and many tasks in the stages. We are currently using profiling tools to see if we can find the exact bottleneck. 

          David Westd added a comment -

          I'm finding that it looks like the majority of time loading the large pipeline is coming from an API call that takes a long time to return JSON. Is there a way to do remote debugging on a running plugin instance for performance profiling?

          David Westd added a comment - I'm finding that it looks like the majority of time loading the large pipeline is coming from an API call that takes a long time to return JSON. Is there a way to do remote debugging on a running plugin instance for performance profiling?

          dbwest naylorr Are you using upstream downstream job pipelines or Jenkins pipelines? 

          dbwest It is possible to remote debug, just attach a profiler or debugger to the Jenkins process.

           

          Patrik Boström added a comment - dbwest naylorr  Are you using upstream downstream job pipelines or Jenkins pipelines?  dbwest  It is possible to remote debug, just attach a profiler or debugger to the Jenkins process.  

            patbos Patrik Boström
            naylorr Ryan Naylor
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated: