Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-2487

"Please wait, getting ready to work" takes very long

    • Icon: Improvement Improvement
    • Resolution: Incomplete
    • Icon: Critical Critical
    • other
    • None
    • Platform: All, OS: All

      We're running quite an extensive setup with dozens of projects and thousands of
      builds for each project. At the moment we're purging build artefacts but keep
      the actual build history+metadata indefinitely. Maybe because of that, startup
      up hudson now takes in excess of 40 minutes. vmstat shows the disk being busy.

      It would be preferable if startup time of hudson were unaffected by the number
      of builds or projects. I'm putting this as "enhancement", but it's really a
      rather important one for us.

          [JENKINS-2487] "Please wait, getting ready to work" takes very long

          ejono added a comment -

          I have about 100 projects, many of which have thousands of archived builds, and
          it takes about 45 minutes to start up as well.

          ejono added a comment - I have about 100 projects, many of which have thousands of archived builds, and it takes about 45 minutes to start up as well.

          ejono added a comment -

          adding myself to the CC list

          ejono added a comment - adding myself to the CC list

              • Issue 3523 has been marked as a duplicate of this issue. ***

          Kohsuke Kawaguchi added a comment - Issue 3523 has been marked as a duplicate of this issue. ***

          bradfritz added a comment -

          adding myself to CC list

          bradfritz added a comment - adding myself to CC list

          R. Tyler Croy added a comment -

          Adding myself to CC

          R. Tyler Croy added a comment - Adding myself to CC

          mdonohue added a comment -
              • Issue 2890 has been marked as a duplicate of this issue. ***

          mdonohue added a comment - Issue 2890 has been marked as a duplicate of this issue. ***

          huybrechts added a comment -

          huybrechts added a comment - See http://n4.nabble.com/Restart-hudson-takes-too-much-time-td386719.html for some tips.

          mdillon added a comment -

          I notice in the referenced thread that Kohsuke mentions refactoring the Hudson core to allow build data to be lazily loaded. I would absolutely love to see that happen.

          Is this JIRA issue the one that that work would be done under?

          mdillon added a comment - I notice in the referenced thread that Kohsuke mentions refactoring the Hudson core to allow build data to be lazily loaded. I would absolutely love to see that happen. Is this JIRA issue the one that that work would be done under?

          I'd love to see this happening too.
          Downstream buildview plugin forexample takes 'ages' with several thousands of builds to go through even when changing the source code to reduce the amount of debug/info generated.

          Andrea Barbieri added a comment - I'd love to see this happening too. Downstream buildview plugin forexample takes 'ages' with several thousands of builds to go through even when changing the source code to reduce the amount of debug/info generated.

          salimfadhley added a comment -

          Bump!

          This is one of the worst aspects of Hudson. Can anybody suggest a work-around? Would simply discarding some historic builds reduce the startup-time? What is Hudson actually doing that could take so long?

          salimfadhley added a comment - Bump! This is one of the worst aspects of Hudson. Can anybody suggest a work-around? Would simply discarding some historic builds reduce the startup-time? What is Hudson actually doing that could take so long?

          Croesus Kall added a comment - - edited

          My setup takes nearly two hours to start. It has more than 100 jobs. It runs on unix, and the iostat and vmstat commands seem to denote minimal activity during startup.
          When fine logging is enabled, the log first gets a long stream of:

          FINE: Queue maintenance started hudson.model.Queue
          

          Croesus Kall added a comment - - edited My setup takes nearly two hours to start. It has more than 100 jobs. It runs on unix, and the iostat and vmstat commands seem to denote minimal activity during startup. When fine logging is enabled, the log first gets a long stream of: FINE: Queue maintenance started hudson.model.Queue

          taksan added a comment -

          We also have an environment with 400+ jobs, 48 slaves and literally hundreds of thousands of builds.
          With this setup, our jenkins takes about 80min to start.

          taksan added a comment - We also have an environment with 400+ jobs, 48 slaves and literally hundreds of thousands of builds. With this setup, our jenkins takes about 80min to start.

          Herman Meerlo added a comment -

          Ok, this is really ridiculous. What on earth is it doing for 45 minutes?

          Herman Meerlo added a comment - Ok, this is really ridiculous. What on earth is it doing for 45 minutes?

          is there any logs to see what it is doing on startup? It seems from the comments that it is recursively finding each build and loading it in to memory. Makes jenkins unscalable from a start up point of view.

          Any action on this?

          martin naughton added a comment - is there any logs to see what it is doing on startup? It seems from the comments that it is recursively finding each build and loading it in to memory. Makes jenkins unscalable from a start up point of view. Any action on this?

          Oleg Nenashev added a comment -

          There's no info to analyze the issue.
          This issue could be caused by plugins, memory overflows, disk throughput, etc.
          As example, my installations with ~100 slaves, 1000 jobs and 50Gb of the build history start within 10 minutes, hence it is not a "system-wide" problem

          Please submit Jenkins startup logs or run your Jenkins installations inside the profiler.

          Oleg Nenashev added a comment - There's no info to analyze the issue. This issue could be caused by plugins, memory overflows, disk throughput, etc. As example, my installations with ~100 slaves, 1000 jobs and 50Gb of the build history start within 10 minutes, hence it is not a "system-wide" problem Please submit Jenkins startup logs or run your Jenkins installations inside the profiler.

          francisdb added a comment -

          And a service that needs 10 minutes to start is ok?

          francisdb added a comment - And a service that needs 10 minutes to start is ok?

          taksan added a comment -

          Just some insight for this issue:

          We had profiled jenkins during our slow startup and found that what was really taking a long time was reading all the builds.

          To work around the problem, we did an exhaustive cleanup and setup a very aggressive policy to clean up old builds (only a few selected jobs keep a longer history).

          We were able to reduce the startup time from 80min to 5 minutes.

          Other things that help: because the problem is due to intensive I/O reading every build file, improving the hardware also helps to alleviate the problem.

          taksan added a comment - Just some insight for this issue: We had profiled jenkins during our slow startup and found that what was really taking a long time was reading all the builds. To work around the problem, we did an exhaustive cleanup and setup a very aggressive policy to clean up old builds (only a few selected jobs keep a longer history). We were able to reduce the startup time from 80min to 5 minutes. Other things that help: because the problem is due to intensive I/O reading every build file, improving the hardware also helps to alleviate the problem.

          Oleg Nenashev added a comment -

          @francisdb
          > And a service that needs 10 minutes to start is ok?
          Many services like DBMS may require several hours to startup, so several minutes are not the showstopper in general.

          Jenkins developers have implemented much features to decrease the data traffic on the startup (lazy loading of builds, delayed tasks, etc.). BTW, Jenkins definitely need to initialize much data objects and to establish connections with nodes, hence it is not possible to nullify the initialization time.

          Any plugin can "contribute" to the startup time, hence it is mandatory to know about the Jenkins configuration to localize your issues.

          Oleg Nenashev added a comment - @francisdb > And a service that needs 10 minutes to start is ok? Many services like DBMS may require several hours to startup, so several minutes are not the showstopper in general. Jenkins developers have implemented much features to decrease the data traffic on the startup (lazy loading of builds, delayed tasks, etc.). BTW, Jenkins definitely need to initialize much data objects and to establish connections with nodes, hence it is not possible to nullify the initialization time. Any plugin can "contribute" to the startup time, hence it is mandatory to know about the Jenkins configuration to localize your issues.

          >Many services like DBMS may require several hours to startup, so several minutes are not the showstopper in general.

          This statement received in email notification made me recover my password to reply. Seriously? Do you really believe that this should be considered an excuse for this issue?

          Sergey Valkanesko added a comment - >Many services like DBMS may require several hours to startup, so several minutes are not the showstopper in general. This statement received in email notification made me recover my password to reply. Seriously? Do you really believe that this should be considered an excuse for this issue?

          Oleg Nenashev added a comment -

          Nobody excuses for the "issue".
          Many systems (including Jenkins) are slow by design due to the runtime specifics.
          In the case of Jenkins there're known bottlenecks: queue handling, security checks, listeners, etc.
          The performance could be improved step-by-step, but it is not a single issue in the code.

          If you have such strong complains, just bother to provide the info about your installation and to profile your Jenkins installation.
          Any other contributions to Jenkins core or plugins would be appreciated as well.

          Oleg Nenashev added a comment - Nobody excuses for the "issue". Many systems (including Jenkins) are slow by design due to the runtime specifics. In the case of Jenkins there're known bottlenecks: queue handling, security checks, listeners, etc. The performance could be improved step-by-step, but it is not a single issue in the code. If you have such strong complains, just bother to provide the info about your installation and to profile your Jenkins installation. Any other contributions to Jenkins core or plugins would be appreciated as well.

          francisdb added a comment -

          If all builds ever executed need to be read in before jenkins kan show it's main ui this indeed sounds like a design issue. Anyway, I no longer have a big instance running so I can live with the ~1 min startup time we have here.

          francisdb added a comment - If all builds ever executed need to be read in before jenkins kan show it's main ui this indeed sounds like a design issue. Anyway, I no longer have a big instance running so I can live with the ~1 min startup time we have here.

          Oleg Nenashev added a comment -

          @francisdb
          > If all builds ever executed need to be read in before jenkins kan show it's main ui this indeed sounds like a design issue
          This issue has been already solved in 1.485 (JENKINS-8754). However, there were many regression issues caused by the lazy loading, hence it is recommended to use 1.532+
          Now Jenkins loads only the required set if builds. Some plugins may require all builds to be loaded, but it is a topic for additional issues on Jenkins JIRA.

          Oleg Nenashev added a comment - @francisdb > If all builds ever executed need to be read in before jenkins kan show it's main ui this indeed sounds like a design issue This issue has been already solved in 1.485 ( JENKINS-8754 ). However, there were many regression issues caused by the lazy loading, hence it is recommended to use 1.532+ Now Jenkins loads only the required set if builds. Some plugins may require all builds to be loaded, but it is a topic for additional issues on Jenkins JIRA.

          Jesse Glick added a comment -

          This issue is too vague and general to be worth keeping open. If there are specific startup performance problems reproducible in current versions of Jenkins they should be filed with details.

          Jesse Glick added a comment - This issue is too vague and general to be worth keeping open. If there are specific startup performance problems reproducible in current versions of Jenkins they should be filed with details.

          Daniel Beck added a comment -

          Daniel Beck added a comment - Relevant information for filing performance-related issues can be found here: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue https://wiki.jenkins-ci.org/display/JENKINS/Obtaining+a+thread+dump

          Paul Draper added a comment - - edited

          @huybrechts, http://n4.nabble.com/Restart-hudson-takes-too-much-time-td386719.html is dead.

          @Jesse this is pretty specific. Create "dozens of projects and thousands of builds" (as per the issue description) and then start Jenkins.
          Large numbers of projects and builds slowing down Jenkins startup has been corroborated by many people on this issue.

          Are you not able to reproduce this issue?
          What details do you lack?

          Paul Draper added a comment - - edited @huybrechts, http://n4.nabble.com/Restart-hudson-takes-too-much-time-td386719.html is dead. @Jesse this is pretty specific. Create "dozens of projects and thousands of builds" (as per the issue description) and then start Jenkins. Large numbers of projects and builds slowing down Jenkins startup has been corroborated by many people on this issue. Are you not able to reproduce this issue? What details do you lack?

          Daniel Beck added a comment -

          has been corroborated by many people on this issue.

          It's less clear if you consider that Jenkins implemented lazy loading of build records from Oct 2012 to some time in early/mid 2013 (until the major issues were worked out).

          Then there's the case that some plugins have a tendency to require loading all build records, which defeats this optimization.

          What details do you lack?

          Stack traces would be helpful. Lists of installed plugins. Maybe even Support Core plugin support bundles. And of course everything with recent Jenkins versions only.

          And even then, at least my expectation is that many different causes contribute to a long startup time, which will need to be fixed one by one rather than gathered in a catch-all issue.

          Daniel Beck added a comment - has been corroborated by many people on this issue. It's less clear if you consider that Jenkins implemented lazy loading of build records from Oct 2012 to some time in early/mid 2013 (until the major issues were worked out). Then there's the case that some plugins have a tendency to require loading all build records, which defeats this optimization. What details do you lack? Stack traces would be helpful. Lists of installed plugins. Maybe even Support Core plugin support bundles. And of course everything with recent Jenkins versions only. And even then, at least my expectation is that many different causes contribute to a long startup time, which will need to be fixed one by one rather than gathered in a catch-all issue.

            Unassigned Unassigned
            blackmuzzle blackmuzzle
            Votes:
            13 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated:
              Resolved: