Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-2487

"Please wait, getting ready to work" takes very long

    • Icon: Improvement Improvement
    • Resolution: Incomplete
    • Icon: Critical Critical
    • other
    • None
    • Platform: All, OS: All

      We're running quite an extensive setup with dozens of projects and thousands of
      builds for each project. At the moment we're purging build artefacts but keep
      the actual build history+metadata indefinitely. Maybe because of that, startup
      up hudson now takes in excess of 40 minutes. vmstat shows the disk being busy.

      It would be preferable if startup time of hudson were unaffected by the number
      of builds or projects. I'm putting this as "enhancement", but it's really a
      rather important one for us.

          [JENKINS-2487] "Please wait, getting ready to work" takes very long

          taksan added a comment -

          Just some insight for this issue:

          We had profiled jenkins during our slow startup and found that what was really taking a long time was reading all the builds.

          To work around the problem, we did an exhaustive cleanup and setup a very aggressive policy to clean up old builds (only a few selected jobs keep a longer history).

          We were able to reduce the startup time from 80min to 5 minutes.

          Other things that help: because the problem is due to intensive I/O reading every build file, improving the hardware also helps to alleviate the problem.

          taksan added a comment - Just some insight for this issue: We had profiled jenkins during our slow startup and found that what was really taking a long time was reading all the builds. To work around the problem, we did an exhaustive cleanup and setup a very aggressive policy to clean up old builds (only a few selected jobs keep a longer history). We were able to reduce the startup time from 80min to 5 minutes. Other things that help: because the problem is due to intensive I/O reading every build file, improving the hardware also helps to alleviate the problem.

          Oleg Nenashev added a comment -

          @francisdb
          > And a service that needs 10 minutes to start is ok?
          Many services like DBMS may require several hours to startup, so several minutes are not the showstopper in general.

          Jenkins developers have implemented much features to decrease the data traffic on the startup (lazy loading of builds, delayed tasks, etc.). BTW, Jenkins definitely need to initialize much data objects and to establish connections with nodes, hence it is not possible to nullify the initialization time.

          Any plugin can "contribute" to the startup time, hence it is mandatory to know about the Jenkins configuration to localize your issues.

          Oleg Nenashev added a comment - @francisdb > And a service that needs 10 minutes to start is ok? Many services like DBMS may require several hours to startup, so several minutes are not the showstopper in general. Jenkins developers have implemented much features to decrease the data traffic on the startup (lazy loading of builds, delayed tasks, etc.). BTW, Jenkins definitely need to initialize much data objects and to establish connections with nodes, hence it is not possible to nullify the initialization time. Any plugin can "contribute" to the startup time, hence it is mandatory to know about the Jenkins configuration to localize your issues.

          >Many services like DBMS may require several hours to startup, so several minutes are not the showstopper in general.

          This statement received in email notification made me recover my password to reply. Seriously? Do you really believe that this should be considered an excuse for this issue?

          Sergey Valkanesko added a comment - >Many services like DBMS may require several hours to startup, so several minutes are not the showstopper in general. This statement received in email notification made me recover my password to reply. Seriously? Do you really believe that this should be considered an excuse for this issue?

          Oleg Nenashev added a comment -

          Nobody excuses for the "issue".
          Many systems (including Jenkins) are slow by design due to the runtime specifics.
          In the case of Jenkins there're known bottlenecks: queue handling, security checks, listeners, etc.
          The performance could be improved step-by-step, but it is not a single issue in the code.

          If you have such strong complains, just bother to provide the info about your installation and to profile your Jenkins installation.
          Any other contributions to Jenkins core or plugins would be appreciated as well.

          Oleg Nenashev added a comment - Nobody excuses for the "issue". Many systems (including Jenkins) are slow by design due to the runtime specifics. In the case of Jenkins there're known bottlenecks: queue handling, security checks, listeners, etc. The performance could be improved step-by-step, but it is not a single issue in the code. If you have such strong complains, just bother to provide the info about your installation and to profile your Jenkins installation. Any other contributions to Jenkins core or plugins would be appreciated as well.

          francisdb added a comment -

          If all builds ever executed need to be read in before jenkins kan show it's main ui this indeed sounds like a design issue. Anyway, I no longer have a big instance running so I can live with the ~1 min startup time we have here.

          francisdb added a comment - If all builds ever executed need to be read in before jenkins kan show it's main ui this indeed sounds like a design issue. Anyway, I no longer have a big instance running so I can live with the ~1 min startup time we have here.

          Oleg Nenashev added a comment -

          @francisdb
          > If all builds ever executed need to be read in before jenkins kan show it's main ui this indeed sounds like a design issue
          This issue has been already solved in 1.485 (JENKINS-8754). However, there were many regression issues caused by the lazy loading, hence it is recommended to use 1.532+
          Now Jenkins loads only the required set if builds. Some plugins may require all builds to be loaded, but it is a topic for additional issues on Jenkins JIRA.

          Oleg Nenashev added a comment - @francisdb > If all builds ever executed need to be read in before jenkins kan show it's main ui this indeed sounds like a design issue This issue has been already solved in 1.485 ( JENKINS-8754 ). However, there were many regression issues caused by the lazy loading, hence it is recommended to use 1.532+ Now Jenkins loads only the required set if builds. Some plugins may require all builds to be loaded, but it is a topic for additional issues on Jenkins JIRA.

          Jesse Glick added a comment -

          This issue is too vague and general to be worth keeping open. If there are specific startup performance problems reproducible in current versions of Jenkins they should be filed with details.

          Jesse Glick added a comment - This issue is too vague and general to be worth keeping open. If there are specific startup performance problems reproducible in current versions of Jenkins they should be filed with details.

          Daniel Beck added a comment -

          Daniel Beck added a comment - Relevant information for filing performance-related issues can be found here: https://wiki.jenkins-ci.org/display/JENKINS/How+to+report+an+issue https://wiki.jenkins-ci.org/display/JENKINS/Obtaining+a+thread+dump

          Paul Draper added a comment - - edited

          @huybrechts, http://n4.nabble.com/Restart-hudson-takes-too-much-time-td386719.html is dead.

          @Jesse this is pretty specific. Create "dozens of projects and thousands of builds" (as per the issue description) and then start Jenkins.
          Large numbers of projects and builds slowing down Jenkins startup has been corroborated by many people on this issue.

          Are you not able to reproduce this issue?
          What details do you lack?

          Paul Draper added a comment - - edited @huybrechts, http://n4.nabble.com/Restart-hudson-takes-too-much-time-td386719.html is dead. @Jesse this is pretty specific. Create "dozens of projects and thousands of builds" (as per the issue description) and then start Jenkins. Large numbers of projects and builds slowing down Jenkins startup has been corroborated by many people on this issue. Are you not able to reproduce this issue? What details do you lack?

          Daniel Beck added a comment -

          has been corroborated by many people on this issue.

          It's less clear if you consider that Jenkins implemented lazy loading of build records from Oct 2012 to some time in early/mid 2013 (until the major issues were worked out).

          Then there's the case that some plugins have a tendency to require loading all build records, which defeats this optimization.

          What details do you lack?

          Stack traces would be helpful. Lists of installed plugins. Maybe even Support Core plugin support bundles. And of course everything with recent Jenkins versions only.

          And even then, at least my expectation is that many different causes contribute to a long startup time, which will need to be fixed one by one rather than gathered in a catch-all issue.

          Daniel Beck added a comment - has been corroborated by many people on this issue. It's less clear if you consider that Jenkins implemented lazy loading of build records from Oct 2012 to some time in early/mid 2013 (until the major issues were worked out). Then there's the case that some plugins have a tendency to require loading all build records, which defeats this optimization. What details do you lack? Stack traces would be helpful. Lists of installed plugins. Maybe even Support Core plugin support bundles. And of course everything with recent Jenkins versions only. And even then, at least my expectation is that many different causes contribute to a long startup time, which will need to be fixed one by one rather than gathered in a catch-all issue.

            Unassigned Unassigned
            blackmuzzle blackmuzzle
            Votes:
            13 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated:
              Resolved: