Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-24915

Extremely slow startup with template-project in combination with jobConfigHistory


      Startup takes 30 min for a Jenkins installation with ~1000 jobs where most of them is using template builders or publishers of template-project plugin when you have jobConfigHistory plugins.

      All jobs are loaded by Jenkins at start up. OK so far. But the template-project loops on all jobs again, and updates (twice!) every job that has template builders or publishers. Combined with a slow and questionable sleep implementation in jobConfigHistory this is disastrous for start up time.

      There may be a lot of other plugins listening to job config save events, so there could be other combinations that is poisonous.

      Note that load, disk, and CPU usage is low, we're mostly just waiting.

      I think the main problem is in template-project, but the jobConfigHistory should probably change algorithm as well. 500 ms is a very long time in computer world. 30 ms is probably more reasonable number.

      "Jenkins initialization thread" prio=10 tid=0x00007f6708883000 nid=0x5fa9 sleeping[0x00007f66fdcc4000]
         java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at hudson.plugins.jobConfigHistory.FileHistoryDao.createNewHistoryDir(FileHistoryDao.java:217)
        at hudson.plugins.jobConfigHistory.FileHistoryDao.getRootDir(FileHistoryDao.java:118)
        at hudson.plugins.jobConfigHistory.FileHistoryDao.createNewHistoryEntry(FileHistoryDao.java:406)
        at hudson.plugins.jobConfigHistory.FileHistoryDao.createNewHistoryEntryAndCopyConfig(FileHistoryDao.java:248)
        at hudson.plugins.jobConfigHistory.FileHistoryDao.saveItem(FileHistoryDao.java:264)
        at hudson.plugins.jobConfigHistory.JobConfigHistorySaveableListener.onChange(JobConfigHistorySaveableListener.java:29)
        at hudson.model.listeners.SaveableListener.fireOnChange(SaveableListener.java:81)
        at hudson.model.AbstractItem.save(AbstractItem.java:462)
        - locked <0x00000007a2cee670> (a hudson.model.FreeStyleProject)
        at hudson.model.Job.save(Job.java:166)
        - locked <0x00000007a2cee670> (a hudson.model.FreeStyleProject)
        at hudson.model.AbstractProject.save(AbstractProject.java:303)
        - locked <0x00000007a2cee670> (a hudson.model.FreeStyleProject)
        at hudson.model.Job.removeProperty(Job.java:498)
        at hudson.model.Job.removeProperty(Job.java:511)
        at hudson.plugins.templateproject.ItemListenerImpl.onLoaded(ItemListenerImpl.java:42)
        at jenkins.model.Jenkins.<init>(Jenkins.java:863)

            mfriedenhagen Mirko Friedenhagen
            marcus_phi Marcus Philip
            7 Vote for this issue
            11 Start watching this issue