Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-19874

Loading job config history with over 20,000 entries takes over 5 minutes

      When trying to access the Job Configuration History of a Job with over 20000 changes the page takes a lot of time to load (over 5 minutes) and the page is unresponsive for that time.
      I tried it on version 2.4 and also on the latest code 2.5 - compiled on 30.9.13.
      See attached screen shots from the Monitoring plugin.

          [JENKINS-19874] Loading job config history with over 20,000 entries takes over 5 minutes

          Hello Sagi,

          this is unfortunately not unexpected. Given a HD seek time of 5 ms, it will take the plugin 20000*5ms (100s) just to access all history files, which afterwards have to be read and deserialized to get the name of the user and the operation. In the latest master we try to build an abstraction layer so in the future we may be able to use other means as e.g. git to handle situations like this better. Right now I can only give the advice to use the purging functionality, where older entries will be deleted.

          Regards
          Mirko

          Mirko Friedenhagen added a comment - Hello Sagi, this is unfortunately not unexpected. Given a HD seek time of 5 ms, it will take the plugin 20000*5ms (100s) just to access all history files, which afterwards have to be read and deserialized to get the name of the user and the operation. In the latest master we try to build an abstraction layer so in the future we may be able to use other means as e.g. git to handle situations like this better. Right now I can only give the advice to use the purging functionality, where older entries will be deleted. Regards Mirko

          Hi Mirko,

          Thanks for the swift response...
          I'll have a look into purging old history.
          If I may, I have a small suggestion - how about using Paging in the display?
          Instead of loading ALL the history at once, you only read the amount of files and separate the results into different pages.
          This way you can show, for instance, 100 results per page and you don't need to read, load and display all the rest of the information.
          The user will be able to easily access the last changes (which are usually more interesting) and still maintain all the job config history.

          What do you think?

          Thanks,
          Sagi

          Sagi Sinai-Glazer added a comment - Hi Mirko, Thanks for the swift response... I'll have a look into purging old history. If I may, I have a small suggestion - how about using Paging in the display? Instead of loading ALL the history at once, you only read the amount of files and separate the results into different pages. This way you can show, for instance, 100 results per page and you don't need to read, load and display all the rest of the information. The user will be able to easily access the last changes (which are usually more interesting) and still maintain all the job config history. What do you think? Thanks, Sagi

          Hello Sagi,

          paging would definitely help, but right now we try to create the abstraction which will hopefully help in doing paging as well.

          Regards
          Mirko

          Mirko Friedenhagen added a comment - Hello Sagi, paging would definitely help, but right now we try to create the abstraction which will hopefully help in doing paging as well. Regards Mirko

          Hi Mirko,

          Any ETA on the abstraction?

          As for purging old history, I tried updating the global settings for Job Config History and set "Max number of history entries to keep" to 5000, but it still show all the 20K entries (and they are still on the file system).
          Am I missing something?
          Is this feature only for future entries?
          Is there a way to purge existing old entries?

          Thanks,
          Sagi

          Sagi Sinai-Glazer added a comment - Hi Mirko, Any ETA on the abstraction? As for purging old history, I tried updating the global settings for Job Config History and set "Max number of history entries to keep" to 5000, but it still show all the 20K entries (and they are still on the file system). Am I missing something? Is this feature only for future entries? Is there a way to purge existing old entries? Thanks, Sagi

          Hello Sagi,

          • No ETA, I do this mostly in my spare time and it is a lot of work.
          • The purging is only done when saving a new configuration and I am pretty sure it will take a while to purge 15000 entries.
          • When you have shell access you might be able to go to $JENKINS_HOME/config-history/jobs/OFFENDING_JOB and delete the 15000 entries there followed by a restart of Jenkins.

          Mirko Friedenhagen added a comment - Hello Sagi, No ETA, I do this mostly in my spare time and it is a lot of work. The purging is only done when saving a new configuration and I am pretty sure it will take a while to purge 15000 entries. When you have shell access you might be able to go to $JENKINS_HOME/config-history/jobs/OFFENDING_JOB and delete the 15000 entries there followed by a restart of Jenkins.

          Can we close this issue?

          Stefan Brausch added a comment - Can we close this issue?

          Stefan - If it's not going to be fixed - then you can close it. I have no use for an open-unaddressed JIRA ticket.
          Having said that, I would prefer to keep it as a backlog item...
          Purging part of the history does help, but it's still not efficient to go through ALL the entries.
          As far as I can tell - enabling paging or some other form of partial loading is the only way to actually resolve this issue.

          Sagi Sinai-Glazer added a comment - Stefan - If it's not going to be fixed - then you can close it. I have no use for an open-unaddressed JIRA ticket. Having said that, I would prefer to keep it as a backlog item... Purging part of the history does help, but it's still not efficient to go through ALL the entries. As far as I can tell - enabling paging or some other form of partial loading is the only way to actually resolve this issue.

          Opened a new feature request for paging.

          Stefan Brausch added a comment - Opened a new feature request for paging.

          Close this issue.

          Stefan Brausch added a comment - Close this issue.

            mfriedenhagen Mirko Friedenhagen
            esinsag Sagi Sinai-Glazer
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: