Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-59404

High CPU usage in MapperBase.importFromFile in Benchmark Plugin

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • benchmark-plugin
    • None
    • Jenkins ver. 2.184
      Benchmark 1.0.9

      Several days ago we've noticed that our Jenkins master had almost all its 12 core under 100% load. Apparently it's been that way for several days straight. Using Monitoring plugin we've figured out there was about 10 threads with tremendous amount of time spent in `java.util.TreeMap.put`. Killing those threads one by one didn't break anything, so we assumed it to be some strange jenkins quirk and went on. AFAIK stack traces weren't informative.

      Today I noticed there are 3 threads each eating 100% of core again. This time stack traces are more informative (see attachments). I cannot confirm that those two issues with CPU load are directly correlated, but the current one seems legit enough.

          [JENKINS-59404] High CPU usage in MapperBase.importFromFile in Benchmark Plugin

          Hi Artalus,

          How big is the history of  builds maintained for jobs?

          When accessing the result page, the plugin  loads the  result files for all builds in a multithreaded fashion. At the time, it uses  all available threads minus a few to maintain standard operations. This loading depending on the size of the  history of builds  can reduce CPU availability on the computer until loading is complete. It was implemented to speed load the result page. I could add an option to select the number of threads being used.

          I am currently traveling but I can look into implementing the change next week. Otherwise, if you have the time, you can create your own custom branch and submit your proposed change.

          Let me know,

          Daniel Mercier added a comment - Hi Artalus, How big is the history of  builds maintained for jobs? When accessing the result page, the plugin  loads the  result files for all builds in a multithreaded fashion. At the time, it uses  all available threads minus a few to maintain standard operations. This loading depending on the size of the  history of builds  can reduce CPU availability on the computer until loading is complete. It was implemented to speed load the result page. I could add an option to select the number of threads being used. I am currently traveling but I can look into implementing the change next week. Otherwise, if you have the time, you can create your own custom branch and submit your proposed change. Let me know,

          Artalus S. added a comment -

          Hi Daniel. I am unsure that the problem we are experiencing is related to multithreaded file loading. There are several jobs with up to 1000 builds in history, but the ones that use Benchmark store about 100 builds. It seems to me that there are some thread getting stuck constantly reloading files – either this, or some files causing for (enContent : jsonObject.entrySet()) { ... this.builds.add(build);  in importFromFile() to loop much more times that it should.

          Artalus S. added a comment - Hi Daniel. I am unsure that the problem we are experiencing is related to multithreaded file loading. There are several jobs with up to 1000 builds in history, but the ones that use Benchmark store about 100 builds. It seems to me that there are some thread getting stuck constantly reloading files – either this, or some files causing for (enContent : jsonObject.entrySet()) { ... this.builds.add(build);   in importFromFile() to loop much more times that it should.

          Hi Artalus,

          Back to work. There is a good chance that the threads are kept alive because I forgot to close the InputStreamReader. The behavior would match your observations and it is a safe change to add anyway. All the Unit tests are passing so I am pushing the modification to version '1.0.11' .

          I apologize for the impact it had on your system.

          Let me know if you still observe the CPU overload.

          Thank you,

          Daniel

          Daniel Mercier added a comment - Hi Artalus, Back to work. There is a good chance that the threads are kept alive because I forgot to close the InputStreamReader. The behavior would match your observations and it is a safe change to add anyway. All the Unit tests are passing so I am pushing the modification to version '1.0.11' . I apologize for the impact it had on your system. Let me know if you still observe the CPU overload. Thank you, Daniel

            damercie Daniel Mercier
            artalus Artalus S.
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated: