-
Bug
-
Resolution: Fixed
-
Critical
https://gist.githubusercontent.com/jglick/de0cf09312d8fce00ab1/raw/gistfile1.txt
Every time Queue.maintain is called, which is often, for every buildable job in the queue, which could be a lot, Job.getEstimatedDurationCandidates is called. During startup this could do some work loading the last few builds, but we expect those to be held in memory thereafter. However simply finding the last successful/any build involves PeepholePermalink.resolve, which hits the disk every time, with no caching. (It claims to "cache" but this is caching via symlink, not in memory.)
While we are waiting for the disk I/O operation to complete, the Queue is locked, and lots of other threads will typically be waiting for it.
This is fairly nasty - it caused a lot of UI loading problems on builds.apache.org, and high CPU usage on the master with 1.554.1. I'm now running 1.554.1 with a tiny patch to make Job.getEstimatedDurationCandidates always return an empty list, and woof, the performance improvement was dramatic, to say the least. I think this is particularly painful with MavenModuleSet projects, to no real surprise.