Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22822

Queue.maintain does disk I/O via PeepholePermalink.resolve

      https://gist.githubusercontent.com/jglick/de0cf09312d8fce00ab1/raw/gistfile1.txt

      Every time Queue.maintain is called, which is often, for every buildable job in the queue, which could be a lot, Job.getEstimatedDurationCandidates is called. During startup this could do some work loading the last few builds, but we expect those to be held in memory thereafter. However simply finding the last successful/any build involves PeepholePermalink.resolve, which hits the disk every time, with no caching. (It claims to "cache" but this is caching via symlink, not in memory.)

      While we are waiting for the disk I/O operation to complete, the Queue is locked, and lots of other threads will typically be waiting for it.

          [JENKINS-22822] Queue.maintain does disk I/O via PeepholePermalink.resolve

          Jesse Glick created issue -

          Andrew Bayer added a comment -

          This is fairly nasty - it caused a lot of UI loading problems on builds.apache.org, and high CPU usage on the master with 1.554.1. I'm now running 1.554.1 with a tiny patch to make Job.getEstimatedDurationCandidates always return an empty list, and woof, the performance improvement was dramatic, to say the least. I think this is particularly painful with MavenModuleSet projects, to no real surprise.

          Andrew Bayer added a comment - This is fairly nasty - it caused a lot of UI loading problems on builds.apache.org, and high CPU usage on the master with 1.554.1. I'm now running 1.554.1 with a tiny patch to make Job.getEstimatedDurationCandidates always return an empty list, and woof, the performance improvement was dramatic, to say the least. I think this is particularly painful with MavenModuleSet projects, to no real surprise.
          Andrew Bayer made changes -
          Priority Original: Major [ 3 ] New: Critical [ 2 ]
          Jesse Glick made changes -
          Assignee New: Jesse Glick [ jglick ]
          Jesse Glick made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Jesse Glick made changes -
          Labels Original: lazy-loading performance permalink queue symlink New: lazy-loading lts-candidate performance permalink queue symlink

          Jesse Glick added a comment -

          Mocking out getEstimatedDurationCandidates does not prove that this is the issue—you need to show that the problem is actually symlink resolution—but based on thread dumps I have seen, I am optimistic that avoiding the symlink calls should help.

          @stephenconnolly and I found this issue while reviewing preliminary results of scalability testing on a test Jenkins instance in EC2 with some thousands of jobs constantly running builds.

          Jesse Glick added a comment - Mocking out getEstimatedDurationCandidates does not prove that this is the issue—you need to show that the problem is actually symlink resolution—but based on thread dumps I have seen, I am optimistic that avoiding the symlink calls should help. @stephenconnolly and I found this issue while reviewing preliminary results of scalability testing on a test Jenkins instance in EC2 with some thousands of jobs constantly running builds.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          core/src/main/java/jenkins/model/PeepholePermalink.java
          core/src/test/java/jenkins/model/PeepholePermalinkTest.java
          http://jenkins-ci.org/commit/jenkins/2ff7a905ac866aac5aea7aae0cb97ee8906b5829
          Log:
          [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/jenkins/model/PeepholePermalink.java core/src/test/java/jenkins/model/PeepholePermalinkTest.java http://jenkins-ci.org/commit/jenkins/2ff7a905ac866aac5aea7aae0cb97ee8906b5829 Log: [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O.
          SCM/JIRA link daemon made changes -
          Resolution New: Fixed [ 1 ]
          Status Original: In Progress [ 3 ] New: Resolved [ 5 ]

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3412
          [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O. (Revision 2ff7a905ac866aac5aea7aae0cb97ee8906b5829)

          Result = SUCCESS
          Jesse Glick : 2ff7a905ac866aac5aea7aae0cb97ee8906b5829
          Files :

          • core/src/test/java/jenkins/model/PeepholePermalinkTest.java
          • core/src/main/java/jenkins/model/PeepholePermalink.java
          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #3412 [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O. (Revision 2ff7a905ac866aac5aea7aae0cb97ee8906b5829) Result = SUCCESS Jesse Glick : 2ff7a905ac866aac5aea7aae0cb97ee8906b5829 Files : core/src/test/java/jenkins/model/PeepholePermalinkTest.java core/src/main/java/jenkins/model/PeepholePermalink.java changelog.html

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: