Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-22822

Queue.maintain does disk I/O via PeepholePermalink.resolve

      https://gist.githubusercontent.com/jglick/de0cf09312d8fce00ab1/raw/gistfile1.txt

      Every time Queue.maintain is called, which is often, for every buildable job in the queue, which could be a lot, Job.getEstimatedDurationCandidates is called. During startup this could do some work loading the last few builds, but we expect those to be held in memory thereafter. However simply finding the last successful/any build involves PeepholePermalink.resolve, which hits the disk every time, with no caching. (It claims to "cache" but this is caching via symlink, not in memory.)

      While we are waiting for the disk I/O operation to complete, the Queue is locked, and lots of other threads will typically be waiting for it.

          [JENKINS-22822] Queue.maintain does disk I/O via PeepholePermalink.resolve

          Andrew Bayer added a comment -

          This is fairly nasty - it caused a lot of UI loading problems on builds.apache.org, and high CPU usage on the master with 1.554.1. I'm now running 1.554.1 with a tiny patch to make Job.getEstimatedDurationCandidates always return an empty list, and woof, the performance improvement was dramatic, to say the least. I think this is particularly painful with MavenModuleSet projects, to no real surprise.

          Andrew Bayer added a comment - This is fairly nasty - it caused a lot of UI loading problems on builds.apache.org, and high CPU usage on the master with 1.554.1. I'm now running 1.554.1 with a tiny patch to make Job.getEstimatedDurationCandidates always return an empty list, and woof, the performance improvement was dramatic, to say the least. I think this is particularly painful with MavenModuleSet projects, to no real surprise.

          Jesse Glick added a comment -

          Mocking out getEstimatedDurationCandidates does not prove that this is the issue—you need to show that the problem is actually symlink resolution—but based on thread dumps I have seen, I am optimistic that avoiding the symlink calls should help.

          @stephenconnolly and I found this issue while reviewing preliminary results of scalability testing on a test Jenkins instance in EC2 with some thousands of jobs constantly running builds.

          Jesse Glick added a comment - Mocking out getEstimatedDurationCandidates does not prove that this is the issue—you need to show that the problem is actually symlink resolution—but based on thread dumps I have seen, I am optimistic that avoiding the symlink calls should help. @stephenconnolly and I found this issue while reviewing preliminary results of scalability testing on a test Jenkins instance in EC2 with some thousands of jobs constantly running builds.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          changelog.html
          core/src/main/java/jenkins/model/PeepholePermalink.java
          core/src/test/java/jenkins/model/PeepholePermalinkTest.java
          http://jenkins-ci.org/commit/jenkins/2ff7a905ac866aac5aea7aae0cb97ee8906b5829
          Log:
          [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: changelog.html core/src/main/java/jenkins/model/PeepholePermalink.java core/src/test/java/jenkins/model/PeepholePermalinkTest.java http://jenkins-ci.org/commit/jenkins/2ff7a905ac866aac5aea7aae0cb97ee8906b5829 Log: [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #3412
          [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O. (Revision 2ff7a905ac866aac5aea7aae0cb97ee8906b5829)

          Result = SUCCESS
          Jesse Glick : 2ff7a905ac866aac5aea7aae0cb97ee8906b5829
          Files :

          • core/src/test/java/jenkins/model/PeepholePermalinkTest.java
          • core/src/main/java/jenkins/model/PeepholePermalink.java
          • changelog.html

          dogfood added a comment - Integrated in jenkins_main_trunk #3412 [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O. (Revision 2ff7a905ac866aac5aea7aae0cb97ee8906b5829) Result = SUCCESS Jesse Glick : 2ff7a905ac866aac5aea7aae0cb97ee8906b5829 Files : core/src/test/java/jenkins/model/PeepholePermalinkTest.java core/src/main/java/jenkins/model/PeepholePermalink.java changelog.html

          Andrew Bayer added a comment -

          I'll try cherrypicking this onto a build of 1.554.1 and seeing how the results do. Thanks!

          Andrew Bayer added a comment - I'll try cherrypicking this onto a build of 1.554.1 and seeing how the results do. Thanks!

          Jesse Glick added a comment -

          Yes, please do.

          Jesse Glick added a comment - Yes, please do.

          Andrew Bayer added a comment -

          Looking very good so far - queue is only up to 16, so it could get worse later, but no page load slowdown at all currently.

          Andrew Bayer added a comment - Looking very good so far - queue is only up to 16, so it could get worse later, but no page load slowdown at all currently.

          Andrew Bayer added a comment -

          So yeah, big performance improvement on the UI - great fix, jglick! Thanks!

          Andrew Bayer added a comment - So yeah, big performance improvement on the UI - great fix, jglick! Thanks!

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/src/main/java/jenkins/model/PeepholePermalink.java
          core/src/test/java/jenkins/model/PeepholePermalinkTest.java
          http://jenkins-ci.org/commit/jenkins/c74dc66f4e9d954b43b64ff5be83b74ebb0f3a54
          Log:
          [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O.

          (cherry picked from commit 2ff7a905ac866aac5aea7aae0cb97ee8906b5829)

          Conflicts:
          changelog.html

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/model/PeepholePermalink.java core/src/test/java/jenkins/model/PeepholePermalinkTest.java http://jenkins-ci.org/commit/jenkins/c74dc66f4e9d954b43b64ff5be83b74ebb0f3a54 Log: [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O. (cherry picked from commit 2ff7a905ac866aac5aea7aae0cb97ee8906b5829) Conflicts: changelog.html

          Code changed in jenkins
          User: Jesse Glick
          Path:
          core/src/main/java/jenkins/model/PeepholePermalink.java
          core/src/test/java/jenkins/model/PeepholePermalinkTest.java
          http://jenkins-ci.org/commit/jenkins/2853564b2387a19893975ed6f042c3c25bf5e062
          Log:
          [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O.

          (cherry picked from commit 2ff7a905ac866aac5aea7aae0cb97ee8906b5829)

          Conflicts:
          changelog.html

          Compare: https://github.com/jenkinsci/jenkins/compare/f3848d2d4865...2853564b2387

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: core/src/main/java/jenkins/model/PeepholePermalink.java core/src/test/java/jenkins/model/PeepholePermalinkTest.java http://jenkins-ci.org/commit/jenkins/2853564b2387a19893975ed6f042c3c25bf5e062 Log: [FIXED JENKINS-22822] PeepholePermalink should maintain an in-memory cache of symlinks to avoid disk I/O. (cherry picked from commit 2ff7a905ac866aac5aea7aae0cb97ee8906b5829) Conflicts: changelog.html Compare: https://github.com/jenkinsci/jenkins/compare/f3848d2d4865...2853564b2387

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: