Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-20108

SSH slaves can block for a long time in NativePRNG

      Have encountered some reports of slow slave performance on a Unix master using many slaves where the thread dumps show all but one slave connection thread waiting for a single lock, which is held by a thread that looks like this:

      "Pipe writer thread: ..." - Thread ...
         java.lang.Thread.State: RUNNABLE
          at sun.security.provider.NativePRNG$RandomIO.implNextBytes(NativePRNG.java:255)
          - locked <598aec0c> (a java.lang.Object)
          at sun.security.provider.NativePRNG$RandomIO.access$200(NativePRNG.java:108)
          at sun.security.provider.NativePRNG.engineNextBytes(NativePRNG.java:97)
          at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
          - locked <329129da> (a java.security.SecureRandom)
          at java.security.SecureRandom.next(SecureRandom.java:455)
          at java.util.Random.nextInt(Random.java:189)
          at com.trilead.ssh2.transport.TransportConnection.sendMessage(TransportConnection.java:154)
      

      From what I can tell neither the Jenkins SSH Slaves plugin nor the Trilead SSH library are to blame, as they produce a different SecureRandom instance for each slave. Rather it is NativePRNG (the default implementation on typical Linux installations among others) which uses a global lock, to synchronize access to /dev/random and /dev/urandom; and random can block waiting for sufficient entropy to accumulate.

      It might help for the SSH Slaves plugin to offer a java.security.SecureRandom based on sun.security.provider.SecureRandom, which does not acquire a global lock to process connection data. (It may take longer to set up a connection, since it needs to seed the random-number generator based on thread activity.)

      Unconfirmed workarounds:

      • Edit the JRE's $JAVA_HOME/lib/security/java.security to comment out the line securerandom.source=file:/dev/urandom (should switch back to the generic implementation)
      • Running -Djava.security.egd=file:/dev/./urandom (should force use of urandom which is supposed to be nonblocking)

          [JENKINS-20108] SSH slaves can block for a long time in NativePRNG

          Chris Reynolds added a comment - - edited

          Chris Reynolds added a comment - - edited Created https://issues.jenkins-ci.org/browse/JENKINS-25241

          Jesse Glick added a comment -

          csreynolds it is best to file a bug with the regression keyword blocking this one, so it can be assigned and its fix tracked.

          Jesse Glick added a comment - csreynolds it is best to file a bug with the regression keyword blocking this one, so it can be assigned and its fix tracked.

          Code changed in jenkins
          User: Kohsuke Kawaguchi
          Path:
          core/pom.xml
          http://jenkins-ci.org/commit/jenkins/68eb26633b1c042d90a426efac7b6e60dd4b2ede
          Log:
          [FIXED JENKINS-20108]

          Integrated the trilead with SecureRandom instantiation fix.

          (cherry picked from commit 70f5968160b0a1cf81dc451fd2aacf1e0f9201ea)

          Conflicts:
          changelog.html
          core/pom.xml

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Kohsuke Kawaguchi Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/68eb26633b1c042d90a426efac7b6e60dd4b2ede Log: [FIXED JENKINS-20108] Integrated the trilead with SecureRandom instantiation fix. (cherry picked from commit 70f5968160b0a1cf81dc451fd2aacf1e0f9201ea) Conflicts: changelog.html core/pom.xml

          Code changed in jenkins
          User: Oliver Gondža
          Path:
          core/pom.xml
          http://jenkins-ci.org/commit/jenkins/8fbd44db2b2ebf81947a6c3c5861bc30b48cc775
          Log:
          Revert "[FIXED JENKINS-20108]"

          This reverts commit 68eb26633b1c042d90a426efac7b6e60dd4b2ede.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oliver Gondža Path: core/pom.xml http://jenkins-ci.org/commit/jenkins/8fbd44db2b2ebf81947a6c3c5861bc30b48cc775 Log: Revert " [FIXED JENKINS-20108] " This reverts commit 68eb26633b1c042d90a426efac7b6e60dd4b2ede.

          Daniel Beck added a comment -

          This caused JENKINS-25241 so unfit for LTS.

          Daniel Beck added a comment - This caused JENKINS-25241 so unfit for LTS.

          Jesse Glick added a comment -

          JENKINS-25241 has been fixed in 2.4.5 so I think this can be reconsidered for 1.580.3.

          Jesse Glick added a comment - JENKINS-25241 has been fixed in 2.4.5 so I think this can be reconsidered for 1.580.3.

          Daniel Beck added a comment -

          Jesse: IMO conditional on that issue not happening on the still bundled Subversion plugin 1.x. Forcing users to upgrade to 2.x wouldn't be nice in a minor LTS update.

          Daniel Beck added a comment - Jesse: IMO conditional on that issue not happening on the still bundled Subversion plugin 1.x. Forcing users to upgrade to 2.x wouldn't be nice in a minor LTS update.

          Jesse Glick added a comment -

          True, I forgot that we are still bundling an obsolete version of the plugin.

          Jesse Glick added a comment - True, I forgot that we are still bundling an obsolete version of the plugin.

          Rejected for 1.580.3. Removing lts-candidate label entirely, as this will be part of next LTS line anyway.

          Oliver Gondža added a comment - Rejected for 1.580.3. Removing lts-candidate label entirely, as this will be part of next LTS line anyway.

          dogfood added a comment -

          Integrated in jenkins_main_trunk #4292
          [FIXED JENKINS-20108] (Revision 68eb26633b1c042d90a426efac7b6e60dd4b2ede)
          Revert "[FIXED JENKINS-20108]" (Revision 8fbd44db2b2ebf81947a6c3c5861bc30b48cc775)

          Result = UNSTABLE
          ogondza : 68eb26633b1c042d90a426efac7b6e60dd4b2ede
          Files :

          • core/pom.xml

          ogondza : 8fbd44db2b2ebf81947a6c3c5861bc30b48cc775
          Files :

          • core/pom.xml

          dogfood added a comment - Integrated in jenkins_main_trunk #4292 [FIXED JENKINS-20108] (Revision 68eb26633b1c042d90a426efac7b6e60dd4b2ede) Revert " [FIXED JENKINS-20108] " (Revision 8fbd44db2b2ebf81947a6c3c5861bc30b48cc775) Result = UNSTABLE ogondza : 68eb26633b1c042d90a426efac7b6e60dd4b2ede Files : core/pom.xml ogondza : 8fbd44db2b2ebf81947a6c3c5861bc30b48cc775 Files : core/pom.xml

            kohsuke Kohsuke Kawaguchi
            jglick Jesse Glick
            Votes:
            1 Vote for this issue
            Watchers:
            12 Start watching this issue

              Created:
              Updated:
              Resolved: