Similar to the well-known problem in ssh-slaves but affecting the Jenkins SSH server. I noticed that workflow-plugin functional tests were running very slowly with low CPU usage, and saw in thread dumps
"Executing testDeleteSubFolder(org.jenkinsci.plugins.workflow.steps.DeleteDirStepTest)" ...
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:267)
- locked <...> (a jenkins.model.Jenkins$7)
at jenkins.InitReactorRunner.run(InitReactorRunner.java:44)
at jenkins.model.Jenkins.executeReactor(Jenkins.java:915)
at jenkins.model.Jenkins.<init>(Jenkins.java:814)
at hudson.model.Hudson.<init>(Hudson.java:83)
at org.jvnet.hudson.test.JenkinsRule.newHudson(JenkinsRule.java:559)
at org.jvnet.hudson.test.JenkinsRule.before(JenkinsRule.java:346)
at ...
"SSHD.init" ...
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:255)
at sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:410)
at sun.security.provider.NativePRNG$RandomIO.implGenerateSeed(NativePRNG.java:427)
- locked <...> (a java.lang.Object)
at sun.security.provider.NativePRNG$RandomIO.access$500(NativePRNG.java:329)
at sun.security.provider.NativePRNG.engineGenerateSeed(NativePRNG.java:224)
at java.security.SecureRandom.generateSeed(SecureRandom.java:533)
at org.apache.sshd.common.random.BouncyCastleRandom.<init>(BouncyCastleRandom.java:57)
at org.apache.sshd.common.random.BouncyCastleRandom$Factory.create(BouncyCastleRandom.java:48)
at org.apache.sshd.common.random.BouncyCastleRandom$Factory.create(BouncyCastleRandom.java:41)
at org.apache.sshd.common.random.SingletonRandomFactory.<init>(SingletonRandomFactory.java:37)
at org.apache.sshd.SshServer.setUpDefaultServer(SshServer.java:452)
at org.jenkinsci.main.modules.sshd.SSHD.start(SSHD.java:83)
- locked <...> (a org.jenkinsci.main.modules.sshd.SSHD)
at org.jenkinsci.main.modules.sshd.SSHD.init(SSHD.java:146)
at sun.reflect.GeneratedMethodAccessor149.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:106)
at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:176)
at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
at jenkins.model.Jenkins$7.runTask(Jenkins.java:904)
at ...
The bundled server should be switched to not rely on NativePRNG, which can be quite slow.
sudo apt-get install haveged seems to have improved test speed in my case, but I should not have to do this.
Bug
Major