Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-50642

intermittent JenkinsRule test timeout failure

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      Sometimes, the JenkinsRule based tests in the git-plugin can fail intermittently when the default test is set to 180.

      It appears pretty often if you just run "mvn test" in the git plugin source code, it is fairly common to see a few tests marked as flaky get re-run and pass on the second run.

      If you increase the test timeout to say, 10 minutes, then the tests will pass. A few other tests that aren't marked as flaky also sometimes fail, and it is difficult to debug exactly what's causing them.

      [INFO]
      [INFO] Results:
      [INFO]
      [WARNING] Flakes:
      [WARNING] hudson.plugins.git.GitChangeSetBadArgsTest.testFindOrCreateEmptyCommitter(hudson.plugins.git.GitChangeSetBadArgsTest)
      [ERROR] Run 1: GitChangeSetBadArgsTest>Object.wait:502->Object.wait:-2 » TestTimedOut test ti...
      [INFO] Run 2: PASS
      [INFO]
      [WARNING] hudson.plugins.git.GitPublisherTest.testMergeAndPushFF(hudson.plugins.git.GitPublisherTest)
      [ERROR] Run 1: GitPublisherTest>Object.wait:502->Object.wait:-2 » TestTimedOut test timed out...
      [INFO] Run 2: PASS
      [INFO]
      [WARNING] hudson.plugins.git.GitPublisherTest.testMergeAndPushWithSystemEnvVar(hudson.plugins.git.GitPublisherTest)
      [ERROR] Run 1: GitPublisherTest>Object.wait:502->Object.wait:-2 » TestTimedOut test timed out...
      [INFO] Run 2: PASS
      [INFO]
      [WARNING] hudson.plugins.git.GitSCMTest.testEmailCommitter(hudson.plugins.git.GitSCMTest)
      [ERROR] Run 1: GitSCMTest>Object.wait:502->Object.wait:-2 » TestTimedOut test timed out after...
      [INFO] Run 2: PASS
      [INFO]
      [WARNING] hudson.plugins.git.GitStatusTest.testDoNotifyCommitWithTwoBranches(hudson.plugins.git.GitStatusTest)
      [ERROR] Run 1: GitStatusTest>Object.wait:502->Object.wait:-2 » TestTimedOut test timed out af...
      [INFO] Run 2: PASS
      [INFO]
      [WARNING] jenkins.plugins.git.GitStepTest.multipleSCMs(jenkins.plugins.git.GitStepTest)
      [ERROR] Run 1: GitStepTest.multipleSCMs
      [INFO] Run 2: PASS
      [INFO]
      [WARNING] jenkins.plugins.git.GitStepTest.roundtrip(jenkins.plugins.git.GitStepTest)
      [ERROR] Run 1: GitStepTest.roundtrip:75 » TestTimedOut test timed out after 180 seconds
      [INFO] Run 2: PASS
      [INFO]
      [INFO]

       

      Interestingly, if you single out a test to run manually, it does not timeout, so it is difficult to debug or determine the problem that causes the test to fail.

       

       

      I suspect that there is some sort of contention issue when running the tests, or something within the test body takes too long sometimes. But I haven't been able to root cause it.

      I'll try to update this with more information when I can catch specific tests failing.

        Attachments

          Activity

          jekeller Jacob Keller created issue -
          Hide
          jekeller Jacob Keller added a comment -
          Error
          test timed out after 180 seconds
          Stacktrace
          org.junit.runners.model.TestTimedOutException: test timed out after 180 seconds
          at java.lang.Object.wait(Native Method)
          at java.lang.Object.wait(Object.java:502)
          at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:267)
          at jenkins.InitReactorRunner.run(InitReactorRunner.java:44)
          at jenkins.model.Jenkins.executeReactor(Jenkins.java:916)
          at jenkins.model.Jenkins.<init>(Jenkins.java:815)
          at hudson.model.Hudson.<init>(Hudson.java:85)
          at org.jvnet.hudson.test.JenkinsRule.newHudson(JenkinsRule.java:611)
          at org.jvnet.hudson.test.JenkinsRule.before(JenkinsRule.java:384)
          at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:537)
          at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
          at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.lang.Thread.run(Thread.java:748)
          Standard Output
          === Starting jenkins.plugins.git.GitSCMFileSystemTest
          Standard Error
          0.009 [id=2163]	INFO	o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http://localhost:40627/jenkins/
          0.066 [id=2169]	INFO	jenkins.InitReactorRunner$1#onAttained: Started initialization
          0.077 [id=2182]	INFO	jenkins.InitReactorRunner$1#onAttained: Listed all plugins
          0.085 [id=2184]	INFO	jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
          0.085 [id=2182]	INFO	jenkins.InitReactorRunner$1#onAttained: Started all plugins
          0.088 [id=2174]	INFO	jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
          0.238 [id=2174]	INFO	jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
          0.239 [id=2173]	WARNING	h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping
          180.001 [id=1]	WARNING	o.j.hudson.test.JenkinsRule$2#evaluate: Test timed out (after 180 seconds).
          "Jenkins cron thread" Id=2167 Group=FailOnTimeoutGroup WAITING on java.util.TaskQueue@795f8317
          at java.lang.Object.wait(Native Method)
          - waiting on java.util.TaskQueue@795f8317
          at java.lang.Object.wait(Object.java:502)
          at java.util.TimerThread.mainLoop(Timer.java:526)
          at java.util.TimerThread.run(Timer.java:505)
          "jenkins.util.Timer [#10]" Id=2196 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "jenkins.util.Timer [#1]" Id=2168 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "jenkins.util.Timer [#2]" Id=2188 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "jenkins.util.Timer [#3]" Id=2189 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "jenkins.util.Timer [#4]" Id=2190 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "jenkins.util.Timer [#5]" Id=2191 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "jenkins.util.Timer [#6]" Id=2192 Group=FailOnTimeoutGroup TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "jenkins.util.Timer [#7]" Id=2193 Group=FailOnTimeoutGroup TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "jenkins.util.Timer [#8]" Id=2194 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "jenkins.util.Timer [#9]" Id=2195 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "Jetty Thread Pool" Id=2165 Group=FailOnTimeoutGroup RUNNABLE (in native)
          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
          - locked sun.nio.ch.Util$3@355c94be
          - locked java.util.Collections$UnmodifiableSet@97a145b
          - locked sun.nio.ch.EPollSelectorImpl@c386958
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
          at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:234)
          at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:187)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:211)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126)
          at org.eclipse.jetty.io.ManagedSelector$$Lambda$9/1551303995.run(Unknown Source)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@16047f9e
          "Jetty Thread Pool-acceptor-0@6e744712-ServerConnector@60b3b4ac{HTTP/1.1,[http/1.1]}{localhost:40627}" Id=2166 Group=FailOnTimeoutGroup RUNNABLE (in native)
          at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
          at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
          at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
          - locked java.lang.Object@633fd91
          at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:352)
          at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:603)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@7a63fe5e
          "JGit-WorkQueue" Id=149 Group=FailOnTimeoutGroup TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@44d64d4e
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@44d64d4e
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "Scheduler-843831320" Id=2164 Group=FailOnTimeoutGroup TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@157ec23b
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@157ec23b
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "SSHD.init" Id=2180 Group=FailOnTimeoutGroup RUNNABLE (in native)
          at java.io.FileInputStream.readBytes(Native Method)
          at java.io.FileInputStream.read(FileInputStream.java:255)
          at sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:424)
          at sun.security.provider.NativePRNG$RandomIO.implGenerateSeed(NativePRNG.java:441)
          - locked java.lang.Object@176555c
          at sun.security.provider.NativePRNG$RandomIO.access$500(NativePRNG.java:331)
          at sun.security.provider.NativePRNG.engineGenerateSeed(NativePRNG.java:226)
          at java.security.SecureRandom.generateSeed(SecureRandom.java:533)
          at org.apache.sshd.common.random.BouncyCastleRandom.<init>(BouncyCastleRandom.java:57)
          at org.apache.sshd.common.random.BouncyCastleRandom$Factory.create(BouncyCastleRandom.java:48)
          at org.apache.sshd.common.random.BouncyCastleRandom$Factory.create(BouncyCastleRandom.java:41)
          at org.apache.sshd.common.random.SingletonRandomFactory.<init>(SingletonRandomFactory.java:37)
          at org.apache.sshd.SshServer.setUpDefaultServer(SshServer.java:452)
          at org.jenkinsci.main.modules.sshd.SSHD.start(SSHD.java:83)
          - locked org.jenkinsci.main.modules.sshd.SSHD@7c781c42
          at org.jenkinsci.main.modules.sshd.SSHD.init(SSHD.java:146)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:106)
          at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:176)
          at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
          at jenkins.model.Jenkins$7.runTask(Jenkins.java:905)
          at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
          at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          Number of locked synchronizers = 1
          - java.util.concurrent.ThreadPoolExecutor$Worker@2eb779f
          "main" Id=1 Group=main RUNNABLE
          at sun.management.ThreadImpl.dumpThreads0(Native Method)
          at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
          at hudson.Functions.getThreadInfos(Functions.java:1176)
          at org.jvnet.hudson.test.JenkinsRule.dumpThreads(JenkinsRule.java:442)
          at org.jvnet.hudson.test.JenkinsRule.access$100(JenkinsRule.java:246)
          at org.jvnet.hudson.test.JenkinsRule$2.evaluate(JenkinsRule.java:577)
          at org.junit.rules.RunRules.evaluate(RunRules.java:20)
          at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
          at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
          at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272)
          at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236)
          at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
          at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386)
          at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323)
          at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143)
          "ping-20s" Id=11 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@eb6ec6
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@eb6ec6
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "qtp2090563811-64" Id=64 Group=main RUNNABLE (in native)
          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
          - locked sun.nio.ch.Util$3@2152ab30
          - locked java.util.Collections$UnmodifiableSet@3902bd2c
          - locked sun.nio.ch.EPollSelectorImpl@7d32e714
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
          at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:234)
          at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:187)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:211)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126)
          at org.eclipse.jetty.io.ManagedSelector$$Lambda$9/1551303995.run(Unknown Source)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
          at java.lang.Thread.run(Thread.java:748)
          "qtp2090563811-65" Id=65 Group=main RUNNABLE (in native)
          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
          - locked sun.nio.ch.Util$3@619f2afc
          - locked java.util.Collections$UnmodifiableSet@df921b1
          - locked sun.nio.ch.EPollSelectorImpl@4db60246
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
          at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:234)
          at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:187)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:211)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126)
          at org.eclipse.jetty.io.ManagedSelector$$Lambda$9/1551303995.run(Unknown Source)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
          at java.lang.Thread.run(Thread.java:748)
          "qtp2090563811-66" Id=66 Group=main RUNNABLE (in native)
          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
          - locked sun.nio.ch.Util$3@60a19573
          - locked java.util.Collections$UnmodifiableSet@44a085e5
          - locked sun.nio.ch.EPollSelectorImpl@134ff8f8
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
          at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:234)
          at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:187)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:211)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126)
          at org.eclipse.jetty.io.ManagedSelector$$Lambda$9/1551303995.run(Unknown Source)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
          at java.lang.Thread.run(Thread.java:748)
          "qtp2090563811-67-acceptor-0@3596eb42-ServerConnector@3b152928{HTTP/1.1,[http/1.1]}{0.0.0.0:36383}" Id=67 Group=main RUNNABLE (in native)
          at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
          at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
          at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
          - locked java.lang.Object@66f28a1f
          at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:352)
          at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:603)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
          at java.lang.Thread.run(Thread.java:748)
          "qtp2090563811-68" Id=68 Group=main RUNNABLE (in native)
          at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
          at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
          at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
          at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
          - locked sun.nio.ch.Util$3@3166f664
          - locked java.util.Collections$UnmodifiableSet@47ac613b
          - locked sun.nio.ch.EPollSelectorImpl@60d6fdd4
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
          at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
          at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:234)
          at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:187)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:211)
          at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126)
          at org.eclipse.jetty.io.ManagedSelector$$Lambda$9/1551303995.run(Unknown Source)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
          at java.lang.Thread.run(Thread.java:748)
          "qtp2090563811-69" Id=69 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@526a9908
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@526a9908
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
          at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:564)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:627)
          at java.lang.Thread.run(Thread.java:748)
          "qtp2090563811-70" Id=70 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@526a9908
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@526a9908
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
          at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:564)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:627)
          at java.lang.Thread.run(Thread.java:748)
          "qtp2090563811-71" Id=71 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@526a9908
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@526a9908
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
          at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:564)
          at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
          at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:627)
          at java.lang.Thread.run(Thread.java:748)
          "Scheduler-928734079" Id=72 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1dd74143
          at sun.misc.Unsafe.park(Native Method)
          - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1dd74143
          at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
          at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
          at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:748)
          "surefire-forkedjvm-command-thread" Id=10 Group=main RUNNABLE (in native)
          at java.io.FileInputStream.readBytes(Native Method)
          at java.io.FileInputStream.read(FileInputStream.java:255)
          at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
          at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
          - locked java.io.BufferedInputStream@34d45ec0
          at java.io.DataInputStream.readInt(DataInputStream.java:387)
          at org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:115)
          at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:386)
          at java.lang.Thread.run(Thread.java:748)
          "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@18137eab
          at java.lang.Object.wait(Native Method)
          - waiting on java.lang.ref.ReferenceQueue$Lock@18137eab
          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
          at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
          at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
          "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@727320fa
          at java.lang.Object.wait(Native Method)
          - waiting on java.lang.ref.Reference$Lock@727320fa
          at java.lang.Object.wait(Object.java:502)
          at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
          at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
          "Signal Dispatcher" Id=4 Group=system RUNNABLE
          

           Here's one of the stack traces for the test. I can't tell what each thread is doing and which one might actually be blocked.

          Show
          jekeller Jacob Keller added a comment - Error test timed out after 180 seconds Stacktrace org.junit.runners.model.TestTimedOutException: test timed out after 180 seconds at java.lang. Object .wait(Native Method) at java.lang. Object .wait( Object .java:502) at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:267) at jenkins.InitReactorRunner.run(InitReactorRunner.java:44) at jenkins.model.Jenkins.executeReactor(Jenkins.java:916) at jenkins.model.Jenkins.<init>(Jenkins.java:815) at hudson.model.Hudson.<init>(Hudson.java:85) at org.jvnet.hudson.test.JenkinsRule.newHudson(JenkinsRule.java:611) at org.jvnet.hudson.test.JenkinsRule.before(JenkinsRule.java:384) at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:537) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang. Thread .run( Thread .java:748) Standard Output === Starting jenkins.plugins.git.GitSCMFileSystemTest Standard Error 0.009 [id=2163] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http: //localhost:40627/jenkins/ 0.066 [id=2169] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 0.077 [id=2182] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 0.085 [id=2184] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 0.085 [id=2182] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins 0.088 [id=2174] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 0.238 [id=2174] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs 0.239 [id=2173] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping 180.001 [id=1] WARNING o.j.hudson.test.JenkinsRule$2#evaluate: Test timed out (after 180 seconds). "Jenkins cron thread" Id=2167 Group=FailOnTimeoutGroup WAITING on java.util.TaskQueue@795f8317 at java.lang. Object .wait(Native Method) - waiting on java.util.TaskQueue@795f8317 at java.lang. Object .wait( Object .java:502) at java.util.TimerThread.mainLoop(Timer.java:526) at java.util.TimerThread.run(Timer.java:505) "jenkins.util.Timer [#10]" Id=2196 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "jenkins.util.Timer [#1]" Id=2168 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "jenkins.util.Timer [#2]" Id=2188 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "jenkins.util.Timer [#3]" Id=2189 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "jenkins.util.Timer [#4]" Id=2190 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "jenkins.util.Timer [#5]" Id=2191 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "jenkins.util.Timer [#6]" Id=2192 Group=FailOnTimeoutGroup TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "jenkins.util.Timer [#7]" Id=2193 Group=FailOnTimeoutGroup TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "jenkins.util.Timer [#8]" Id=2194 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "jenkins.util.Timer [#9]" Id=2195 Group=FailOnTimeoutGroup WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@77c10a5f at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "Jetty Thread Pool" Id=2165 Group=FailOnTimeoutGroup RUNNABLE (in native ) at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked sun.nio.ch.Util$3@355c94be - locked java.util.Collections$UnmodifiableSet@97a145b - locked sun.nio.ch.EPollSelectorImpl@c386958 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:234) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:187) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:211) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126) at org.eclipse.jetty.io.ManagedSelector$$Lambda$9/1551303995.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@16047f9e "Jetty Thread Pool-acceptor-0@6e744712-ServerConnector@60b3b4ac{HTTP/1.1,[http/1.1]}{localhost:40627}" Id=2166 Group=FailOnTimeoutGroup RUNNABLE (in native ) at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) - locked java.lang. Object @633fd91 at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:352) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:603) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@7a63fe5e "JGit-WorkQueue" Id=149 Group=FailOnTimeoutGroup TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@44d64d4e at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@44d64d4e at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "Scheduler-843831320" Id=2164 Group=FailOnTimeoutGroup TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@157ec23b at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@157ec23b at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "SSHD.init" Id=2180 Group=FailOnTimeoutGroup RUNNABLE (in native ) at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:255) at sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:424) at sun.security.provider.NativePRNG$RandomIO.implGenerateSeed(NativePRNG.java:441) - locked java.lang. Object @176555c at sun.security.provider.NativePRNG$RandomIO.access$500(NativePRNG.java:331) at sun.security.provider.NativePRNG.engineGenerateSeed(NativePRNG.java:226) at java.security.SecureRandom.generateSeed(SecureRandom.java:533) at org.apache.sshd.common.random.BouncyCastleRandom.<init>(BouncyCastleRandom.java:57) at org.apache.sshd.common.random.BouncyCastleRandom$Factory.create(BouncyCastleRandom.java:48) at org.apache.sshd.common.random.BouncyCastleRandom$Factory.create(BouncyCastleRandom.java:41) at org.apache.sshd.common.random.SingletonRandomFactory.<init>(SingletonRandomFactory.java:37) at org.apache.sshd.SshServer.setUpDefaultServer(SshServer.java:452) at org.jenkinsci.main.modules.sshd.SSHD.start(SSHD.java:83) - locked org.jenkinsci.main.modules.sshd.SSHD@7c781c42 at org.jenkinsci.main.modules.sshd.SSHD.init(SSHD.java:146) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:106) at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:176) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282) at jenkins.model.Jenkins$7.runTask(Jenkins.java:905) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) Number of locked synchronizers = 1 - java.util.concurrent.ThreadPoolExecutor$Worker@2eb779f "main" Id=1 Group=main RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454) at hudson.Functions.getThreadInfos(Functions.java:1176) at org.jvnet.hudson.test.JenkinsRule.dumpThreads(JenkinsRule.java:442) at org.jvnet.hudson.test.JenkinsRule.access$100(JenkinsRule.java:246) at org.jvnet.hudson.test.JenkinsRule$2.evaluate(JenkinsRule.java:577) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:272) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:386) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:323) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:143) "ping-20s" Id=11 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@eb6ec6 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@eb6ec6 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "qtp2090563811-64" Id=64 Group=main RUNNABLE (in native ) at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked sun.nio.ch.Util$3@2152ab30 - locked java.util.Collections$UnmodifiableSet@3902bd2c - locked sun.nio.ch.EPollSelectorImpl@7d32e714 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:234) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:187) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:211) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126) at org.eclipse.jetty.io.ManagedSelector$$Lambda$9/1551303995.run(Unknown Source) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) at java.lang. Thread .run( Thread .java:748) "qtp2090563811-65" Id=65 Group=main RUNNABLE (in native ) at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked sun.nio.ch.Util$3@619f2afc - locked java.util.Collections$UnmodifiableSet@df921b1 - locked sun.nio.ch.EPollSelectorImpl@4db60246 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:234) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:187) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:211) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126) at org.eclipse.jetty.io.ManagedSelector$$Lambda$9/1551303995.run(Unknown Source) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) at java.lang. Thread .run( Thread .java:748) "qtp2090563811-66" Id=66 Group=main RUNNABLE (in native ) at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked sun.nio.ch.Util$3@60a19573 - locked java.util.Collections$UnmodifiableSet@44a085e5 - locked sun.nio.ch.EPollSelectorImpl@134ff8f8 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:234) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:187) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:211) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126) at org.eclipse.jetty.io.ManagedSelector$$Lambda$9/1551303995.run(Unknown Source) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) at java.lang. Thread .run( Thread .java:748) "qtp2090563811-67-acceptor-0@3596eb42-ServerConnector@3b152928{HTTP/1.1,[http/1.1]}{0.0.0.0:36383}" Id=67 Group=main RUNNABLE (in native ) at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) - locked java.lang. Object @66f28a1f at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:352) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:603) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) at java.lang. Thread .run( Thread .java:748) "qtp2090563811-68" Id=68 Group=main RUNNABLE (in native ) at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked sun.nio.ch.Util$3@3166f664 - locked java.util.Collections$UnmodifiableSet@47ac613b - locked sun.nio.ch.EPollSelectorImpl@60d6fdd4 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:234) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:187) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:211) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:126) at org.eclipse.jetty.io.ManagedSelector$$Lambda$9/1551303995.run(Unknown Source) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) at java.lang. Thread .run( Thread .java:748) "qtp2090563811-69" Id=69 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@526a9908 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@526a9908 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:564) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:627) at java.lang. Thread .run( Thread .java:748) "qtp2090563811-70" Id=70 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@526a9908 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@526a9908 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:564) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:627) at java.lang. Thread .run( Thread .java:748) "qtp2090563811-71" Id=71 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@526a9908 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@526a9908 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:564) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:627) at java.lang. Thread .run( Thread .java:748) "Scheduler-928734079" Id=72 Group=main TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1dd74143 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1dd74143 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang. Thread .run( Thread .java:748) "surefire-forkedjvm-command-thread" Id=10 Group=main RUNNABLE (in native ) at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:255) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) - locked java.io.BufferedInputStream@34d45ec0 at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.maven.surefire.booter.MasterProcessCommand.decode(MasterProcessCommand.java:115) at org.apache.maven.surefire.booter.CommandReader$CommandRunnable.run(CommandReader.java:386) at java.lang. Thread .run( Thread .java:748) "Finalizer" Id=3 Group=system WAITING on java.lang.ref.ReferenceQueue$Lock@18137eab at java.lang. Object .wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@18137eab at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) "Reference Handler" Id=2 Group=system WAITING on java.lang.ref.Reference$Lock@727320fa at java.lang. Object .wait(Native Method) - waiting on java.lang.ref.Reference$Lock@727320fa at java.lang. Object .wait( Object .java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) "Signal Dispatcher" Id=4 Group=system RUNNABLE  Here's one of the stack traces for the test. I can't tell what each thread is doing and which one might actually be blocked.
          Hide
          jekeller Jacob Keller added a comment -

          Paradoxically, if I increase the timeout run, sometimes the steps take less actual time to run:

           

          [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 481.838 s - in jenkins.plugins.git.GitStepTest with -Djenkins.test.timeout=360

           

          [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 654.984 s <<< FAILURE! - in jenkins.plugins.git.GitStepTest with -Djenkins.test.timeout=180 (the default)

           

          I think this is because something (I am not sure what) is causing the test to be re-run (note the Tests Run = 7 vs 8 on the failing case).

           

          Also note if i run the test solo using -Dtest=GitStepTest it is much faster:

          [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 92.27 s - in jenkins.plugins.git.GitStepTest

           

          I suspect that something is going wrong when run in the normal way, resulting in the increased test time....

          Show
          jekeller Jacob Keller added a comment - Paradoxically, if I increase the timeout run, sometimes the steps take less actual time to run:   [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 481.838 s - in jenkins.plugins.git.GitStepTest with -Djenkins.test.timeout=360   [ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 654.984 s <<< FAILURE! - in jenkins.plugins.git.GitStepTest with -Djenkins.test.timeout=180 (the default)   I think this is because something (I am not sure what) is causing the test to be re-run (note the Tests Run = 7 vs 8 on the failing case).   Also note if i run the test solo using -Dtest=GitStepTest it is much faster: [INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 92.27 s - in jenkins.plugins.git.GitStepTest   I suspect that something is going wrong when run in the normal way, resulting in the increased test time....
          Hide
          jekeller Jacob Keller added a comment -

          More information: I dug into the surefire test reports of one of the slow tests:

          === Starting jenkins.plugins.git.GitSCMFileSystemTest

          0.018 [id=184] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http://localhost:33389/jenkins/
          0.078 [id=190] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
          0.082 [id=203] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
          0.089 [id=195] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
          0.093 [id=191] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
          0.093 [id=192] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
          0.310 [id=197] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
          0.316 [id=203] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping
          168.000 [id=212] INFO o.j.main.modules.sshd.SSHD#start: Started SSHD at port 38287
          168.001 [id=212] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization

           

          Notice how it takes almost 167 seconds just to get to where the SSHD is started? The actual tests themselves are actually incredibly short:

           

          <testcase name="mixedContent" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.092"/>
          <testcase name="given_filesystem_when_askingChangesSinceSameRevision_then_changesAreEmpty" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.004"/>
          <testcase name="lastModified_Smokes" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.124"/>
          <testcase name="ofSource_Smokes" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.074"/>
          <testcase name="ofSourceRevision" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.116"/>
          <testcase name="directoryTraversal" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.092"/>
          <testcase name="given_filesystem_when_askingChangesSinceNewRevision_then_changesArePopulatedButEmpty" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.012"/>
          <testcase name="slashyBranches" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.085"/>
          <testcase name="given_filesystem_when_askingChangesSinceOldRevision_then_changesArePopulated" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.018"/>

           

          So we're basically spending most of the test run time (168.877 s - in jenkins.plugins.git.GitSCMFileSystemTest) just in that pre-run code.....

          Show
          jekeller Jacob Keller added a comment - More information: I dug into the surefire test reports of one of the slow tests: === Starting jenkins.plugins.git.GitSCMFileSystemTest 0.018 [id=184] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http://localhost:33389/jenkins/ 0.078 [id=190] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 0.082 [id=203] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 0.089 [id=195] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 0.093 [id=191] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins 0.093 [id=192] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 0.310 [id=197] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs 0.316 [id=203] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping 168.000 [id=212] INFO o.j.main.modules.sshd.SSHD#start: Started SSHD at port 38287 168.001 [id=212] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization   Notice how it takes almost 167 seconds just to get to where the SSHD is started? The actual tests themselves are actually incredibly short:   <testcase name="mixedContent" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.092"/> <testcase name="given_filesystem_when_askingChangesSinceSameRevision_then_changesAreEmpty" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.004"/> <testcase name="lastModified_Smokes" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.124"/> <testcase name="ofSource_Smokes" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.074"/> <testcase name="ofSourceRevision" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.116"/> <testcase name="directoryTraversal" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.092"/> <testcase name="given_filesystem_when_askingChangesSinceNewRevision_then_changesArePopulatedButEmpty" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.012"/> <testcase name="slashyBranches" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.085"/> <testcase name="given_filesystem_when_askingChangesSinceOldRevision_then_changesArePopulated" classname="jenkins.plugins.git.GitSCMFileSystemTest" time="0.018"/>   So we're basically spending most of the test run time (168.877 s - in jenkins.plugins.git.GitSCMFileSystemTest) just in that pre-run code.....
          Hide
          jekeller Jacob Keller added a comment -

          Something causes the tests to take significantly longer when run together, vs when run on their own...

          Show
          jekeller Jacob Keller added a comment - Something causes the tests to take significantly longer when run together, vs when run on their own...
          Hide
          jekeller Jacob Keller added a comment -

          I've configured the logger to also output FINE log messages, and am running the tests now, hopefully that will shed light into what exactly is taking so long during Jenkins initialization.

          Show
          jekeller Jacob Keller added a comment - I've configured the logger to also output FINE log messages, and am running the tests now, hopefully that will shed light into what exactly is taking so long during Jenkins initialization.
          Hide
          jekeller Jacob Keller added a comment - - edited

          When I run an individual test it takes almost no time between finalizing setup and starting the SSHD server:

           

          11.812 [id=32] FINE jenkins.InitReactorRunner$1#onTaskCompleted: Completed Finalizing set up
          11.812 [id=40] FINE jenkins.InitReactorRunner$1#onAttained: Attained Finalizing set up
          11.971 [id=24] INFO o.j.main.modules.sshd.SSHD#start: Started SSHD at port 35077
          11.971 [id=24] FINE jenkins.InitReactorRunner$1#onTaskCompleted: Completed SSHD.init
          11.975 [id=34] FINE jenkins.InitReactorRunner$1#onTaskCompleted: Completed GroovyInitScript.init

           

          But when I run the tests as a group...

           

           0.757 [id=463] FINE jenkins.InitReactorRunner$1#onTaskCompleted: Completed Finalizing set up
          0.757 [id=463] FINE jenkins.InitReactorRunner$1#onAttained: Attained Finalizing set up
          5.065 [id=468] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c
          10.065 [id=470] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c
          15.066 [id=473] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c
          20.066 [id=475] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c
          25.066 [id=440] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c
          30.067 [id=469] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c
          35.067 [id=472] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c
          40.068 [id=473] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c
          45.068 [id=476] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c
          50.069 [id=468] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c
          55.069 [id=470] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c
          58.574 [id=464] INFO o.j.main.modules.sshd.SSHD#start: Started SSHD at port 46515
          58.574 [id=464] FINE jenkins.InitReactorRunner$1#onTaskCompleted: Completed SSHD.init

           

           

          I snipped the queue maintenance messages, but there's nothing else between these messages when logged at FINE

          Show
          jekeller Jacob Keller added a comment - - edited When I run an individual test it takes almost no time between finalizing setup and starting the SSHD server:   11.812 [id=32] FINE jenkins.InitReactorRunner$1#onTaskCompleted: Completed Finalizing set up 11.812 [id=40] FINE jenkins.InitReactorRunner$1#onAttained: Attained Finalizing set up 11.971 [id=24] INFO o.j.main.modules.sshd.SSHD#start: Started SSHD at port 35077 11.971 [id=24] FINE jenkins.InitReactorRunner$1#onTaskCompleted: Completed SSHD.init 11.975 [id=34] FINE jenkins.InitReactorRunner$1#onTaskCompleted: Completed GroovyInitScript.init   But when I run the tests as a group...    0.757 [id=463] FINE jenkins.InitReactorRunner$1#onTaskCompleted: Completed Finalizing set up 0.757 [id=463] FINE jenkins.InitReactorRunner$1#onAttained: Attained Finalizing set up 5.065 [id=468] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c 10.065 [id=470] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c 15.066 [id=473] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c 20.066 [id=475] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c 25.066 [id=440] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c 30.067 [id=469] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c 35.067 [id=472] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c 40.068 [id=473] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c 45.068 [id=476] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c 50.069 [id=468] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c 55.069 [id=470] FINE hudson.model.Queue#maintain: Queue maintenance started hudson.model.Queue@1e884c4c 58.574 [id=464] INFO o.j.main.modules.sshd.SSHD#start: Started SSHD at port 46515 58.574 [id=464] FINE jenkins.InitReactorRunner$1#onTaskCompleted: Completed SSHD.init     I snipped the queue maintenance messages, but there's nothing else between these messages when logged at FINE
          Hide
          jekeller Jacob Keller added a comment - - edited

          I am pretty sure this can be resolved by disabling the SSHD server at startup. I am not certain how to do this yet.

          Show
          jekeller Jacob Keller added a comment - - edited I am pretty sure this can be resolved by disabling the SSHD server at startup. I am not certain how to do this yet.
          Hide
          jekeller Jacob Keller added a comment -

          It looks like jenkins-test-harness currently depends on 1.651.1, which does not disable SSHD by default. I'm unsure of how to add configuration at setup to avoid starting SSHD. but I am suspicious that this will resolve the test issues here.

          Show
          jekeller Jacob Keller added a comment - It looks like jenkins-test-harness currently depends on 1.651.1, which does not disable SSHD by default. I'm unsure of how to add configuration at setup to avoid starting SSHD. but I am suspicious that this will resolve the test issues here.
          Hide
          jekeller Jacob Keller added a comment -

          I don't know if this is the best solution, but I think we could extend the JenkinsRule to allow a preset data which disables the SSHD server. I can't find a better way that actually disables the server at startup. I dislike that it is an opt-in rather than an "opt in to re-enable the SSHD server in the tests that need it", since it would further require changes to the git tests..

           

          Show
          jekeller Jacob Keller added a comment - I don't know if this is the best solution, but I think we could extend the JenkinsRule to allow a preset data which disables the SSHD server. I can't find a better way that actually disables the server at startup. I dislike that it is an opt-in rather than an "opt in to re-enable the SSHD server in the tests that need it", since it would further require changes to the git tests..  
          Hide
          jekeller Jacob Keller added a comment -

          A PresetData won't work with using @ClassRule to setup the JenkinsRule (I can't figure a way to get the PresetData to run prior to the JenkinsRule setup...)

          Show
          jekeller Jacob Keller added a comment - A PresetData won't work with using @ClassRule to setup the JenkinsRule (I can't figure a way to get the PresetData to run prior to the JenkinsRule setup...)
          Hide
          jekeller Jacob Keller added a comment -

          I did a test with a modified jenkins-test-harness that always disabled the SSHD server, and it improved this dramatically, with tests taking <5minutes now. I still don't have a good method to actually implement this yet, but I definitely think we should improve this if we can.

          Show
          jekeller Jacob Keller added a comment - I did a test with a modified jenkins-test-harness that always disabled the SSHD server, and it improved this dramatically, with tests taking <5minutes now. I still don't have a good method to actually implement this yet, but I definitely think we should improve this if we can.
          Hide
          jekeller Jacob Keller added a comment -

          Maybe Jesse Glick, or Stephen Connolly  have some input?

          Show
          jekeller Jacob Keller added a comment - Maybe Jesse Glick , or Stephen Connolly   have some input?
          Hide
          jekeller Jacob Keller added a comment -

          For record, here is the initial launch time for when the SSHD server is disabled. It's comparable for when the time is fast, but this is consistent whether I run only one test via -Dtest, vs if I run
          the entire suite.

          === Starting testBuildChooserContext(hudson.plugins.git.GitSCMTest)
             0.051 [id=22]        INFO    o.jvnet.hudson.test.WarExploder#explode: Picking up existing exploded jenkins.war at /home/jekeller/git/jenkins/git-plugin/target/jenkins-for-test
             0.276 [id=22]        INFO    o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http://localhost:41917/jenkins/
             0.854 [id=29]        INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization
             9.177 [id=49]        INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
             9.219 [id=40]        INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
             9.238 [id=45]        INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins
             9.244 [id=48]        INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
            10.397 [id=30]        INFO    jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
            10.493 [id=29]        WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping
            11.212 [id=37]        INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
          

          Additionally, it appears some work is avoided for multiple tests in the same class since the following test is much faster:

          === Starting testPolling_environmentValueInBranchSpec(hudson.plugins.git.GitSCMTest)
             0.005 [id=7592]      INFO    o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http://localhost:32975/jenkins/
             0.058 [id=7598]      INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization
             0.059 [id=7610]      INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins
             0.062 [id=7603]      INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
             0.065 [id=7614]      INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins
             0.065 [id=7621]      INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
             0.210 [id=7614]      WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping
             0.334 [id=7612]      INFO    jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
             0.337 [id=7605]      INFO    jenkins.InitReactorRunner$1#onAttained: Completed initialization
          

          I am still very confused why running multiple tests (not even in parallel, or with multiple threads) ends up resulting in the slowdown, but disabling the SSHD server seems to fix it.

          Show
          jekeller Jacob Keller added a comment - For record, here is the initial launch time for when the SSHD server is disabled. It's comparable for when the time is fast, but this is consistent whether I run only one test via -Dtest, vs if I run the entire suite. === Starting testBuildChooserContext(hudson.plugins.git.GitSCMTest) 0.051 [id=22] INFO o.jvnet.hudson.test.WarExploder#explode: Picking up existing exploded jenkins.war at /home/jekeller/git/jenkins/git-plugin/target/jenkins- for -test 0.276 [id=22] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http: //localhost:41917/jenkins/ 0.854 [id=29] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 9.177 [id=49] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 9.219 [id=40] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 9.238 [id=45] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins 9.244 [id=48] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 10.397 [id=30] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs 10.493 [id=29] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping 11.212 [id=37] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization Additionally, it appears some work is avoided for multiple tests in the same class since the following test is much faster: === Starting testPolling_environmentValueInBranchSpec(hudson.plugins.git.GitSCMTest) 0.005 [id=7592] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http: //localhost:32975/jenkins/ 0.058 [id=7598] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 0.059 [id=7610] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 0.062 [id=7603] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 0.065 [id=7614] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins 0.065 [id=7621] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 0.210 [id=7614] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping 0.334 [id=7612] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs 0.337 [id=7605] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization I am still very confused why running multiple tests (not even in parallel, or with multiple threads) ends up resulting in the slowdown, but disabling the SSHD server seems to fix it.
          markewaite Mark Waite made changes -
          Field Original Value New Value
          Assignee Mark Waite [ markewaite ] Jacob Keller [ jekeller ]
          Hide
          markewaite Mark Waite added a comment -

          I'm thrilled that you've found what appears to be a persistent source of long running tests in test environments. Thanks for the investigation!

          Are your changes to jenkins-test-harness something you'd be willing to share with me so that I could evaluate them in my test environments? I'd love to remove the source of the periodic timeouts that I receive when running tests on varied environments.

          Show
          markewaite Mark Waite added a comment - I'm thrilled that you've found what appears to be a persistent source of long running tests in test environments. Thanks for the investigation! Are your changes to jenkins-test-harness something you'd be willing to share with me so that I could evaluate them in my test environments? I'd love to remove the source of the periodic timeouts that I receive when running tests on varied environments.
          Hide
          jglick Jesse Glick added a comment -

          Do not bother. Mark Waite already has a branch (IIRC) for this plugin which uses a more contemporary Jenkins baseline. There is no point in wasting development effort supporting PRs built against obsolete lines.

          If you must support old lines and the Linux CI system has insufficient entropy, the only workaround I know of is installing the haveged package.

          Show
          jglick Jesse Glick added a comment - Do not bother. Mark Waite already has a branch (IIRC) for this plugin which uses a more contemporary Jenkins baseline. There is no point in wasting development effort supporting PRs built against obsolete lines. If you must support old lines and the Linux CI system has insufficient entropy, the only workaround I know of is installing the haveged package.
          Hide
          markewaite Mark Waite added a comment - - edited

          Jesse Glick I think I've seen the same behavior from the require-jdk-8 pull request.

          Jacob Keller if you run your tests using code from that pull request, do you see the same problems?

          Show
          markewaite Mark Waite added a comment - - edited Jesse Glick I think I've seen the same behavior from the require-jdk-8 pull request . Jacob Keller if you run your tests using code from that pull request, do you see the same problems?
          Hide
          jekeller Jacob Keller added a comment -

          > I'm thrilled that you've found what appears to be a persistent source of long running tests in test environments. Thanks for the investigation!

          Mostly persisted due to the timeout failures.

          > Are your changes to jenkins-test-harness something you'd be willing to share with me so that I could evaluate them in my test environments? I'd love to remove the source of the periodic timeouts that I receive when running tests on varied environments.

          If you really wanna see what I did: https://github.com/jacob-keller/jenkins-test-harness/tree/jk-disable-sshd

          > Jacob Keller if you run your tests using code from that pull request, do you see the same problems?

          I'll give it a shot.

          Show
          jekeller Jacob Keller added a comment - > I'm thrilled that you've found what appears to be a persistent source of long running tests in test environments. Thanks for the investigation! Mostly persisted due to the timeout failures. > Are your changes to jenkins-test-harness something you'd be willing to share with me so that I could evaluate them in my test environments? I'd love to remove the source of the periodic timeouts that I receive when running tests on varied environments. If you really wanna see what I did: https://github.com/jacob-keller/jenkins-test-harness/tree/jk-disable-sshd > Jacob Keller if you run your tests using code from that pull request, do you see the same problems? I'll give it a shot.
          Hide
          jekeller Jacob Keller added a comment -

          Additionally, I added some logging to the SSHD plugin, and I was able to extract this information:

          === Starting changelogAndPolling(jenkins.plugins.git.GitStepTest)

          0.009 [id=380] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http://localhost:44701/jenkins/
          0.059 [id=386] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization
          0.060 [id=398] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
          0.066 [id=391] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins
          0.070 [id=389] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins
          0.071 [id=389] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions
          0.276 [id=409] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping
          0.709 [id=399] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs
          0.711 [id=391] INFO o.j.main.modules.sshd.SSHD#start: SSHD - start()
          0.712 [id=391] INFO o.j.main.modules.sshd.SSHD#start: start() - stopped server
          177.722 [id=391] INFO o.j.main.modules.sshd.SSHD#start: start() - set port
          177.723 [id=391] INFO o.j.main.modules.sshd.SSHD#start: start() - set key pair
          177.723 [id=391] INFO o.j.main.modules.sshd.SSHD#start: start() - set public key
          177.734 [id=391] INFO o.j.main.modules.sshd.SSHD#start: Started SSHD at port 39243
          177.735 [id=406] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization

          It appears that we go into SSHD start() and get stuck early in setup.

          Suddenly, a bunch of things made sense and I think the problem has to do with the platform not generating enough entropy so calls to get random numbers result installs waiting for entropy in /dev/random...

          That definitely explains the problem and why it's platform specific. I suspect my system I'm testing on does not have enough entropy so when running and re-running the SSHD server in rapid succession, it gets consumed, until the SSHD initialization stalls.

          Using something like haveged on the VFs would be a good solution I think. I'm looking into whether I can enable the hw rng on my system to avoid the stalls.

          > If you must support old lines and the Linux CI system has insufficient entropy, the only workaround I know of is installing the haveged package.

          Nice tip to think in terms of SSH key setup and entropy requirements.

          Show
          jekeller Jacob Keller added a comment - Additionally, I added some logging to the SSHD plugin, and I was able to extract this information: === Starting changelogAndPolling(jenkins.plugins.git.GitStepTest) 0.009 [id=380] INFO o.jvnet.hudson.test.JenkinsRule#createWebServer: Running on http://localhost:44701/jenkins/ 0.059 [id=386] INFO jenkins.InitReactorRunner$1#onAttained: Started initialization 0.060 [id=398] INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins 0.066 [id=391] INFO jenkins.InitReactorRunner$1#onAttained: Prepared all plugins 0.070 [id=389] INFO jenkins.InitReactorRunner$1#onAttained: Started all plugins 0.071 [id=389] INFO jenkins.InitReactorRunner$1#onAttained: Augmented all extensions 0.276 [id=409] WARNING h.ExtensionFinder$GuiceFinder$FaultTolerantScope$1#error: Failed to instantiate optional component org.jenkinsci.plugins.workflow.steps.scm.GitStep$DescriptorImpl; skipping 0.709 [id=399] INFO jenkins.InitReactorRunner$1#onAttained: Loaded all jobs 0.711 [id=391] INFO o.j.main.modules.sshd.SSHD#start: SSHD - start() 0.712 [id=391] INFO o.j.main.modules.sshd.SSHD#start: start() - stopped server 177.722 [id=391] INFO o.j.main.modules.sshd.SSHD#start: start() - set port 177.723 [id=391] INFO o.j.main.modules.sshd.SSHD#start: start() - set key pair 177.723 [id=391] INFO o.j.main.modules.sshd.SSHD#start: start() - set public key 177.734 [id=391] INFO o.j.main.modules.sshd.SSHD#start: Started SSHD at port 39243 177.735 [id=406] INFO jenkins.InitReactorRunner$1#onAttained: Completed initialization It appears that we go into SSHD start() and get stuck early in setup. Suddenly, a bunch of things made sense and I think the problem has to do with the platform not generating enough entropy so calls to get random numbers result installs waiting for entropy in /dev/random... That definitely explains the problem and why it's platform specific. I suspect my system I'm testing on does not have enough entropy so when running and re-running the SSHD server in rapid succession, it gets consumed, until the SSHD initialization stalls. Using something like haveged on the VFs would be a good solution I think. I'm looking into whether I can enable the hw rng on my system to avoid the stalls. > If you must support old lines and the Linux CI system has insufficient entropy, the only workaround I know of is installing the haveged package. Nice tip to think in terms of SSH key setup and entropy requirements.
          Hide
          jekeller Jacob Keller added a comment -

          I also just discovered that my linux system had been running without RNGD for a few months, which may explain the failures (no RNGD means that I don't have good entropy source on the system).

          Show
          jekeller Jacob Keller added a comment - I also just discovered that my linux system had been running without RNGD for a few months, which may explain the failures (no RNGD means that I don't have good entropy source on the system).
          Hide
          jekeller Jacob Keller added a comment - - edited

          As a comment, Jenkins did not start disabling the SSHD server by default until 2.83 it looks like (based on the disablement landing in SSHD 2.1, and that version not being pulled in until 2.83..

          So the jenkins-jdk-8 branch likely will fail for the same reason (low entropy when spawning the SSHD server).

          EDIT: I ran the tests, and checked the output,it appears that SSHD is not run when we build with 2.60.3 as the base.

          Note that you may be able to get the remote VFs for the Jenkins CI server to enable something like virtio-rng if you have sufficient RNG from the hosts, but haveged may be sufficient as well.

          Show
          jekeller Jacob Keller added a comment - - edited As a comment, Jenkins did not start disabling the SSHD server by default until 2.83 it looks like (based on the disablement landing in SSHD 2.1, and that version not being pulled in until 2.83.. So the jenkins-jdk-8 branch likely will fail for the same reason (low entropy when spawning the SSHD server). EDIT: I ran the tests, and checked the output,it appears that SSHD is not run when we build with 2.60.3 as the base. Note that you may be able to get the remote VFs for the Jenkins CI server to enable something like virtio-rng if you have sufficient RNG from the hosts, but haveged may be sufficient as well.
          Hide
          jglick Jesse Glick added a comment -

          the jenkins-jdk-8 branch likely will fail for the same reason

          I guess you mean MarkEWaite:require-jdk-8. That is on 2.60.x—already quite old! May as well just base it on the current LTS (2.107.x) IMO.

          Show
          jglick Jesse Glick added a comment - the jenkins-jdk-8 branch likely will fail for the same reason I guess you mean MarkEWaite:require-jdk-8 . That is on 2.60.x—already quite old! May as well just base it on the current LTS (2.107.x) IMO.
          Hide
          markewaite Mark Waite added a comment -

          I've intentionally based it on 2.60.3 to bring it to Java 8 and still give as many users as possible the chance to update to the new plugin version when it releases, without requiring that they are on the latest Jenkins LTS.

          Once the JENKINS-48061 fix is complete, verified, and released, I'll merge the require-jdk-8 branch to master and deliver a beta release of git plugin 4.0.0. At that time, I'll re-evaluate the adoption numbers of various Jenkins releases so that I can follow the same pattern that Stephen Connolly is using with the credentials plugin.

          Show
          markewaite Mark Waite added a comment - I've intentionally based it on 2.60.3 to bring it to Java 8 and still give as many users as possible the chance to update to the new plugin version when it releases, without requiring that they are on the latest Jenkins LTS. Once the JENKINS-48061 fix is complete, verified, and released, I'll merge the require-jdk-8 branch to master and deliver a beta release of git plugin 4.0.0. At that time, I'll re-evaluate the adoption numbers of various Jenkins releases so that I can follow the same pattern that Stephen Connolly is using with the credentials plugin.
          Hide
          jekeller Jacob Keller added a comment -

          The test failures are occurring on the VMs see: https://ci.jenkins.io/job/Plugins/job/git-plugin/job/PR-579/5/display/redirect

          How difficult would it be to get haveged installed on the linux VM? That's what I got running on my local system to resolve the issue (as the particular system does not have a keyboard/mouse locally, nor does it have a hardware RNG source for rng-tools to use).

          I also verified that my original assumption that it wasn't fixed by 2.60.3 was wrong, so I think just upgrading to 2.60.3 and stopping tests on the old version would fix this as well.

          Show
          jekeller Jacob Keller added a comment - The test failures are occurring on the VMs see: https://ci.jenkins.io/job/Plugins/job/git-plugin/job/PR-579/5/display/redirect How difficult would it be to get haveged installed on the linux VM? That's what I got running on my local system to resolve the issue (as the particular system does not have a keyboard/mouse locally, nor does it have a hardware RNG source for rng-tools to use). I also verified that my original assumption that it wasn't fixed by 2.60.3 was wrong, so I think just upgrading to 2.60.3 and stopping tests on the old version would fix this as well.
          Hide
          markewaite Mark Waite added a comment -

          Jacob Keller it appears that the JDK 8 changes which have been included in the git plugin master branch 14 May 2018 have resolved this issue, at least in the cases that I've checked. Can this bug be closed?

          Show
          markewaite Mark Waite added a comment - Jacob Keller it appears that the JDK 8 changes which have been included in the git plugin master branch 14 May 2018 have resolved this issue, at least in the cases that I've checked. Can this bug be closed?
          Hide
          jekeller Jacob Keller added a comment -

          Yea, upgrading to depending on the 2.60.3 Jenkins server avoids the issue due to no longer starting up SSH. Will close.

          Show
          jekeller Jacob Keller added a comment - Yea, upgrading to depending on the 2.60.3 Jenkins server avoids the issue due to no longer starting up SSH. Will close.
          Hide
          jekeller Jacob Keller added a comment -

          The root cause of this failure is due to launching the SSH server in the Jenkins test harness, which generates an SSH key. This key generation requires entropy, and Linux blocks when there isn't enough entropy available.

          In Virtual Machines, especially when running high test volumes, it is very easy to consume all of the available entropy, causing the tests to block when attempting to start the JenkinsRule. If entropy took too long to generate, it resulted in random test failures.

          This was fixed/worked around by upgrading to a newer version of Jenkins which has the SSH server disabled by default, thus avoiding the root cause of key generation.

          Show
          jekeller Jacob Keller added a comment - The root cause of this failure is due to launching the SSH server in the Jenkins test harness, which generates an SSH key. This key generation requires entropy, and Linux blocks when there isn't enough entropy available. In Virtual Machines, especially when running high test volumes, it is very easy to consume all of the available entropy, causing the tests to block when attempting to start the JenkinsRule. If entropy took too long to generate, it resulted in random test failures. This was fixed/worked around by upgrading to a newer version of Jenkins which has the SSH server disabled by default, thus avoiding the root cause of key generation.
          jekeller Jacob Keller made changes -
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Closed [ 6 ]

            People

            Assignee:
            jekeller Jacob Keller
            Reporter:
            jekeller Jacob Keller
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: