It seems that TimeoutException thrown in Jetty 8, which was recoverable (the channel would stay open and then continue to process data), is fatal in Jetty 9. This will prevent CLI commands for running for more than 30s.

          [JENKINS-46659] TimeoutException from Jetty 9 not recoverable

          Jesse Glick added a comment -

          CC olamy. Discovered by kohsuke.

          Jesse Glick added a comment - CC olamy . Discovered by kohsuke .

          Jesse Glick added a comment -

          Does not seem to be reproducible using java -jar jenkins.war; not sure what might be different in the bundled Jetty. Needs investigation.

          Jesse Glick added a comment - Does not seem to be reproducible using java -jar jenkins.war ; not sure what might be different in the bundled Jetty. Needs investigation.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          Jenkinsfile
          test/src/test/java/hudson/cli/CLIActionTest.java
          http://jenkins-ci.org/commit/jenkins/810a21534d5183055f303fd6f2bf69eb85e19367
          Log:
          JENKINS-46659 Suppressing failing CLIActionTest.interleavedStdio.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: Jenkinsfile test/src/test/java/hudson/cli/CLIActionTest.java http://jenkins-ci.org/commit/jenkins/810a21534d5183055f303fd6f2bf69eb85e19367 Log: JENKINS-46659 Suppressing failing CLIActionTest.interleavedStdio.

          Olivier Lamy added a comment -

          jglick any stack trace?

          Olivier Lamy added a comment - jglick any stack trace?

          Jesse Glick added a comment -

          olamy

          Sep 08, 2017 5:30:04 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run
          FINE: ignoring idle timeout, perhaps from Jetty
          java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
          	at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1047)
          	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307)
          	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:250)
          	at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:58)
          	at hudson.cli.FlightRecorderInputStream.read(FlightRecorderInputStream.java:88)
          	at java.io.DataInputStream.readInt(DataInputStream.java:387)
          	at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:111)
          Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
          	at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
          	at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
          	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
          	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
          	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
          	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          	at java.lang.Thread.run(Thread.java:748)
          

          The stack trace on its own is not particularly interesting because we got the same in Jetty 8. The difference is that 8 allowed Jenkins to catch the error but then go right back to reading from the channel, whereas 9 seems to repeatedly throw the same error—it is not recoverable.

          Probably PlainCLIProtocol needs to be extended to include a ping packet type.

          Jesse Glick added a comment - olamy Sep 08, 2017 5:30:04 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run FINE: ignoring idle timeout, perhaps from Jetty java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1047) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:250) at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:58) at hudson.cli.FlightRecorderInputStream.read(FlightRecorderInputStream.java:88) at java.io.DataInputStream.readInt(DataInputStream.java:387) at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:111) Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) The stack trace on its own is not particularly interesting because we got the same in Jetty 8. The difference is that 8 allowed Jenkins to catch the error but then go right back to reading from the channel, whereas 9 seems to repeatedly throw the same error—it is not recoverable. Probably PlainCLIProtocol needs to be extended to include a ping packet type.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          cli/src/main/java/hudson/cli/CLI.java
          cli/src/main/java/hudson/cli/PlainCLIProtocol.java
          test/src/test/java/hudson/cli/CLIActionTest.java
          http://jenkins-ci.org/commit/jenkins/dc06c3f03b026c8faafa3e080f3a307c97f4fd42
          Log:
          [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cli/src/main/java/hudson/cli/CLI.java cli/src/main/java/hudson/cli/PlainCLIProtocol.java test/src/test/java/hudson/cli/CLIActionTest.java http://jenkins-ci.org/commit/jenkins/dc06c3f03b026c8faafa3e080f3a307c97f4fd42 Log: [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.

          Daniel Beck added a comment -

          Merged towards 2.80.

          Daniel Beck added a comment - Merged towards 2.80.

          Code changed in jenkins
          User: Jesse Glick
          Path:
          cli/src/main/java/hudson/cli/CLI.java
          cli/src/main/java/hudson/cli/PlainCLIProtocol.java
          http://jenkins-ci.org/commit/jenkins/dda5903c99a46df0d70b7d143d4fd14389bd433b
          Log:
          [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.

          (cherry picked from commit dc06c3f03b026c8faafa3e080f3a307c97f4fd42)

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cli/src/main/java/hudson/cli/CLI.java cli/src/main/java/hudson/cli/PlainCLIProtocol.java http://jenkins-ci.org/commit/jenkins/dda5903c99a46df0d70b7d143d4fd14389bd433b Log: [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running. (cherry picked from commit dc06c3f03b026c8faafa3e080f3a307c97f4fd42)

            jglick Jesse Glick
            jglick Jesse Glick
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: