[JENKINS-46659] TimeoutException from Jetty 9 not recoverable - Jenkins Jira

XML

Word

Printable

Details

Type: Bug
Status: Resolved (View Workflow)
Priority: Major
Resolution: Fixed
Component/s: core
Labels:
- 2.73.3-fixed
- cli
- jetty
- regression

Similar Issues:

Show

Description

It seems that TimeoutException thrown in Jetty 8, which was recoverable (the channel would stay open and then continue to process data), is fatal in Jetty 9. This will prevent CLI commands for running for more than 30s.

Attachments

Issue Links

blocks

JENKINS-41745 Remoting-free CLI

Resolved

JENKINS-43713 Upgrade to Jetty 9.4.x

Resolved

links to

CloudBees Internal PRE-17

discussion in PR 2795

PR 3011

Activity

Ascending order - Click to sort in descending order

Jesse Glick added a comment - 2017-09-05 22:54

CC olamy. Discovered by kohsuke.

Jesse Glick added a comment - 2017-09-05 22:54 CC olamy . Discovered by kohsuke .

Jesse Glick added a comment - 2017-09-05 23:04

Does not seem to be reproducible using java -jar jenkins.war; not sure what might be different in the bundled Jetty. Needs investigation.

Jesse Glick added a comment - 2017-09-05 23:04 Does not seem to be reproducible using java -jar jenkins.war ; not sure what might be different in the bundled Jetty. Needs investigation.

SCM/JIRA link daemon added a comment - 2017-09-06 01:42

Code changed in jenkins
User: Jesse Glick
Path:
Jenkinsfile
test/src/test/java/hudson/cli/CLIActionTest.java
http://jenkins-ci.org/commit/jenkins/810a21534d5183055f303fd6f2bf69eb85e19367
Log:
~~JENKINS-46659~~ Suppressing failing CLIActionTest.interleavedStdio.

SCM/JIRA link daemon added a comment - 2017-09-06 01:42 Code changed in jenkins User: Jesse Glick Path: Jenkinsfile test/src/test/java/hudson/cli/CLIActionTest.java http://jenkins-ci.org/commit/jenkins/810a21534d5183055f303fd6f2bf69eb85e19367 Log: JENKINS-46659 Suppressing failing CLIActionTest.interleavedStdio.

Olivier Lamy added a comment - 2017-09-06 05:23

jglick any stack trace?

Olivier Lamy added a comment - 2017-09-06 05:23 jglick any stack trace?

Jesse Glick added a comment - 2017-09-08 21:30

olamy

Sep 08, 2017 5:30:04 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run
FINE: ignoring idle timeout, perhaps from Jetty
java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
	at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1047)
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307)
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:250)
	at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:58)
	at hudson.cli.FlightRecorderInputStream.read(FlightRecorderInputStream.java:88)
	at java.io.DataInputStream.readInt(DataInputStream.java:387)
	at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:111)
Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
	at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
	at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)

The stack trace on its own is not particularly interesting because we got the same in Jetty 8. The difference is that 8 allowed Jenkins to catch the error but then go right back to reading from the channel, whereas 9 seems to repeatedly throw the same error—it is not recoverable.

Probably PlainCLIProtocol needs to be extended to include a ping packet type.

Jesse Glick added a comment - 2017-09-08 21:30 olamy Sep 08, 2017 5:30:04 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run FINE: ignoring idle timeout, perhaps from Jetty java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1047) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:250) at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:58) at hudson.cli.FlightRecorderInputStream.read(FlightRecorderInputStream.java:88) at java.io.DataInputStream.readInt(DataInputStream.java:387) at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:111) Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) The stack trace on its own is not particularly interesting because we got the same in Jetty 8. The difference is that 8 allowed Jenkins to catch the error but then go right back to reading from the channel, whereas 9 seems to repeatedly throw the same error—it is not recoverable. Probably PlainCLIProtocol needs to be extended to include a ping packet type.

SCM/JIRA link daemon added a comment - 2017-09-22 08:54

Code changed in jenkins
User: Jesse Glick
Path:
cli/src/main/java/hudson/cli/CLI.java
cli/src/main/java/hudson/cli/PlainCLIProtocol.java
test/src/test/java/hudson/cli/CLIActionTest.java
http://jenkins-ci.org/commit/jenkins/dc06c3f03b026c8faafa3e080f3a307c97f4fd42
Log:
[FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.

SCM/JIRA link daemon added a comment - 2017-09-22 08:54 Code changed in jenkins User: Jesse Glick Path: cli/src/main/java/hudson/cli/CLI.java cli/src/main/java/hudson/cli/PlainCLIProtocol.java test/src/test/java/hudson/cli/CLIActionTest.java http://jenkins-ci.org/commit/jenkins/dc06c3f03b026c8faafa3e080f3a307c97f4fd42 Log: [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.

Daniel Beck added a comment - 2017-09-24 17:14

Merged towards 2.80.

Daniel Beck added a comment - 2017-09-24 17:14 Merged towards 2.80.

SCM/JIRA link daemon added a comment - 2017-11-08 11:35

Code changed in jenkins
User: Jesse Glick
Path:
cli/src/main/java/hudson/cli/CLI.java
cli/src/main/java/hudson/cli/PlainCLIProtocol.java
http://jenkins-ci.org/commit/jenkins/dda5903c99a46df0d70b7d143d4fd14389bd433b
Log:
[FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.

(cherry picked from commit dc06c3f03b026c8faafa3e080f3a307c97f4fd42)

SCM/JIRA link daemon added a comment - 2017-11-08 11:35 Code changed in jenkins User: Jesse Glick Path: cli/src/main/java/hudson/cli/CLI.java cli/src/main/java/hudson/cli/PlainCLIProtocol.java http://jenkins-ci.org/commit/jenkins/dda5903c99a46df0d70b7d143d4fd14389bd433b Log: [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running. (cherry picked from commit dc06c3f03b026c8faafa3e080f3a307c97f4fd42)

People

Assignee:: Jesse Glick

Reporter:: Jesse Glick

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 2017-09-05 22:53

Updated:: 2017-12-01 00:33

Resolved:: 2017-09-24 17:14