Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-46659

TimeoutException from Jetty 9 not recoverable

    XMLWordPrintable

Details

    Description

      It seems that TimeoutException thrown in Jetty 8, which was recoverable (the channel would stay open and then continue to process data), is fatal in Jetty 9. This will prevent CLI commands for running for more than 30s.

      Attachments

        Issue Links

          Activity

            jglick Jesse Glick added a comment -

            CC olamy. Discovered by kohsuke.

            jglick Jesse Glick added a comment - CC olamy . Discovered by kohsuke .
            jglick Jesse Glick added a comment -

            Does not seem to be reproducible using java -jar jenkins.war; not sure what might be different in the bundled Jetty. Needs investigation.

            jglick Jesse Glick added a comment - Does not seem to be reproducible using java -jar jenkins.war ; not sure what might be different in the bundled Jetty. Needs investigation.

            Code changed in jenkins
            User: Jesse Glick
            Path:
            Jenkinsfile
            test/src/test/java/hudson/cli/CLIActionTest.java
            http://jenkins-ci.org/commit/jenkins/810a21534d5183055f303fd6f2bf69eb85e19367
            Log:
            JENKINS-46659 Suppressing failing CLIActionTest.interleavedStdio.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: Jenkinsfile test/src/test/java/hudson/cli/CLIActionTest.java http://jenkins-ci.org/commit/jenkins/810a21534d5183055f303fd6f2bf69eb85e19367 Log: JENKINS-46659 Suppressing failing CLIActionTest.interleavedStdio.
            olamy Olivier Lamy added a comment -

            jglick any stack trace?

            olamy Olivier Lamy added a comment - jglick any stack trace?
            jglick Jesse Glick added a comment -

            olamy

            Sep 08, 2017 5:30:04 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run
            FINE: ignoring idle timeout, perhaps from Jetty
            java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
            	at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1047)
            	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307)
            	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:250)
            	at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:58)
            	at hudson.cli.FlightRecorderInputStream.read(FlightRecorderInputStream.java:88)
            	at java.io.DataInputStream.readInt(DataInputStream.java:387)
            	at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:111)
            Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms
            	at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
            	at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
            	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            	at java.lang.Thread.run(Thread.java:748)
            

            The stack trace on its own is not particularly interesting because we got the same in Jetty 8. The difference is that 8 allowed Jenkins to catch the error but then go right back to reading from the channel, whereas 9 seems to repeatedly throw the same error—it is not recoverable.

            Probably PlainCLIProtocol needs to be extended to include a ping packet type.

            jglick Jesse Glick added a comment - olamy Sep 08, 2017 5:30:04 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run FINE: ignoring idle timeout, perhaps from Jetty java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms at org.eclipse.jetty.server.HttpInput$ErrorState.noContent(HttpInput.java:1047) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:250) at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:58) at hudson.cli.FlightRecorderInputStream.read(FlightRecorderInputStream.java:88) at java.io.DataInputStream.readInt(DataInputStream.java:387) at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:111) Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166) at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) The stack trace on its own is not particularly interesting because we got the same in Jetty 8. The difference is that 8 allowed Jenkins to catch the error but then go right back to reading from the channel, whereas 9 seems to repeatedly throw the same error—it is not recoverable. Probably PlainCLIProtocol needs to be extended to include a ping packet type.

            Code changed in jenkins
            User: Jesse Glick
            Path:
            cli/src/main/java/hudson/cli/CLI.java
            cli/src/main/java/hudson/cli/PlainCLIProtocol.java
            test/src/test/java/hudson/cli/CLIActionTest.java
            http://jenkins-ci.org/commit/jenkins/dc06c3f03b026c8faafa3e080f3a307c97f4fd42
            Log:
            [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cli/src/main/java/hudson/cli/CLI.java cli/src/main/java/hudson/cli/PlainCLIProtocol.java test/src/test/java/hudson/cli/CLIActionTest.java http://jenkins-ci.org/commit/jenkins/dc06c3f03b026c8faafa3e080f3a307c97f4fd42 Log: [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.
            danielbeck Daniel Beck added a comment -

            Merged towards 2.80.

            danielbeck Daniel Beck added a comment - Merged towards 2.80.

            Code changed in jenkins
            User: Jesse Glick
            Path:
            cli/src/main/java/hudson/cli/CLI.java
            cli/src/main/java/hudson/cli/PlainCLIProtocol.java
            http://jenkins-ci.org/commit/jenkins/dda5903c99a46df0d70b7d143d4fd14389bd433b
            Log:
            [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running.

            (cherry picked from commit dc06c3f03b026c8faafa3e080f3a307c97f4fd42)

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Jesse Glick Path: cli/src/main/java/hudson/cli/CLI.java cli/src/main/java/hudson/cli/PlainCLIProtocol.java http://jenkins-ci.org/commit/jenkins/dda5903c99a46df0d70b7d143d4fd14389bd433b Log: [FIXED JENKINS-46659] Avoid Jetty timeouts by sending a ping-like packet every 10s while a CLI command is running. (cherry picked from commit dc06c3f03b026c8faafa3e080f3a307c97f4fd42)

            People

              jglick Jesse Glick
              jglick Jesse Glick
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: