-
Bug
-
Resolution: Duplicate
-
Critical
-
None
We have to restart out hanging jenkins instance because of this issue. I'm not changing it to Critical priority.
Jenkins version : 1.539
Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /monitoring. Reason: Error reading from remote server
WARNING: /monitoring
org.apache.commons.jelly.JellyTagException: jar:file:/tmp/jetty-127.0.0.1-8080-jenkins.war--any-/webapp/WEB-INF/lib/jenkins-core-1.539.jar!/lib/layout/layout.jelly:75:72: <st:include> org.eclipse.jetty.io.EofException
at org.apache.commons.jelly.impl.TagScript.handleException(TagScript.java:726)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:279)
at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:269)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:120)
at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:81)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:63)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:53)
at org.kohsuke.stapler.jelly.JellyFacet$1.dispatch(JellyFacet.java:95)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631)
at hudson.init.impl.InstallUncaughtExceptionHandler$1.reportException(InstallUncaughtExceptionHandler.java:25)
at org.kohsuke.stapler.compression.CompressionFilter.reportException(CompressionFilter.java:63)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1474)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:370)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:949)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1011)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
Caused by: org.eclipse.jetty.io.EofException
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:914)
at org.eclipse.jetty.http.AbstractGenerator.flush(AbstractGenerator.java:443)
at org.eclipse.jetty.server.HttpOutput.flush(HttpOutput.java:100)
at org.eclipse.jetty.server.AbstractHttpConnection$Output.flush(AbstractHttpConnection.java:1101)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:161)
at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:107)
at com.jcraft.jzlib.DeflaterOutputStream.deflate(DeflaterOutputStream.java:144)
at com.jcraft.jzlib.DeflaterOutputStream.write(DeflaterOutputStream.java:102)
at org.kohsuke.stapler.compression.FilterServletOutputStream.write(FilterServletOutputStream.java:31)
at org.kohsuke.stapler.jelly.DefaultScriptInvoker$1.write(DefaultScriptInvoker.java:99)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:220)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:290)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:294)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:140)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229)
at java.io.BufferedWriter.flush(BufferedWriter.java:253)
at org.dom4j.io.XMLWriter.flush(XMLWriter.java:276)
at org.apache.commons.jelly.XMLOutput.flush(XMLOutput.java:127)
at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:271)
... 46 more
Caused by: java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:146)
at sun.nio.ch.SocketChannelImpl.write0(SocketChannelImpl.java:373)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:406)
at java.nio.channels.SocketChannel.write(SocketChannel.java:384)
at org.eclipse.jetty.io.nio.ChannelEndPoint.gatheringFlush(ChannelEndPoint.java:376)
at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:335)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:376)
at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:838)
- duplicates
-
-
- Resolved
-
- is duplicated by
-
-
- Closed
-
- is related to
-
JENKINS-20532 HTTP session count
-
- Resolved
-
[JENKINS-20640] Jenkins become very slow and had 'proxy error'
In the monitoring plugin, how many sessions are counted?
And can you attach the image given by the following?
http://localhost:8080/monitoring?part=graph&graph=httpSessions
(where localhost and 8080 are the host and port of your server)
Sam Xiao
added a comment - evernat
See attached screenshot, you can clearly see the spike on the http session right after our upgrade to 1.539
Sam Xiao
added a comment - evernat
See attached screenshot, you can clearly see the spike on the http session right after our upgrade to 1.539 The related issue about HTTP sessions, JENKINS-20352, is now fixed in the monitoring plugin v1.48.
The monitoring plugin v1.48 should be available in the update center of your Jenkins, or from http://mirrors.jenkins-ci.org/plugins/monitoring/, in a few hours from now (24h max?).
I suggest to try that version of the plugin.
But I can't say if that version will fix your issue, or if you have another bigger issue, related to Jetty 8 or anything else.
Sam Xiao
added a comment - Thanks, but I thought the HTTP session problem is different than the proxy error. Or are they actually the same or related?
Sam Xiao
added a comment - evernat
Thanks, but I thought the HTTP session problem is different than the proxy error. Or are they actually the same or related? 
Chris Z
added a comment - Hi, I've got similar issue. In monitoring you can see peeks of active threads. When I get screenshoot (when jenkins unfreez) i will attach it.
Chris Z
added a comment - Hi, I've got similar issue. In monitoring you can see peeks of active threads. When I get screenshoot (when jenkins unfreez) i will attach it. This issue is perhaps related to the HTTP session issue.
To be sure that you don't have the HTTP session issue, I suggest to upgrade to the monitoring plugin v1.48 if you use Jenkins v1.535 or later.
Sam Xiao
added a comment - I don't think these are monitor plugin specific bugs, I've seen the proxy errors almost every endpoints. I think there's an upstream bug somewhere caused this, and only when Jenkins is under heavy load.
Sam Xiao
added a comment - I don't think these are monitor plugin specific bugs, I've seen the proxy errors almost every endpoints. I think there's an upstream bug somewhere caused this, and only when Jenkins is under heavy load. @evernat Do you have a link to the bug about HTTP sessions? I'm seeing problems (with all endpoints) with both our 1.532.1 Jenkins servers.
@Christian Höltje
The bug with HTTP sessions is JENKINS-20532
It happened since Jenkins v1.535 with the change to Jetty.
The bug is fixed since the Monitoring plugin v1.48
Copied from a dup:
JENKINS-20639