-
Bug
-
Resolution: Won't Fix
-
Major
-
None
-
Jenkins master 2.55 and 2.54 on RHEL7.3 launched using Oracle 1.8 JDK,
RHEL6.8 client using Oracle 1.8 JDK, RHEL7 client using Oracle 1.8 JDK, RHEL7 client using OpenJDK1.8
Running any Jenkins CLI command throws the exception below. TCP port for JNLP was active, user running CLI command had full admin permissions, error happens using '-noCertificateCheck' flag or omitting it, Anonymous user has overall read permission. Able to reproduce this error on Jenkins 2.55 & 2.54 ... backing down to 2.53 works just fine.
[devops@removed ~]$ /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/bin/java -jar /tmp/jenkins-cli.jar -s https://removed.swg.usma.ibm.com/jenkins -noCertificateCheck help
Apr 18, 2017 2:52:34 PM hudson.cli.CLI _main
INFO: Skipping HTTPS certificate checks altogether. Note that this is not secure at all.
Apr 18, 2017 2:52:51 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run
WARNING: unknown operation #10: 0x0a
0x0a
0x0a
0x0a
0x0a
Apr 18, 2017 2:52:51 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run
WARNING: null
hudson.cli.DiagnosedStreamCorruptionException
Read back: 0x0a
0x0a
0x0a
0x0a
0x0a
' ' 0x0a
' <!DOCTYPE html><html><head resURL="/jenkins/static/ae219983" data-rooturl="/jenkins" data-resurl="/jenkins/static/ae219983">' 0x0a
' ' 0x0a
0x0a
' <title>Jenkins [Jenkins]</title><link rel="stylesheet" href="/jenkins/static/ae219983/css/layout-common.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/css/style.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/css/color.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/css/responsive-grid.css" type="text/css" /><link rel="shortcut icon" href="/jenkins/static/ae219983/favicon.ico" type="image/vnd.microsoft.icon" /><link color="black" rel="mask-icon" href="/jenkins/images/mask-icon.svg" /><script>var isRunAsTest=false; var rootURL="/jenkins"; var resURL="/jenkins/static/ae219983";</script><script src="/jenkins/static/ae219983/scripts/prototype.js" type="text/javascript"></script><script src="/jenkins/static/ae219983/scripts/behavior.js" type="text/javascript"></script><script src='/jenkins/adjuncts/ae219983/org/kohsuke/stapler/bind.js' type='text/javascript'></script><script src="/jenkins/static/ae219983/scripts/yui/yahoo/yahoo-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/dom/dom-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/event/event-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/animation/animation-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/dragdrop/dragdrop-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/container/container-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/connection/connection-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/datasource/datasource-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/autocomplete/autocomplete-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/menu/menu-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/element/element-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/button/button-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/storage/storage-min.js"></script><script src="/jenkins/static/ae219983/scripts/hudson-behavior.js" type="text/javascript"></script><script src="/jenkins/static/ae219983/scripts/sortable.js" type="text/javascript"></script><script>crumb.init("", "");</script><link rel="stylesheet" href="/jenkins/static/ae219983/scripts/yui/container/assets/container.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/scripts/yui/assets/skins/sam/skin.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/scripts/yui/container/assets/skins/sam/container.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/scripts/yui/button/assets/skins/sam/button.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/scripts/yui/menu/assets/skins/sam/menu.css" type="text/css" /><link rel="search" href="/jenkins/opensearch.xml" type="application/opensearchdescription+xml" title="Jenkins" /><meta name="ROBOTS" content="INDEX,NOFOLLOW" /><meta name="viewport" content="width=device-width, initial-scale=1" /><script src="/jenkins/adjuncts/ae219983/org/kohsuke/stapler/jquery/jquery.full.js" type="text/javascript"></script><script>var Q=jQuery.noConflict()</script><script>' 0x0a
' if(window.Prototype && JSON) \{' 0x0a
' var _json_stringify = JSON.stringify;' 0x0a
' JSON.stringify = function(value) \{' 0x0a
' var _array_tojson = Array.prototype.toJSON;' 0x0a
' delete Array.prototype.toJSON;' 0x0a
' var r=_json_stringify(value);' 0x0a
' Array.prototype.toJSON = _array_tojson;' 0x0a
' return r;' 0x0a
' };' 0x0a
' } ' 0x0a
' </script><script src="/jenkins/plugin/extended-choice-parameter/js/selectize.min.js" type="text/javascript"></script><script src="/jenkins/plugin/extended-choice-parameter/js/jsoneditor.min.js" type="text/javascript"></script><script src="/jenkins/plugin/extended-choice-parameter/js/jquery.jsonview.min.js" type="text/javascript"></script><link rel="stylesheet" href="/jenkins/plugin/extended-choice-parameter/css/jquery.jsonview.css" /><link rel="stylesheet" id="icon_stylesheet" href="/jenkins/plugin/extended-choice-parameter/css/selectize.css" /><link rel="stylesheet" id="icon_stylesheet" href="/jenkins/plugin/extended-choice-parameter/css/selectize.bootstrap2.css" /><link rel="stylesheet" id="theme_stylesheet" /><link rel="stylesheet" id="icon_stylesheet" /><script src="/jenkins/static/ae219983/scripts/yui/cookie/cookie-min.js"></script><script src="/jenkins/static/ae219983/jsbundles/page-init.js" type="text/javascript"></script></head><body data-model-type="hudson.model.Hudson" id="jenkins" class="yui-skin-sam two-column jenkins-2.55" data-version="2.55"><a href="#skip2content" class="skiplink">Skip to content</a><div id="page-head"><div id="header"><div class="logo"><a id="jenkins-home-link" href="/jenkins/"><img src="/jenkins/static/ae219983/images/headshot.png" alt="title" id="jenkins-head-icon" /><img src="/jenkins/static/ae219983/images/title.png" alt="title" width="139" id="jenkins-name-icon" height="34" /></a></div><div class="login">' 0xc2 0xa0 '<a href="/jenkins/login?from=%2Fjenkins%2Fcli"><b>log in</b></a></div><div class="searchbox hidden-xs"><form method="get" name="search" action="/jenkins/search/" style="position:relative;" class="no-json"><div id="search-box-minWidth"></div><div id="search-box-sizer"></div><div id="searchform"><input name="q" placeholder="search" id="search-box" class="has-default-text" />' 0xc2 0xa0 '<a href="https://jenkins.io/redirect/search-box"><img src="/jenkins/static/ae219983/images/16x16/help.png" style="width: 16px; height: 16px; " class="icon-help icon-sm" /></a><div id="search-box-completion"></div><script>createSearchBox("/jenkins/search/");</script></div></form></div></div><div id="breadcrumbBar"><tr id="top-nav"><td id="left-top-nav" colspan="2"><link rel='stylesheet' href='/jenkins/adjuncts/ae219983/lib/layout/breadcrumbs.css' type='text/css' /><script src='/jenkins/adjuncts/ae219983/lib/layout/breadcrumbs.js' type='text/javascript'></script><div class="top-sticker noedge"><div class="top-sticker-inner"><div id="right-top-nav"></div><ul id="breadcrumbs"><li class="item"><a href="/jenkins/" class="model-link inside">Jenkins</a></li><li href="/jenkins/" class="children"></li></ul><div id="breadcrumb-menu-target"></div></div></div></td></tr></div></div><div id="page-body" class="clear"><div id="side-panel"><div class="task"><a href="https://jenkins.io/" class="task-icon-link"><img src="/jenkins/static/ae219983/images/24x24/next.png" style="width: 24px; height: 24px; width: 24px; height: 24px; margin: 2px;" class="icon-next icon-md" /></a>' 0xc2 0xa0 '<a href="https://jenkins.io/" class="task-link">Jenkins project</a></div><div class="task"><a href="https://jenkins.io/redirect/report-an-issue" class="task-icon-link"><img src="/jenkins/static/ae219983/images/24x24/gear2.png" style="width: 24px; height: 24px; width: 24px; height: 24px; margin: 2px;" class="icon-gear2 icon-md" /></a>' 0xc2 0xa0 '<a href="https://jenkins.io/redirect/report-an-issue" class="task-link">Bug tracker</a></div><div class="task"><a href="https://jenkins.io/redirect/mailing-lists" class="task-icon-link"><img src="/jenkins/static/ae219983/images/24x24/search.png" style="width: 24px; height: 24px; width: 24px; height: 24px; margin: 2px;" class="icon-search icon-md" /></a>' 0xc2 0xa0 '<a href="https://jenkins.io/redirect/mailing-lists" class="task-link">Mailing Lists</a></div><div class="task"><a href="https://twitter.com/jenkinsci" class="task-icon-link"><img src="/jenkins/static/ae219983/images/24x24/user.png" style="width: 24px; height: 24px; width: 24px; height: 24px; margin: 2px;" class="icon-user icon-md" /></a>' 0xc2 0xa0 '<a href="https://twitter.com/jenkinsci" class="task-link">Twitter: @jenkinsci</a></div></div><div id="main-panel"><a name="skip2content"></a><h1 style="text-align: center"><img src="/jenkins/static/ae219983/images/rage.png" width="154" height="179" /><span style="font-size:50px">' 0xc2 0xa0 'Oops!</span></h1><div id="error-description"><p>A problem occurred while processing the request.' 0x0a
' Please check <a href="https://jenkins.io/redirect/issue-tracker">our bug tracker</a> to see if a similar problem has already been reported.' 0x0a
' If it is already reported, please vote and put a comment on it to let us gauge the impact of the problem.' 0x0a
' If you think this is a new issue, please file a new issue.' 0x0a
' When you file an issue, make sure to add the entire stack trace, along with the version of Jenkins and relevant plugins.' 0x0a
' <a href="https://jenkins.io/redirect/users-mailing-list">The users list</a> might be also useful in understanding what has happened.</p><h2>Stack trace</h2><pre style="margin:2em; clear:both">java.io.IOException: HTTP full-duplex channel timeout: de33de07-093e-4b68-b2ff-c2b03cc92116' 0x0a
0x09 'at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:104)' 0x0a
0x09 'at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:171)' 0x0a
0x09 'at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)' 0x0a
0x09 'at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)' 0x0a
0x09 'at org.kohsuke.stapler.Function.renderResponse(Function.java:136)' 0x0a
0x09 'at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:676)' 0x0a
0x09 'at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)' 0x0a
0x09 'at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)' 0x0a
0x09 'at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)' 0x0a
0x09 'at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)' 0x0a
0x09 'at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)' 0x0a
0x09 'at org.kohsuke.stapler.Stapler.service(Stapler.java:238)' 0x0a
0x09 'at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)' 0x0a
0x09 'at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)' 0x0a
0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)' 0x0a
0x09 'at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)' 0x0a
0x09 'at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)' 0x0a
0x09 'at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)' 0x0a
0x09 'at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)' 0x0a
0x09 'at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)' 0x0a
0x09 'at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)' 0x0a
0x09 'at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:220)' 0x0a
0x09 'at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:197)' 0x0a
0x09 'at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85)' 0x0a
0x09 'at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:110)' 0x0a
0x09 'at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)' 0x0a
0x09 'at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)' 0x0a
0x09 'at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)' 0x0a
0x09 'at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)' 0x0a
0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)' 0x0a
0x09 'at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)' 0x0a
0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)' 0x0a
0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)' 0x0a
0x09 'at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)' 0x0a
0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
0x09 'at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)' 0x0a
0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
0x09 'at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)' 0x0a
0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
0x09 'at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)' 0x0a
0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
0x09 'at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)' 0x0a
0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
0x09 'at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)' 0x0a
0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
0x09 'at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)' 0x0a
0x09 'at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)' 0x0a
0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
0x09 'at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)' 0x0a
0x09 'at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)' 0x0a
0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)' 0x0a
0x09 'at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)' 0x0a
0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)' 0x0a
0x09 'at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)' 0x0a
0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)' 0x0a
0x09 'at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)' 0x0a
0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)' 0x0a
0x09 'at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)' 0x0a
0x09 'at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)' 0x0a
0x09 'at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)' 0x0a
0x09 'at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)' 0x0a
0x09 'at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)' 0x0a
0x09 'at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)' 0x0a
0x09 'at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)' 0x0a
0x09 'at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)' 0x0a
0x09 'at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)' 0x0a
0x09 'at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)' 0x0a
0x09 'at org.eclipse.jetty.server.Server.handle(Server.java:499)' 0x0a
0x09 'at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)' 0x0a
0x09 'at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)' 0x0a
0x09 'at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)' 0x0a
0x09 'at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)' 0x0a
0x09 'at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)' 0x0a
0x09 'at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)' 0x0a
0x09 'at java.lang.Thread.run(Thread.java:745)' 0x0a
'</pre></div></div></div><footer><div class="container-fluid"><div class="row"><div class="col-md-6" id="footer"></div><div class="col-md-18"><span class="page_generated">Page generated: Apr 18, 2017 2:52:51 PM EDT</span><span class="rest_api"><a href="api/">REST API</a></span><span class="jenkins_ver"><a href="https://jenkins.io/">Jenkins ver. 2.55</a></span><div id="l10n-dialog" class="dialog"></div><div id="l10n-footer" style="display:none; float:left"><a href="#" onclick="return showTranslationDialog();"><img src="/jenkins/static/ae219983/plugin/translation/flags.png" />' 0x0a
' Help us localize this page' 0x0a
' </a></div><script>var footer = document.getElementById('l10n-footer');' 0x0a
' var f = document.getElementById('footer');' 0x0a
' f.insertBefore(footer,f.firstChild);' 0x0a
' footer.style.display="block";' 0x0a
0x0a
' var translation=\{}; ' 0x0a
' translation.bundles = "zl/2HjJoYsA4G5Aoa2bNKadKh4JfwqpdRdJbSHr3MQZGr6VaP0TXQpVeM9l/bQPgwmVFWWDNcA3gt85D1DIl8HPL/s3rxkPakAC+jBYIYOXbqgc0/vfL67blvQhM5/yMaNppeEelsTB0ZQstIocXiTgX7R17lgSyFRpleKD1GiOVmB40Jc9dY8KgRoLGUSpWf+gdkOjICSaTOIXsu0p1nFg3SVsKXC5sMlYpweqBtJYTM6H7YxIHNqL5IERf2Sj+xcOL91KX8G/PQhidTxRg0jE++2LW+CHsxY6mp0tq3bQhS8R/YylabRph+94qU5UXqjr/W/uCOkQ9s+eeDEtQExPMda7e8inW/pasENsu2Lbt4IIQeHzYtnBSW9owcLIjXwD9A3JgvstopF+gFOOT5Rcu6f3lBckCs3gns0heTpkjsBIwEjQdPsBFfYl4QY99wOubrXDQBYxIohzMiXdEEA==";' 0x0a
' translation.detectedLocale = "";' 0x0a
0x0a
' function showTranslationDialog() \{' 0x0a
' if(!translation.launchDialog)' 0x0a
' loadScript("/jenkins/static/ae219983/plugin/translation/dialog.js");' 0x0a
' else' 0x0a
' translation.launchDialog();' 0x0a
' return false; ' 0x0a
' }</script><script>scmSyncConfigPageDecorator=makeStaplerProxy('/jenkins/$stapler/bound/4915af10-c7f2-4768-90eb-01d5ece548a2','01819747-82a3-4ff6-9404-4adba2f143b8',['purgeScmSyncConfigLogs']);</script><script type="text/javascript">' 0x0a
' function removeLog() \{' 0x0a
' ' 0x0a
' alert("Only administrators can purge logs !");' 0x0a
' return;' 0x0a
' ' 0x0a
' scmSyncConfigPageDecorator.purgeScmSyncConfigLogs();' 0x0a
' alert("SCM Sync Config errors purged !");' 0x0a
' document.location.reload(true);' 0x0a
' }' 0x0a
0x09 '</script><table width="100%"><tr><td id="footer"><span style="color:gray">SCM Sync status : <img src="/jenkins/static/ae219983/images/16x16/health-80plus.png" alt="" width="16" height="16" />' 0x0a
' Last operation @ Tue Apr 18 14:43:25 EDT 2017' 0x0a
' </span></td></tr></table></div></div></div></footer></body></html>'
Read ahead:
at hudson.cli.FlightRecorderInputStream.analyzeCrash(FlightRecorderInputStream.java:82)
at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:153)
Caused by: java.io.EOFException: Bytes to skip: 168430090 actual: 17752
at org.apache.commons.io.IOUtils.skipFully(IOUtils.java:2360)
at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:132)
- blocks
-
JENKINS-41745 Remoting-free CLI
-
- Resolved
-
- is duplicated by
-
JENKINS-44387 jenkins-cli test fail
-
- Resolved
-
- relates to
-
-
- Resolved
-
- links to
[JENKINS-43666] HTTP full-duplex channel timeout from FullDuplexHttpService.download
java.io.IOException: HTTP full-duplex channel timeout: de33de07-093e-4b68-b2ff-c2b03cc92116
I would expect there's a reverse proxy in front of Jenkins and it is configured with fairly aggressive request timeouts.
Todd Strangio
added a comment - There's an nginx server in front of Jenkins, but something changes between Jenkins versions 2.53 and 2.54.
Todd Strangio
added a comment - There's an nginx server in front of Jenkins, but something changes between Jenkins versions 2.53 and 2.54. but something changes between Jenkins versions 2.53 and 2.54.
https://jenkins-ci.org/changelog/#v2.54
Wei-min Lee
added a comment - Same problem on LTS 2.46.2, but not on 2.46.1. Can this be worked around by using the -remoting mode of the CLI client? (Only if newly downloaded jar, old jars pre-2.54/2.46.2 use this as the default and actually only possible mode)
jglick Well, if they're following https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+behind+an+NGinX+reverse+proxy then really our docs are wrong.
Possibly. Would need to be tested. I have not yet tested against my Apache demo either.
Wei-min Lee
added a comment - Tried -remoting - no success...
java.io.EOFException
at java.io.DataInputStream.readFully(Unknown Source)
at java.io.DataInputStream.readUTF(Unknown Source)
at java.io.DataInputStream.readUTF(Unknown Source)
at hudson.cli.CLI.connectViaCliPort(CLI.java:246)
at hudson.cli.CLI.<init>(CLI.java:137)
at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:82)
at hudson.cli.CLI._main(CLI.java:585)
at hudson.cli.CLI.main(CLI.java:399)
Suppressed: java.io.EOFException: unexpected stream termination
at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:365)
at hudson.remoting.Channel.<init>(Channel.java:461)
at hudson.remoting.Channel.<init>(Channel.java:439)
at hudson.remoting.Channel.<init>(Channel.java:435)
at hudson.remoting.Channel.<init>(Channel.java:423)
at hudson.remoting.Channel.<init>(Channel.java:414)
at hudson.remoting.Channel.<init>(Channel.java:387)
at hudson.cli.CLI.connectViaHttp(CLI.java:164)
at hudson.cli.CLI.<init>(CLI.java:141)
... 3 more
Wei-min Lee
added a comment - Tried -remoting - no success...
java.io.EOFException
at java.io.DataInputStream.readFully(Unknown Source)
at java.io.DataInputStream.readUTF(Unknown Source)
at java.io.DataInputStream.readUTF(Unknown Source)
at hudson.cli.CLI.connectViaCliPort(CLI.java:246)
at hudson.cli.CLI.<init>(CLI.java:137)
at hudson.cli.CLIConnectionFactory.connect(CLIConnectionFactory.java:82)
at hudson.cli.CLI._main(CLI.java:585)
at hudson.cli.CLI.main(CLI.java:399)
Suppressed: java.io.EOFException: unexpected stream termination
at hudson.remoting.ChannelBuilder.negotiate(ChannelBuilder.java:365)
at hudson.remoting.Channel.<init>(Channel.java:461)
at hudson.remoting.Channel.<init>(Channel.java:439)
at hudson.remoting.Channel.<init>(Channel.java:435)
at hudson.remoting.Channel.<init>(Channel.java:423)
at hudson.remoting.Channel.<init>(Channel.java:414)
at hudson.remoting.Channel.<init>(Channel.java:387)
at hudson.cli.CLI.connectViaHttp(CLI.java:164)
at hudson.cli.CLI.<init>(CLI.java:141)
... 3 more 
Ryan Campbell
added a comment - Can you please confirm that you're running the latest jenkins-cli.jar from the release in question?
Ryan Campbell
added a comment - Can you please confirm that you're running the latest jenkins-cli.jar from the release in question? campbellr updated the wiki page, make sure that your nginx config is adapted accordingly:
https://wiki.jenkins-ci.org/pages/diffpages.action?pageId=66847445&originalId=118095978
Wei-min Lee
added a comment - Hi recampbell, I did download the jenkins-cli.jar from my 2.46.2 server when I was having these problems. I had to revert back to 2.46.1 on both the server and the cli jar.
Wei-min Lee
added a comment - Hi recampbell , I did download the jenkins-cli.jar from my 2.46.2 server when I was having these problems. I had to revert back to 2.46.1 on both the server and the cli jar. My mistake...my issue is not with the new cli, but instead with my apache proxy. Don't know why I didn't make the connection after reading the nginx document reference. I initially followed the documentation for apache proxy config (https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache), but clearly it needs updating. Would anyone know what changes need to be made?
Wei-min Lee
added a comment - - edited My mistake...my issue is not with the new cli, but instead with my apache proxy. Don't know why I didn't make the connection after reading the nginx document reference. I initially followed the documentation for apache proxy config ( https://wiki.jenkins-ci.org/display/JENKINS/Running+Jenkins+behind+Apache) , but clearly it needs updating. Would anyone know what changes need to be made? I do not; would need to spend time reproducing using the aforementioned proxy-in-a-container.
Wei-min Lee
added a comment - I'm able to work-around the apache proxy issue by specifying the actual port that jenkins is listening on.
Wei-min Lee
added a comment - I'm able to work-around the apache proxy issue by specifying the actual port that jenkins is listening on. Unfortunately the proposed changes work only for Nginx >= 1.7.11, which supports proxy_request_buffering; I am stuck on an older version packaged with Ubuntu 14.04 and hence the proposed configuration does not work for me.
I am able too to work around this using -s http:localhost:8080 (or whatever port Jenkins is listening on}}.
We have discovered the following possible workaround, which requires falling back to the remoting CLI as suggested by Daniel Beck in a comment above. I'm not sure why this didn't work for bleeme. Correct placement of the -remoting option on the command line is important. I flailed around a bit.
Environment:
Jenkins 2.46.2 server with "Enable CLI over Remoting" checked in the global security config.
Apache httpd reverse proxy.
Jenkins CLI 2.46.2
java -jar jenkins-cli.jar -version Version: 2.46.2
This fails with the error described in this issue:
java -jar jenkins-cli.jar -s http://myjenkinsurl/ help
Forcing the cli to use remoting succeeds.
java -jar jenkins-cli.jar -s http://myjenkinsurl/ -remoting help
Here's another working example.
java -jar jenkins-cli.jar -remoting -s http://myjenkinsurl get-job some_job_name --username myuser --password sillypassword
If it was not already obvious, this is a bad (insecure) workaround. Better to fix your proxy configuration, or use an unproxied URL for Jenkins.
It is not obvious (to me, at least) how to "fix" an Apache httpd reverse proxy. There doesn't appear to be an equivalent to nginx's proxy_request_buffering.
Apache mod_proxy has ProxyReceiveBufferSize, but I haven't discovered a way for buffering to be switched off.
If you do figure it out, consider filing a PR against this repo and we can also update documentation to match.
eddy kim
added a comment - I was getting the same symptom. It turned out to be that the master was using jenkins 2.60 and java 8, but the existing slaves were using java 7. I made sure that the jenkins-client.jar and slave.jar were the latest on those slaves, but i forgot about the jvm. hope that helps others also hitting this problem when upgrading.
eddy kim
added a comment - I was getting the same symptom. It turned out to be that the master was using jenkins 2.60 and java 8, but the existing slaves were using java 7. I made sure that the jenkins-client.jar and slave.jar were the latest on those slaves, but i forgot about the jvm. hope that helps others also hitting this problem when upgrading. Also using Jenkins 2.60.1 and java 8 with nginx 1.10.0 on ubuntu 16.04 Setting the proxy_request_buffering off; directive in Nginx does not solve the issue as this doc would suggest https://wiki.jenkins-ci.org/pages/diffpages.action?pageId=66847445&originalId=118095978 remoting is a workaround but not a satisfactory one imo, neither is bypassing the ssl proxy in-front of jenkins.
jay bendon
added a comment - - edited Also using Jenkins 2.60.1 and java 8 with nginx 1.10.0 on ubuntu 16.04 Setting the proxy_request_buffering off; directive in Nginx does not solve the issue as this doc would suggest https://wiki.jenkins-ci.org/pages/diffpages.action?pageId=66847445&originalId=118095978 remoting is a workaround but not a satisfactory one imo, neither is bypassing the ssl proxy in-front of jenkins. 
Phu Hoang Xuan
added a comment - Reading Nginx' documentation I found a related directive proxy_buffering that is missing from the wiki page. Setting that to off makes things work in my case. Here's the entirety of the proxy setting:
location @jenkins {
proxy_buffering off;
proxy_http_version 1.1;
proxy_pass http://127.0.0.1:65201;
proxy_request_buffering off;
}
Phu Hoang Xuan
added a comment - Reading Nginx' documentation I found a related directive proxy_buffering that is missing from the wiki page. Setting that to off makes things work in my case. Here's the entirety of the proxy setting:
location @jenkins {
proxy_buffering off;
proxy_http_version 1.1;
proxy_pass http: //127.0.0.1:65201;
proxy_request_buffering off;
}
recampbell PTAL at the previous comment. Did you encounter anything like this in your use of nginx?
We are currently using NGINX 1.10.2 (default package) and are on Jenkins 2.63. This is what our config looks like for our HTTPS instance that hopefully somebody will point a problem.
## redirect http to https
server {
listen 80;
server_name our.jenkins.instance.com;
return 301 https://$host$request_uri;
}
## server configuration
server {
listen 443 ssl;
server_name our.jenkins.instance.com;
## add ssl entries when https has been set in config
ssl on;
ssl_certificate_key /etc/nginx/ssl/our.jenkins.instance.com.key;
ssl_certificate /etc/nginx/ssl/our.jenkins.instance.com.pem;
ssl_session_cache shared:SSL:1m;
ssl_prefer_server_ciphers on;
if ($http_x_forwarded_proto = '') {
set $http_x_forwarded_proto $scheme;
}
chunked_transfer_encoding on;
client_max_body_size 0;
location / {
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Fix the "It appears that your reverse proxy set up is broken" error.
proxy_pass http://127.0.0.1:8080;
proxy_read_timeout 90;
proxy_redirect http://127.0.0.1:8080 https://our.jenkins.instance.com;
proxy_http_version 1.1;
proxy_request_buffering off;
}
}
I'm sure we have nothing weird going on, but I am having a difficult time reproducing it without spending a bunch of time.
Ok, I went through many different permutations and I think it is caused by the
proxy_read_timeout 90;
piece. Still need to dive in a little more.
danielbeck the solution above for settings
proxy_buffering off; proxy_request_buffering off;
Seems to be working for me. Will probably be back if they do not
danielbeck, I can confirm that disabling the buffering does solve the broken CLI issue. I find bit silly that this is needs to be done when the nginx proxy is used only on HTTP port and not on SSH/CLI one!
the nginx proxy is used only on HTTP port and not on SSH/CLI one
There are three transports for the CLI:
- SSH, using SSH protocol
- TCP, using deprecated Remoting protocol
- HTTP Duplex, using either the deprecated Remoting protocol, or the new “plain” protocol
This issue is about the last option. The CLI client is using HTTP and going through nginx. Since the HTTP Duplex transport requires connections to be long-lasting and unbuffered, any reverse proxy must be configured to act transparently in these regards.
After applying the workaround from https://issues.jenkins-ci.org/browse/JENKINS-43666?focusedCommentId=311367&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-311367 I was able to make the jar+HTTP approach work.
I am still trying to make jar+ssh and ssh approaches to work, preferably the pure ssh one as that's the most reliable one.
java -jar jenkins-cli.jar -ssh -user ssbarnea -i ~/.ssh/id_rsa -p jenkins.example.com:50000 -logger FINEST help
Aug 22, 2017 10:24:07 PM hudson.cli.CLI _main
FINE: using connection mode SSH
Aug 22, 2017 10:24:08 PM hudson.cli.SSHCLI sshConnection
WARNING: No header 'X-SSH-Endpoint' returned by Jenkins
FAIL: 255
This error looks bit confusing to me because I manually specified it to use `-ssh` method, and as far as I know SSH protocol does not involve HTTP headers.
If you use jenkins-cli.jar -ssh, it will first make a simple (RESTish) HTTP connection to Jenkins just to look up the X-SSH-Endpoint header, which gives it a host and port to use to make the SSH connection. If you use /usr/bin/ssh you are on your own to specify these.
Your error suggests that nginx is dropping this header for some reason, or Jenkins was not serving it to begin with. Check a direct (non-proxied) Jenkins URL for comparison. Anyway that is off topic for this issue; use the users’ list.
Note new test of HTTP Duplex mode since I have seen some issues with Jetty 9.4.x.
https://github.com/jenkinsci/jenkins-test-harness/pull/69 suggests that Jetty’s default configuration may simply be broken on certain machines. Not sure how this would not have been noticed by zillions of other people, though the Jenkins HTTP Duplex trick is pretty strange.
I am seeing this issue as well.
Jenkins Core: 2.60.3
OS Ubuntu: 16.04
Nov 22, 2017 2:14:28 PM hudson.init.impl.InstallUncaughtExceptionHandler$1 reportException
WARNING: null
java.io.IOException: HTTP full-duplex channel timeout: 1e389d87-5a4b-4342-bb47-8b1cd61ee203
at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:104)
at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:171)
at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
at org.kohsuke.stapler.Function.renderResponse(Function.java:136)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:676)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:232)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:209)
at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:113)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.
Samuel Lopez
added a comment - - edited I am seeing this issue as well.
Jenkins Core: 2.60.3
OS Ubuntu: 16.04
Nov 22, 2017 2:14:28 PM hudson.init.impl.InstallUncaughtExceptionHandler$1 reportException
WARNING: null
java.io.IOException: HTTP full-duplex channel timeout: 1e389d87-5a4b-4342-bb47-8b1cd61ee203
at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:104)
at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:171)
at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
at org.kohsuke.stapler.Function.renderResponse(Function.java:136)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:676)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:232)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:209)
at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:113)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter. 
CC jglick