Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-43666

HTTP full-duplex channel timeout from FullDuplexHttpService.download

    • Icon: Bug Bug
    • Resolution: Won't Fix
    • Icon: Major Major
    • cli
    • None
    • Jenkins master 2.55 and 2.54 on RHEL7.3 launched using Oracle 1.8 JDK,
      RHEL6.8 client using Oracle 1.8 JDK, RHEL7 client using Oracle 1.8 JDK, RHEL7 client using OpenJDK1.8

      Running any Jenkins CLI command throws the exception below.  TCP port for JNLP was active, user running CLI command had full admin permissions, error happens using '-noCertificateCheck' flag or omitting it, Anonymous user has overall read permission.  Able to reproduce this error on Jenkins 2.55 & 2.54 ... backing down to 2.53 works just fine.

      [devops@removed ~]$ /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.121-0.b13.el7_3.x86_64/jre/bin/java -jar /tmp/jenkins-cli.jar -s https://removed.swg.usma.ibm.com/jenkins -noCertificateCheck help
      Apr 18, 2017 2:52:34 PM hudson.cli.CLI _main
      INFO: Skipping HTTPS certificate checks altogether. Note that this is not secure at all.
      Apr 18, 2017 2:52:51 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run
      WARNING: unknown operation #10: 0x0a
      0x0a
      0x0a
      0x0a
      0x0a
      Apr 18, 2017 2:52:51 PM hudson.cli.PlainCLIProtocol$EitherSide$Reader run
      WARNING: null
      hudson.cli.DiagnosedStreamCorruptionException
      Read back: 0x0a
      0x0a
      0x0a
      0x0a
      0x0a
      '  ' 0x0a
      '  <!DOCTYPE html><html><head resURL="/jenkins/static/ae219983" data-rooturl="/jenkins" data-resurl="/jenkins/static/ae219983">' 0x0a
      '    ' 0x0a
      0x0a
      '    <title>Jenkins [Jenkins]</title><link rel="stylesheet" href="/jenkins/static/ae219983/css/layout-common.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/css/style.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/css/color.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/css/responsive-grid.css" type="text/css" /><link rel="shortcut icon" href="/jenkins/static/ae219983/favicon.ico" type="image/vnd.microsoft.icon" /><link color="black" rel="mask-icon" href="/jenkins/images/mask-icon.svg" /><script>var isRunAsTest=false; var rootURL="/jenkins"; var resURL="/jenkins/static/ae219983";</script><script src="/jenkins/static/ae219983/scripts/prototype.js" type="text/javascript"></script><script src="/jenkins/static/ae219983/scripts/behavior.js" type="text/javascript"></script><script src='/jenkins/adjuncts/ae219983/org/kohsuke/stapler/bind.js' type='text/javascript'></script><script src="/jenkins/static/ae219983/scripts/yui/yahoo/yahoo-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/dom/dom-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/event/event-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/animation/animation-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/dragdrop/dragdrop-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/container/container-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/connection/connection-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/datasource/datasource-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/autocomplete/autocomplete-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/menu/menu-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/element/element-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/button/button-min.js"></script><script src="/jenkins/static/ae219983/scripts/yui/storage/storage-min.js"></script><script src="/jenkins/static/ae219983/scripts/hudson-behavior.js" type="text/javascript"></script><script src="/jenkins/static/ae219983/scripts/sortable.js" type="text/javascript"></script><script>crumb.init("", "");</script><link rel="stylesheet" href="/jenkins/static/ae219983/scripts/yui/container/assets/container.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/scripts/yui/assets/skins/sam/skin.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/scripts/yui/container/assets/skins/sam/container.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/scripts/yui/button/assets/skins/sam/button.css" type="text/css" /><link rel="stylesheet" href="/jenkins/static/ae219983/scripts/yui/menu/assets/skins/sam/menu.css" type="text/css" /><link rel="search" href="/jenkins/opensearch.xml" type="application/opensearchdescription+xml" title="Jenkins" /><meta name="ROBOTS" content="INDEX,NOFOLLOW" /><meta name="viewport" content="width=device-width, initial-scale=1" /><script src="/jenkins/adjuncts/ae219983/org/kohsuke/stapler/jquery/jquery.full.js" type="text/javascript"></script><script>var Q=jQuery.noConflict()</script><script>' 0x0a
      '      if(window.Prototype && JSON) \{' 0x0a
      '        var _json_stringify = JSON.stringify;' 0x0a
      '        JSON.stringify = function(value) \{' 0x0a
      '            var _array_tojson = Array.prototype.toJSON;' 0x0a
      '            delete Array.prototype.toJSON;' 0x0a
      '            var r=_json_stringify(value);' 0x0a
      '            Array.prototype.toJSON = _array_tojson;' 0x0a
      '            return r;' 0x0a
      '        };' 0x0a
      '     }  ' 0x0a
      '   </script><script src="/jenkins/plugin/extended-choice-parameter/js/selectize.min.js" type="text/javascript"></script><script src="/jenkins/plugin/extended-choice-parameter/js/jsoneditor.min.js" type="text/javascript"></script><script src="/jenkins/plugin/extended-choice-parameter/js/jquery.jsonview.min.js" type="text/javascript"></script><link rel="stylesheet" href="/jenkins/plugin/extended-choice-parameter/css/jquery.jsonview.css" /><link rel="stylesheet" id="icon_stylesheet" href="/jenkins/plugin/extended-choice-parameter/css/selectize.css" /><link rel="stylesheet" id="icon_stylesheet" href="/jenkins/plugin/extended-choice-parameter/css/selectize.bootstrap2.css" /><link rel="stylesheet" id="theme_stylesheet" /><link rel="stylesheet" id="icon_stylesheet" /><script src="/jenkins/static/ae219983/scripts/yui/cookie/cookie-min.js"></script><script src="/jenkins/static/ae219983/jsbundles/page-init.js" type="text/javascript"></script></head><body data-model-type="hudson.model.Hudson" id="jenkins" class="yui-skin-sam two-column jenkins-2.55" data-version="2.55"><a href="#skip2content" class="skiplink">Skip to content</a><div id="page-head"><div id="header"><div class="logo"><a id="jenkins-home-link" href="/jenkins/"><img src="/jenkins/static/ae219983/images/headshot.png" alt="title" id="jenkins-head-icon" /><img src="/jenkins/static/ae219983/images/title.png" alt="title" width="139" id="jenkins-name-icon" height="34" /></a></div><div class="login">' 0xc2 0xa0 '<a href="/jenkins/login?from=%2Fjenkins%2Fcli"><b>log in</b></a></div><div class="searchbox hidden-xs"><form method="get" name="search" action="/jenkins/search/" style="position:relative;" class="no-json"><div id="search-box-minWidth"></div><div id="search-box-sizer"></div><div id="searchform"><input name="q" placeholder="search" id="search-box" class="has-default-text" />' 0xc2 0xa0 '<a href="https://jenkins.io/redirect/search-box"><img src="/jenkins/static/ae219983/images/16x16/help.png" style="width: 16px; height: 16px; " class="icon-help icon-sm" /></a><div id="search-box-completion"></div><script>createSearchBox("/jenkins/search/");</script></div></form></div></div><div id="breadcrumbBar"><tr id="top-nav"><td id="left-top-nav" colspan="2"><link rel='stylesheet' href='/jenkins/adjuncts/ae219983/lib/layout/breadcrumbs.css' type='text/css' /><script src='/jenkins/adjuncts/ae219983/lib/layout/breadcrumbs.js' type='text/javascript'></script><div class="top-sticker noedge"><div class="top-sticker-inner"><div id="right-top-nav"></div><ul id="breadcrumbs"><li class="item"><a href="/jenkins/" class="model-link inside">Jenkins</a></li><li href="/jenkins/" class="children"></li></ul><div id="breadcrumb-menu-target"></div></div></div></td></tr></div></div><div id="page-body" class="clear"><div id="side-panel"><div class="task"><a href="https://jenkins.io/" class="task-icon-link"><img src="/jenkins/static/ae219983/images/24x24/next.png" style="width: 24px; height: 24px; width: 24px; height: 24px; margin: 2px;" class="icon-next icon-md" /></a>' 0xc2 0xa0 '<a href="https://jenkins.io/" class="task-link">Jenkins project</a></div><div class="task"><a href="https://jenkins.io/redirect/report-an-issue" class="task-icon-link"><img src="/jenkins/static/ae219983/images/24x24/gear2.png" style="width: 24px; height: 24px; width: 24px; height: 24px; margin: 2px;" class="icon-gear2 icon-md" /></a>' 0xc2 0xa0 '<a href="https://jenkins.io/redirect/report-an-issue" class="task-link">Bug tracker</a></div><div class="task"><a href="https://jenkins.io/redirect/mailing-lists" class="task-icon-link"><img src="/jenkins/static/ae219983/images/24x24/search.png" style="width: 24px; height: 24px; width: 24px; height: 24px; margin: 2px;" class="icon-search icon-md" /></a>' 0xc2 0xa0 '<a href="https://jenkins.io/redirect/mailing-lists" class="task-link">Mailing Lists</a></div><div class="task"><a href="https://twitter.com/jenkinsci" class="task-icon-link"><img src="/jenkins/static/ae219983/images/24x24/user.png" style="width: 24px; height: 24px; width: 24px; height: 24px; margin: 2px;" class="icon-user icon-md" /></a>' 0xc2 0xa0 '<a href="https://twitter.com/jenkinsci" class="task-link">Twitter: @jenkinsci</a></div></div><div id="main-panel"><a name="skip2content"></a><h1 style="text-align: center"><img src="/jenkins/static/ae219983/images/rage.png" width="154" height="179" /><span style="font-size:50px">' 0xc2 0xa0 'Oops!</span></h1><div id="error-description"><p>A problem occurred while processing the request.' 0x0a
      '        Please check <a href="https://jenkins.io/redirect/issue-tracker">our bug tracker</a> to see if a similar problem has already been reported.' 0x0a
      '        If it is already reported, please vote and put a comment on it to let us gauge the impact of the problem.' 0x0a
      '        If you think this is a new issue, please file a new issue.' 0x0a
      '        When you file an issue, make sure to add the entire stack trace, along with the version of Jenkins and relevant plugins.' 0x0a
      '        <a href="https://jenkins.io/redirect/users-mailing-list">The users list</a> might be also useful in understanding what has happened.</p><h2>Stack trace</h2><pre style="margin:2em; clear:both">java.io.IOException: HTTP full-duplex channel timeout: de33de07-093e-4b68-b2ff-c2b03cc92116' 0x0a
      0x09 'at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:104)' 0x0a
      0x09 'at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:171)' 0x0a
      0x09 'at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)' 0x0a
      0x09 'at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)' 0x0a
      0x09 'at org.kohsuke.stapler.Function.renderResponse(Function.java:136)' 0x0a
      0x09 'at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:676)' 0x0a
      0x09 'at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)' 0x0a
      0x09 'at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)' 0x0a
      0x09 'at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)' 0x0a
      0x09 'at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)' 0x0a
      0x09 'at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)' 0x0a
      0x09 'at org.kohsuke.stapler.Stapler.service(Stapler.java:238)' 0x0a
      0x09 'at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)' 0x0a
      0x09 'at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)' 0x0a
      0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)' 0x0a
      0x09 'at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)' 0x0a
      0x09 'at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)' 0x0a
      0x09 'at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)' 0x0a
      0x09 'at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)' 0x0a
      0x09 'at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)' 0x0a
      0x09 'at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)' 0x0a
      0x09 'at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:220)' 0x0a
      0x09 'at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:197)' 0x0a
      0x09 'at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:85)' 0x0a
      0x09 'at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:110)' 0x0a
      0x09 'at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)' 0x0a
      0x09 'at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)' 0x0a
      0x09 'at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)' 0x0a
      0x09 'at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)' 0x0a
      0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)' 0x0a
      0x09 'at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)' 0x0a
      0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)' 0x0a
      0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)' 0x0a
      0x09 'at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)' 0x0a
      0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
      0x09 'at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)' 0x0a
      0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
      0x09 'at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)' 0x0a
      0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
      0x09 'at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)' 0x0a
      0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
      0x09 'at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)' 0x0a
      0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
      0x09 'at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)' 0x0a
      0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
      0x09 'at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)' 0x0a
      0x09 'at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)' 0x0a
      0x09 'at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)' 0x0a
      0x09 'at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)' 0x0a
      0x09 'at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)' 0x0a
      0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)' 0x0a
      0x09 'at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)' 0x0a
      0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)' 0x0a
      0x09 'at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)' 0x0a
      0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)' 0x0a
      0x09 'at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)' 0x0a
      0x09 'at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)' 0x0a
      0x09 'at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)' 0x0a
      0x09 'at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)' 0x0a
      0x09 'at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)' 0x0a
      0x09 'at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)' 0x0a
      0x09 'at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)' 0x0a
      0x09 'at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)' 0x0a
      0x09 'at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)' 0x0a
      0x09 'at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)' 0x0a
      0x09 'at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)' 0x0a
      0x09 'at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)' 0x0a
      0x09 'at org.eclipse.jetty.server.Server.handle(Server.java:499)' 0x0a
      0x09 'at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)' 0x0a
      0x09 'at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)' 0x0a
      0x09 'at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)' 0x0a
      0x09 'at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)' 0x0a
      0x09 'at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)' 0x0a
      0x09 'at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)' 0x0a
      0x09 'at java.lang.Thread.run(Thread.java:745)' 0x0a
      '</pre></div></div></div><footer><div class="container-fluid"><div class="row"><div class="col-md-6" id="footer"></div><div class="col-md-18"><span class="page_generated">Page generated: Apr 18, 2017 2:52:51 PM EDT</span><span class="rest_api"><a href="api/">REST API</a></span><span class="jenkins_ver"><a href="https://jenkins.io/">Jenkins ver. 2.55</a></span><div id="l10n-dialog" class="dialog"></div><div id="l10n-footer" style="display:none; float:left"><a href="#" onclick="return showTranslationDialog();"><img src="/jenkins/static/ae219983/plugin/translation/flags.png" />' 0x0a
      '      Help us localize this page' 0x0a
      '    </a></div><script>var footer = document.getElementById('l10n-footer');' 0x0a
      '    var f = document.getElementById('footer');' 0x0a
      '    f.insertBefore(footer,f.firstChild);' 0x0a
      '    footer.style.display="block";' 0x0a
      0x0a
      '    var translation=\{}; ' 0x0a
      '    translation.bundles = "zl/2HjJoYsA4G5Aoa2bNKadKh4JfwqpdRdJbSHr3MQZGr6VaP0TXQpVeM9l/bQPgwmVFWWDNcA3gt85D1DIl8HPL/s3rxkPakAC+jBYIYOXbqgc0/vfL67blvQhM5/yMaNppeEelsTB0ZQstIocXiTgX7R17lgSyFRpleKD1GiOVmB40Jc9dY8KgRoLGUSpWf+gdkOjICSaTOIXsu0p1nFg3SVsKXC5sMlYpweqBtJYTM6H7YxIHNqL5IERf2Sj+xcOL91KX8G/PQhidTxRg0jE++2LW+CHsxY6mp0tq3bQhS8R/YylabRph+94qU5UXqjr/W/uCOkQ9s+eeDEtQExPMda7e8inW/pasENsu2Lbt4IIQeHzYtnBSW9owcLIjXwD9A3JgvstopF+gFOOT5Rcu6f3lBckCs3gns0heTpkjsBIwEjQdPsBFfYl4QY99wOubrXDQBYxIohzMiXdEEA==";' 0x0a
      '    translation.detectedLocale = "";' 0x0a
      0x0a
      '    function showTranslationDialog() \{' 0x0a
      '      if(!translation.launchDialog)' 0x0a
      '        loadScript("/jenkins/static/ae219983/plugin/translation/dialog.js");' 0x0a
      '      else' 0x0a
      '        translation.launchDialog();' 0x0a
      '      return false; ' 0x0a
      '    }</script><script>scmSyncConfigPageDecorator=makeStaplerProxy('/jenkins/$stapler/bound/4915af10-c7f2-4768-90eb-01d5ece548a2','01819747-82a3-4ff6-9404-4adba2f143b8',['purgeScmSyncConfigLogs']);</script><script type="text/javascript">' 0x0a
      '        function removeLog() \{' 0x0a
      '            ' 0x0a
      '                alert("Only administrators can purge logs !");' 0x0a
      '                return;' 0x0a
      '            ' 0x0a
      '            scmSyncConfigPageDecorator.purgeScmSyncConfigLogs();' 0x0a
      '            alert("SCM Sync Config errors purged !");' 0x0a
      '            document.location.reload(true);' 0x0a
      '        }' 0x0a
      0x09 '</script><table width="100%"><tr><td id="footer"><span style="color:gray">SCM Sync status : <img src="/jenkins/static/ae219983/images/16x16/health-80plus.png" alt="" width="16" height="16" />' 0x0a
      '            Last operation @ Tue Apr 18 14:43:25 EDT 2017' 0x0a
      '            </span></td></tr></table></div></div></div></footer></body></html>'
      Read ahead: 
       at hudson.cli.FlightRecorderInputStream.analyzeCrash(FlightRecorderInputStream.java:82)
       at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:153)
      Caused by: java.io.EOFException: Bytes to skip: 168430090 actual: 17752
       at org.apache.commons.io.IOUtils.skipFully(IOUtils.java:2360)
       at hudson.cli.PlainCLIProtocol$EitherSide$Reader.run(PlainCLIProtocol.java:132)
      

          [JENKINS-43666] HTTP full-duplex channel timeout from FullDuplexHttpService.download

          Mike Kobit added a comment - - edited

          We are currently using NGINX 1.10.2 (default package) and are on Jenkins 2.63. This is what our config looks like for our HTTPS instance that hopefully somebody will point a problem.

          ## redirect http to https                                           
          server {                                                            
            listen                   80;                                      
            server_name              our.jenkins.instance.com;                   
            return                   301 https://$host$request_uri;           
          }                                                                   
          
          ## server configuration                                             
          server {                                                            
            listen 443 ssl;                                                   
            server_name our.jenkins.instance.com;                                
          
            ## add ssl entries when https has been set in config              
            ssl on;                                                           
            ssl_certificate_key  /etc/nginx/ssl/our.jenkins.instance.com.key;    
            ssl_certificate      /etc/nginx/ssl/our.jenkins.instance.com.pem;    
            ssl_session_cache shared:SSL:1m;                                  
            ssl_prefer_server_ciphers   on;                                   
          
            if ($http_x_forwarded_proto = '') {                               
              set $http_x_forwarded_proto  $scheme;                           
            }                                                                 
            chunked_transfer_encoding on;                                     
            client_max_body_size 0;                                           
            location / {                                                      
              proxy_set_header        Host $host:$server_port;                
              proxy_set_header        X-Real-IP $remote_addr;                 
              proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;                                                                 
              proxy_set_header        X-Forwarded-Proto $scheme;              
              # Fix the "It appears that your reverse proxy set up is broken" error.
              proxy_pass          http://127.0.0.1:8080;
              proxy_read_timeout  90;
              proxy_redirect      http://127.0.0.1:8080 https://our.jenkins.instance.com;
              proxy_http_version 1.1;
              proxy_request_buffering off;
            }
          }
          

          I'm sure we have nothing weird going on, but I am having a difficult time reproducing it without spending a bunch of time.

          Mike Kobit added a comment - - edited We are currently using NGINX 1.10.2 (default package) and are on Jenkins 2.63. This is what our config looks like for our HTTPS instance that hopefully somebody will point a problem. ## redirect http to https server { listen 80; server_name our.jenkins.instance.com; return 301 https: //$host$request_uri; } ## server configuration server { listen 443 ssl; server_name our.jenkins.instance.com; ## add ssl entries when https has been set in config ssl on; ssl_certificate_key /etc/nginx/ssl/our.jenkins.instance.com.key; ssl_certificate /etc/nginx/ssl/our.jenkins.instance.com.pem; ssl_session_cache shared:SSL:1m; ssl_prefer_server_ciphers on; if ($http_x_forwarded_proto = '') { set $http_x_forwarded_proto $scheme; } chunked_transfer_encoding on; client_max_body_size 0; location / { proxy_set_header Host $host:$server_port; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # Fix the "It appears that your reverse proxy set up is broken" error. proxy_pass http: //127.0.0.1:8080; proxy_read_timeout 90; proxy_redirect http: //127.0.0.1:8080 https://our.jenkins.instance.com; proxy_http_version 1.1; proxy_request_buffering off; } } I'm sure we have nothing weird going on, but I am having a difficult time reproducing it without spending a bunch of time.

          Mike Kobit added a comment - - edited

          Ok, I went through many different permutations and I think it is caused by the

          proxy_read_timeout 90;

          piece. Still need to dive in a little more.

          danielbeck the solution above for settings

          proxy_buffering off;
          proxy_request_buffering off;
          

          Seems to be working for me. Will probably be back if they do not

          Mike Kobit added a comment - - edited Ok, I went through many different permutations and I think it is caused by the proxy_read_timeout 90; piece. Still need to dive in a little more. danielbeck the solution above for settings proxy_buffering off; proxy_request_buffering off; Seems to be working for me. Will probably be back if they do not

          Sorin Sbarnea added a comment -

          danielbeck, I can confirm that disabling the buffering does solve the broken CLI issue. I find bit silly that this is needs to be done when the nginx proxy is used only on HTTP port and not on SSH/CLI one!

          Sorin Sbarnea added a comment - danielbeck , I can confirm that disabling the buffering does solve the broken CLI issue. I find bit silly that this is needs to be done when the nginx proxy is used only on HTTP port and not on SSH/CLI one!

          Jesse Glick added a comment -

          ssbarnea

          the nginx proxy is used only on HTTP port and not on SSH/CLI one

          There are three transports for the CLI:

          • SSH, using SSH protocol
          • TCP, using deprecated Remoting protocol
          • HTTP Duplex, using either the deprecated Remoting protocol, or the new “plain” protocol

          This issue is about the last option. The CLI client is using HTTP and going through nginx. Since the HTTP Duplex transport requires connections to be long-lasting and unbuffered, any reverse proxy must be configured to act transparently in these regards.

          Jesse Glick added a comment - ssbarnea the nginx proxy is used only on HTTP port and not on SSH/CLI one There are three transports for the CLI: SSH, using SSH protocol TCP, using deprecated Remoting protocol HTTP Duplex, using either the deprecated Remoting protocol, or the new “plain” protocol This issue is about the last option. The CLI client is using HTTP and going through nginx. Since the HTTP Duplex transport requires connections to be long-lasting and unbuffered, any reverse proxy must be configured to act transparently in these regards.

          Sorin Sbarnea added a comment - - edited

          After applying the workaround from https://issues.jenkins-ci.org/browse/JENKINS-43666?focusedCommentId=311367&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-311367 I was able to make the jar+HTTP approach work.

          I am still trying to make jar+ssh and ssh approaches to work, preferably the pure ssh one as that's the most reliable one.

          java -jar jenkins-cli.jar -ssh -user ssbarnea -i ~/.ssh/id_rsa -p jenkins.example.com:50000 -logger FINEST help
          Aug 22, 2017 10:24:07 PM hudson.cli.CLI _main
          FINE: using connection mode SSH
          Aug 22, 2017 10:24:08 PM hudson.cli.SSHCLI sshConnection
          WARNING: No header 'X-SSH-Endpoint' returned by Jenkins
          FAIL: 255
          

          This error looks bit confusing to me because I manually specified it to use `-ssh` method, and as far as I know SSH protocol does not involve HTTP headers.

          Sorin Sbarnea added a comment - - edited After applying the workaround from https://issues.jenkins-ci.org/browse/JENKINS-43666?focusedCommentId=311367&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-311367  I was able to make the jar+HTTP approach work. I am still trying to make jar+ssh and ssh approaches to work, preferably the pure ssh one as that's the most reliable one. java -jar jenkins-cli.jar -ssh -user ssbarnea -i ~/.ssh/id_rsa -p jenkins.example.com:50000 -logger FINEST help Aug 22, 2017 10:24:07 PM hudson.cli.CLI _main FINE: using connection mode SSH Aug 22, 2017 10:24:08 PM hudson.cli.SSHCLI sshConnection WARNING: No header 'X-SSH-Endpoint' returned by Jenkins FAIL: 255 This error looks bit confusing to me because I manually specified it to use `-ssh` method, and as far as I know SSH protocol does not involve HTTP headers.

          Jesse Glick added a comment -

          If you use jenkins-cli.jar -ssh, it will first make a simple (RESTish) HTTP connection to Jenkins just to look up the X-SSH-Endpoint header, which gives it a host and port to use to make the SSH connection. If you use /usr/bin/ssh you are on your own to specify these.

          Your error suggests that nginx is dropping this header for some reason, or Jenkins was not serving it to begin with. Check a direct (non-proxied) Jenkins URL for comparison. Anyway that is off topic for this issue; use the users’ list.

          Jesse Glick added a comment - If you use jenkins-cli.jar -ssh , it will first make a simple (RESTish) HTTP connection to Jenkins just to look up the X-SSH-Endpoint header, which gives it a host and port to use to make the SSH connection. If you use /usr/bin/ssh you are on your own to specify these. Your error suggests that nginx is dropping this header for some reason, or Jenkins was not serving it to begin with. Check a direct (non-proxied) Jenkins URL for comparison. Anyway that is off topic for this issue; use the users’ list.

          Jesse Glick added a comment -

          Note new test of HTTP Duplex mode since I have seen some issues with Jetty 9.4.x.

          Jesse Glick added a comment - Note new test of HTTP Duplex mode  since I have seen some issues with Jetty 9.4.x.

          Jesse Glick added a comment -

          https://github.com/jenkinsci/jenkins-test-harness/pull/69 suggests that Jetty’s default configuration may simply be broken on certain machines. Not sure how this would not have been noticed by zillions of other people, though the Jenkins HTTP Duplex trick is pretty strange.

          Jesse Glick added a comment - https://github.com/jenkinsci/jenkins-test-harness/pull/69  suggests that Jetty’s default configuration may simply be broken on certain machines. Not sure how this would not have been noticed by zillions of other people, though the Jenkins HTTP Duplex trick is pretty strange.

          Samuel Lopez added a comment - - edited

          I am seeing this issue as well.

           

          Jenkins Core: 2.60.3

          OS Ubuntu: 16.04

           

           

           

          Nov 22, 2017 2:14:28 PM hudson.init.impl.InstallUncaughtExceptionHandler$1 reportException
          WARNING: null
          java.io.IOException: HTTP full-duplex channel timeout: 1e389d87-5a4b-4342-bb47-8b1cd61ee203
          at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:104)
          at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:171)
          at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124)
          at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69)
          at org.kohsuke.stapler.Function.renderResponse(Function.java:136)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:676)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
          at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374)
          at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845)
          at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
          at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
          at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
          at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:232)
          at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:209)
          at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
          at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:113)
          at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
          at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
          at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
          at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92)
          at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
          at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.

          Samuel Lopez added a comment - - edited I am seeing this issue as well.   Jenkins Core: 2.60.3 OS Ubuntu: 16.04       Nov 22, 2017 2:14:28 PM hudson.init.impl.InstallUncaughtExceptionHandler$1 reportException WARNING: null java.io.IOException: HTTP full-duplex channel timeout: 1e389d87-5a4b-4342-bb47-8b1cd61ee203 at jenkins.util.FullDuplexHttpService.download(FullDuplexHttpService.java:104) at jenkins.util.FullDuplexHttpService$Response.generateResponse(FullDuplexHttpService.java:171) at org.kohsuke.stapler.HttpResponseRenderer$Default.handleHttpResponse(HttpResponseRenderer.java:124) at org.kohsuke.stapler.HttpResponseRenderer$Default.generateResponse(HttpResponseRenderer.java:69) at org.kohsuke.stapler.Function.renderResponse(Function.java:136) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:676) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:374) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:715) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:845) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:232) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:209) at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88) at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:113) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:138) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:92) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.

          Jesse Glick added a comment -

          Superseded by -webSocket added in JEP-222: details

          Jesse Glick added a comment - Superseded by -webSocket added in JEP-222: details

            Unassigned Unassigned
            tstrangio Todd Strangio
            Votes:
            12 Vote for this issue
            Watchers:
            21 Start watching this issue

              Created:
              Updated:
              Resolved: