Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-58700

java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent


    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Blocker Blocker
    • core
    • Jenkins version: 2.121.3
      OS: Windows Server 2012 R2 Standard
      Java version 1.8.0_111

      Hi our Jenkins server is facing following issues for around one month,from one per week to currently occur once in 1-2 days.

      The issue is the slaves(~10-20) can be connected to server normally at the beginning, with calling command 'java -jar agent.jar -jnlpurl ......'. And all of them will suddenly disconnected at the same time. But it is confirmed that the network connection of both server and slaves are normal, and 'ping'ing each other are success too.


      Here is the logging.
      Server logging:
      Jul 26, 2019 6:19:17 PM hudson.slaves.ChannelPinger$1 onDead
      INFO: Ping failed. Terminating the channel JNLP4-connect connection from vis_test25_10_1.ahkex.asmpt.com/
      java.util.concurrent.TimeoutException: Ping started at 1564136117631 hasn't completed by 1564136357652
      at hudson.remoting.PingThread.ping(PingThread.java:134)
      at hudson.remoting.PingThread.run(PingThread.java:90)

      Jul 26, 2019 6:19:17 PM jenkins.slaves.DefaultJnlpSlaveReceiver channelClosed
      WARNING: Computer.threadPoolForRemoting 9322 for vis_test25_10_1 terminated
      at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onReadClosed(ChannelApplicationLayer.java:209)
      at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecvClosed(ApplicationLayer.java:222)
      at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecvClosed(ProtocolStack.java:832)
      at org.jenkinsci.remoting.protocol.FilterLayer.onRecvClosed(FilterLayer.java:287)
      at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecvClosed(SSLEngineFilterLayer.java:181)
      at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.switchToNoSecure(SSLEngineFilterLayer.java:283)
      at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processWrite(SSLEngineFilterLayer.java:503)
      at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processQueuedWrites(SSLEngineFilterLayer.java:248)
      at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doSend(SSLEngineFilterLayer.java:200)
      at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.doCloseSend(SSLEngineFilterLayer.java:213)
      at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.doCloseSend(ProtocolStack.java:800)
      at org.jenkinsci.remoting.protocol.ApplicationLayer.doCloseWrite(ApplicationLayer.java:173)
      at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.closeWrite(ChannelApplicationLayer.java:314)
      at hudson.remoting.Channel.close(Channel.java:1450)
      at hudson.remoting.Channel.close(Channel.java:1403)
      at hudson.slaves.SlaveComputer.closeChannel(SlaveComputer.java:799)
      at hudson.slaves.SlaveComputer.access$800(SlaveComputer.java:103)
      at hudson.slaves.SlaveComputer$3.run(SlaveComputer.java:715)
      at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
      at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
      at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      at java.util.concurrent.FutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)
      Slave Logging:
      Jul 26, 2019 6:21:18 PM hudson.remoting.jnlp.Main$CuiListener error
      SEVERE: Failed to connect to ahknts34b:63560
      java.io.IOException: Failed to connect to ahknts34b:63560
      at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:242)
      at hudson.remoting.Engine.connect(Engine.java:686)
      at hudson.remoting.Engine.innerRun(Engine.java:547)
      at hudson.remoting.Engine.run(Engine.java:469)
      Caused by: java.net.ConnectException: Connection refused: connect
      at sun.nio.ch.Net.connect0(Native Method)
      at sun.nio.ch.Net.connect(Unknown Source)
      at sun.nio.ch.Net.connect(Unknown Source)
      at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
      at java.nio.channels.SocketChannel.open(Unknown Source)
      at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:203)
      ... 3 more

       Meanwhile, it is found that all port for listening slaves on server are all status "CLOSE_WAIT".


      FYI, here is the installed plugin list:

      JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1
      Ant Plugin (ant): 1.8
      OWASP Markup Formatter Plugin (antisamy-markup-formatter): 1.5
      Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.5-3.0
      Authentication Tokens API Plugin (authentication-tokens): 1.3
      Autofavorite for Blue Ocean (blueocean-autofavorite): 1.2.4
      Bitbucket Pipeline for Blue Ocean (blueocean-bitbucket-pipeline): 1.16.0
      Common API for Blue Ocean (blueocean-commons): 1.16.0
      Config API for Blue Ocean (blueocean-config): 1.16.0
      Blue Ocean Core JS (blueocean-core-js): 1.16.0
      Dashboard for Blue Ocean (blueocean-dashboard): 1.16.0
      Display URL for Blue Ocean (blueocean-display-url): 2.2.0
      Events API for Blue Ocean (blueocean-events): 1.16.0
      Git Pipeline for Blue Ocean (blueocean-git-pipeline): 1.16.0
      GitHub Pipeline for Blue Ocean (blueocean-github-pipeline): 1.16.0
      i18n for Blue Ocean (blueocean-i18n): 1.16.0
      JIRA Integration for Blue Ocean (blueocean-jira): 1.16.0
      JWT for Blue Ocean (blueocean-jwt): 1.16.0
      Personalization for Blue Ocean (blueocean-personalization): 1.16.0
      Pipeline implementation for Blue Ocean (blueocean-pipeline-api-impl): 1.16.0
      Blue Ocean Pipeline Editor (blueocean-pipeline-editor): 1.16.0
      Pipeline SCM API for Blue Ocean (blueocean-pipeline-scm-api): 1.16.0
      REST Implementation for Blue Ocean (blueocean-rest-impl): 1.16.0
      REST API for Blue Ocean (blueocean-rest): 1.16.0
      Web for Blue Ocean (blueocean-web): 1.16.0
      Blue Ocean (blueocean): 1.16.0
      bouncycastle API Plugin (bouncycastle-api): 2.16.3
      Branch API Plugin (branch-api): 2.3.0
      Build Timeout (build-timeout): 1.19
      user build vars plugin (build-user-vars-plugin): 1.5
      Bitbucket Branch Source Plugin (cloudbees-bitbucket-branch-source): 2.4.4
      Folders Plugin (cloudbees-folder): 6.5.1
      Command Agent Launcher Plugin (command-launcher): 1.2
      Conditional BuildStep (conditional-buildstep): 1.3.6
      Credentials Binding Plugin (credentials-binding): 1.16
      Credentials Plugin (credentials): 2.1.17
      Dashboard View (dashboard-view): 2.10
      Display URL API (display-url-api): 2.3.1
      Docker Commons Plugin (docker-commons): 1.14
      Docker Pipeline (docker-workflow): 1.18
      Durable Task Plugin (durable-task): 1.29
      Email Extension Plugin (email-ext): 2.66
      External Monitor Job Type Plugin (external-monitor-job): 1.7
      Favorite (favorite): 2.3.2
      Git client plugin (git-client): 2.7.2
      GIT server Plugin (git-server): 1.7
      Git plugin (git): 3.9.1
      GitHub API Plugin (github-api): 1.92
      GitHub Branch Source Plugin (github-branch-source): 2.3.6
      GitHub plugin (github): 1.29.2
      Gradle Plugin (gradle): 1.29
      JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 1.1.1
      Handy Uri Templates 2.x API Plugin (handy-uri-templates-2-api): 2.1.6-1.0
      HTML Publisher plugin (htmlpublisher): 1.16
      Jackson 2 API Plugin (jackson2-api): 2.9.8
      Javadoc Plugin (javadoc): 1.4
      JDK Tool Plugin (jdk-tool): 1.1
      Design Language (jenkins-design-language): 1.16.0
      JIRA plugin (jira): 3.0.7
      Join plugin (join): 1.21
      JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1
      jQuery plugin (jquery): 1.12.4-0
      JSch dependency plugin (jsch):
      JUnit Plugin (junit): 1.24
      LDAP Plugin (ldap): 1.20
      Locale plugin (locale): 1.3
      Mailer Plugin (mailer): 1.21
      MapDB API Plugin (mapdb-api):
      Matrix Authorization Strategy Plugin (matrix-auth): 2.3
      Matrix Project Plugin (matrix-project): 1.13
      Maven Integration plugin (maven-plugin): 3.1.2
      Mercurial plugin (mercurial): 2.4
      JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1
      MSBuild Plugin (msbuild): 1.29
      Node and Label parameter plugin (nodelabelparameter): 1.7.2
      PAM Authentication plugin (pam-auth): 1.4
      Parallel Test Executor Plugin (parallel-test-executor): 1.10
      Parameterized Scheduler (parameterized-scheduler): 0.6.2
      Parameterized Trigger plugin (parameterized-trigger): 2.35.2
      Pipeline: Build Step (pipeline-build-step): 2.9
      Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 1.9
      Pipeline: Input Step (pipeline-input-step): 2.10
      Pipeline: Milestone Step (pipeline-milestone-step): 1.3.1
      Pipeline: Model API (pipeline-model-api): 1.3.8
      Pipeline: Declarative Agent API (pipeline-model-declarative-agent): 1.1.1
      Pipeline: Declarative (pipeline-model-definition): 1.3.8
      Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.3.8
      Pipeline: REST API Plugin (pipeline-rest-api): 2.11
      Pipeline: Stage Step (pipeline-stage-step): 2.3
      Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 1.3.8
      Pipeline: Stage View Plugin (pipeline-stage-view): 2.11
      Plain Credentials Plugin (plain-credentials): 1.4
      PowerShell plugin (powershell): 1.3
      Priority Sorter Plugin (PrioritySorter): 3.6.0
      Pub-Sub "light" Bus (pubsub-light): 1.12
      Python Plugin (python): 1.3
      Rebuilder (rebuild): 1.29
      Redmine Metrics Report (redmine-metrics-report): 1.0.1
      Redmine plugin (redmine): 0.21
      Run Condition Plugin (run-condition): 1.0
      SCM API Plugin (scm-api): 2.4.1
      Script Security Plugin (script-security): 1.58
      Server Sent Events (SSE) Gateway Plugin (sse-gateway): 1.17
      SSH Agent Plugin (ssh-agent): 1.15
      SSH Credentials Plugin (ssh-credentials): 1.14
      SSH Slaves plugin (ssh-slaves): 1.26
      Structs Plugin (structs): 1.19
      Subversion Plug-in (subversion): 2.11.1
      Throttle Concurrent Builds Plug-in (throttle-concurrents): 2.0.1
      Timestamper (timestamper): 1.8.10
      Token Macro Plugin (token-macro): 2.5
      Active Choices Plug-in (uno-choice): 2.1
      Variant Plugin (variant): 1.1
      WebSphere Deployer Plugin (websphere-deployer): 1.6.1
      Windows Exe Runner Plugin (windows-exe-runner): 1.2
      Windows Slaves Plugin (windows-slaves): 1.3.1
      Pipeline (workflow-aggregator): 2.5
      Pipeline: API (workflow-api): 2.33
      Pipeline: Basic Steps (workflow-basic-steps): 2.15
      Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.13
      Pipeline: Groovy (workflow-cps): 2.68
      Pipeline: Nodes and Processes (workflow-durable-task-step): 2.28
      Pipeline: Job (workflow-job): 2.32
      Pipeline: Multibranch (workflow-multibranch): 2.21
      Pipeline: SCM Step (workflow-scm-step): 2.7
      Pipeline: Step API (workflow-step-api): 2.19
      Pipeline: Supporting APIs (workflow-support): 3.3
      IncrediBuild Plugin (IncrediBuild): 1.0.0

            Unassigned Unassigned
            cwchu Chu CW
            2 Vote for this issue
            2 Start watching this issue