Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-69509

Unstable websocket connection (java.nio.channels.ClosedChannelException)

      xWe have long running tests (1 hour duration) that we manage through Jenkins. Since we upgrade from 2.364 to 2.365 we are getting a unstable websocket connection between the Jenkins agent and controller sometimes a run will be successful but the past days a passed run without this exception is hard to achieve.

      Resulting in the following exception:

      ....FATAL: command execution failed
      java.nio.channels.ClosedChannelException
          at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:153)
          at jenkins.websocket.WebSockets$1.onWebSocketClose(WebSockets.java:80)
          at jenkins.websocket.Jetty10Provider$2.onWebSocketClose(Jetty10Provider.java:146)
          at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.notifyOnClose(JettyWebSocketFrameHandler.java:308)
          at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onClosed(JettyWebSocketFrameHandler.java:292)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$closeConnection$0(WebSocketCoreSession.java:272)
          at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1445)
          at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1482)
          at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker$1.handle(AbstractHandshaker.java:212)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$closeConnection$1(WebSocketCoreSession.java:272)
          at org.eclipse.jetty.util.Callback$4.completed(Callback.java:184)
          at org.eclipse.jetty.util.Callback$Completing.succeeded(Callback.java:344)
          at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onError(JettyWebSocketFrameHandler.java:268)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$closeConnection$2(WebSocketCoreSession.java:284)
          at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1463)
          at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1482)
          at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker$1.handle(AbstractHandshaker.java:212)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.closeConnection(WebSocketCoreSession.java:284)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$sendFrame$7(WebSocketCoreSession.java:519)
          at org.eclipse.jetty.util.Callback$3.succeeded(Callback.java:155)
          at org.eclipse.jetty.websocket.core.internal.TransformingFlusher.notifyCallbackSuccess(TransformingFlusher.java:197)
          at org.eclipse.jetty.websocket.core.internal.TransformingFlusher$Flusher.process(TransformingFlusher.java:154)
          at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:232)
          at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:214)
          at org.eclipse.jetty.websocket.core.internal.TransformingFlusher.sendFrame(TransformingFlusher.java:77)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.sendFrame(WebSocketCoreSession.java:522)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.close(WebSocketCoreSession.java:239)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.processHandlerError(WebSocketCoreSession.java:371)
          at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.onIdleExpired(WebSocketConnection.java:233)
          at org.eclipse.jetty.io.ssl.SslConnection.onIdleExpired(SslConnection.java:360)
          at org.eclipse.jetty.io.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:407)
          at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
          at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:108)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:829)
      Caused: java.io.IOException: Backing channel 'PerformanceSERVER' is disconnected.
          at hudson.remoting.RemoteInvocationHandler.channelOrFail(RemoteInvocationHandler.java:215)
          at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:285)
          at com.sun.proxy.$Proxy76.isAlive(Unknown Source)
          at hudson.Launcher$RemoteLauncher$ProcImpl.isAlive(Launcher.java:1215)
          at hudson.Launcher$RemoteLauncher$ProcImpl.join(Launcher.java:1207)
          at hudson.tasks.CommandInterpreter.join(CommandInterpreter.java:195)
          at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:145)
          at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
          at hudson.plugins.powershell.PowerShell.perform(PowerShell.java:48)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
          at hudson.model.Build$BuildExecution.build(Build.java:199)
          at hudson.model.Build$BuildExecution.doRun(Build.java:164)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:526)
          at hudson.model.Run.execute(Run.java:1899)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
          at hudson.model.ResourceController.execute(ResourceController.java:107)
          at hudson.model.Executor.run(Executor.java:449)
      FATAL: Unable to delete script file C:\Users\account.adm\AppData\Local\Temp\jenkins9449560261283910130.ps1
      java.nio.channels.ClosedChannelException
          at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:153)
          at jenkins.websocket.WebSockets$1.onWebSocketClose(WebSockets.java:80)
          at jenkins.websocket.Jetty10Provider$2.onWebSocketClose(Jetty10Provider.java:146)
          at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.notifyOnClose(JettyWebSocketFrameHandler.java:308)
          at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onClosed(JettyWebSocketFrameHandler.java:292)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$closeConnection$0(WebSocketCoreSession.java:272)
          at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1445)
          at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1482)
          at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker$1.handle(AbstractHandshaker.java:212)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$closeConnection$1(WebSocketCoreSession.java:272)
          at org.eclipse.jetty.util.Callback$4.completed(Callback.java:184)
          at org.eclipse.jetty.util.Callback$Completing.succeeded(Callback.java:344)
          at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onError(JettyWebSocketFrameHandler.java:268)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$closeConnection$2(WebSocketCoreSession.java:284)
          at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1463)
          at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1482)
          at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker$1.handle(AbstractHandshaker.java:212)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.closeConnection(WebSocketCoreSession.java:284)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$sendFrame$7(WebSocketCoreSession.java:519)
          at org.eclipse.jetty.util.Callback$3.succeeded(Callback.java:155)
          at org.eclipse.jetty.websocket.core.internal.TransformingFlusher.notifyCallbackSuccess(TransformingFlusher.java:197)
          at org.eclipse.jetty.websocket.core.internal.TransformingFlusher$Flusher.process(TransformingFlusher.java:154)
          at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:232)
          at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:214)
          at org.eclipse.jetty.websocket.core.internal.TransformingFlusher.sendFrame(TransformingFlusher.java:77)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.sendFrame(WebSocketCoreSession.java:522)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.close(WebSocketCoreSession.java:239)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.processHandlerError(WebSocketCoreSession.java:371)
          at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.onIdleExpired(WebSocketConnection.java:233)
          at org.eclipse.jetty.io.ssl.SslConnection.onIdleExpired(SslConnection.java:360)
          at org.eclipse.jetty.io.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:407)
          at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
          at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:108)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:829)
      Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@3c2ca039:PerformanceSERVER": Remote call on PerformanceSERVER failed. The channel is closing down or has closed down
          at hudson.remoting.Channel.call(Channel.java:993)
          at hudson.FilePath.act(FilePath.java:1186)
          at hudson.FilePath.act(FilePath.java:1175)
          at hudson.FilePath.delete(FilePath.java:1722)
          at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:163)
          at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:92)
          at hudson.plugins.powershell.PowerShell.perform(PowerShell.java:48)
          at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
          at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:818)
          at hudson.model.Build$BuildExecution.build(Build.java:199)
          at hudson.model.Build$BuildExecution.doRun(Build.java:164)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:526)
          at hudson.model.Run.execute(Run.java:1899)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
          at hudson.model.ResourceController.execute(ResourceController.java:107)
          at hudson.model.Executor.run(Executor.java:449)
      Build step 'PowerShell' marked build as failure
      FATAL: Channel "hudson.remoting.Channel@3c2ca039:PerformanceSERVER": Remote call on PerformanceSERVER failed. The channel is closing down or has closed down
      java.nio.channels.ClosedChannelException
          at jenkins.agents.WebSocketAgents$Session.closed(WebSocketAgents.java:153)
          at jenkins.websocket.WebSockets$1.onWebSocketClose(WebSockets.java:80)
          at jenkins.websocket.Jetty10Provider$2.onWebSocketClose(Jetty10Provider.java:146)
          at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.notifyOnClose(JettyWebSocketFrameHandler.java:308)
          at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onClosed(JettyWebSocketFrameHandler.java:292)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$closeConnection$0(WebSocketCoreSession.java:272)
          at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1445)
          at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1482)
          at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker$1.handle(AbstractHandshaker.java:212)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$closeConnection$1(WebSocketCoreSession.java:272)
          at org.eclipse.jetty.util.Callback$4.completed(Callback.java:184)
          at org.eclipse.jetty.util.Callback$Completing.succeeded(Callback.java:344)
          at org.eclipse.jetty.websocket.common.JettyWebSocketFrameHandler.onError(JettyWebSocketFrameHandler.java:268)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$closeConnection$2(WebSocketCoreSession.java:284)
          at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1463)
          at org.eclipse.jetty.server.handler.ContextHandler.handle(ContextHandler.java:1482)
          at org.eclipse.jetty.websocket.core.server.internal.AbstractHandshaker$1.handle(AbstractHandshaker.java:212)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.closeConnection(WebSocketCoreSession.java:284)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.lambda$sendFrame$7(WebSocketCoreSession.java:519)
          at org.eclipse.jetty.util.Callback$3.succeeded(Callback.java:155)
          at org.eclipse.jetty.websocket.core.internal.TransformingFlusher.notifyCallbackSuccess(TransformingFlusher.java:197)
          at org.eclipse.jetty.websocket.core.internal.TransformingFlusher$Flusher.process(TransformingFlusher.java:154)
          at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:232)
          at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:214)
          at org.eclipse.jetty.websocket.core.internal.TransformingFlusher.sendFrame(TransformingFlusher.java:77)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.sendFrame(WebSocketCoreSession.java:522)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.close(WebSocketCoreSession.java:239)
          at org.eclipse.jetty.websocket.core.internal.WebSocketCoreSession.processHandlerError(WebSocketCoreSession.java:371)
          at org.eclipse.jetty.websocket.core.internal.WebSocketConnection.onIdleExpired(WebSocketConnection.java:233)
          at org.eclipse.jetty.io.ssl.SslConnection.onIdleExpired(SslConnection.java:360)
          at org.eclipse.jetty.io.AbstractEndPoint.onIdleExpired(AbstractEndPoint.java:407)
          at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
          at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:108)
          at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
          at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
          at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
          at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
          at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
          at java.base/java.lang.Thread.run(Thread.java:829)
      Caused: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@3c2ca039:PerformanceSERVER": Remote call on PerformanceSERVER failed. The channel is closing down or has closed down
          at hudson.remoting.Channel.call(Channel.java:993)
          at hudson.Launcher$RemoteLauncher.kill(Launcher.java:1150)
          at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:538)
          at hudson.model.Run.execute(Run.java:1899)
          at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
          at hudson.model.ResourceController.execute(ResourceController.java:107)
          at hudson.model.Executor.run(Executor.java:449)
      Finished: FAILURE

      I have included the logs of the agent in question as an attachment to this issue.

      These are the details of the faulty Jenkins node:

       

      Naam  â†“ Waarde
      awt.toolkit sun.awt.windows.WToolkit
      file.encoding Cp1252
      file.separator |
      java.awt.graphicsenv sun.awt.Win32GraphicsEnvironment
      java.awt.printerjob sun.awt.windows.WPrinterJob
      java.class.path c:\Jenkinsnodeservice\slave.jar
      java.class.version 55.0
      java.home C:\Program Files\Amazon Corretto\jdk11.0.15_9
      java.io.tmpdir C:\Users\el80135.adm\AppData\Local\Temp|
      java.library.path C:\Program Files\Amazon Corretto\jdk11.0.15_9\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Amazon Corretto\jdk11.0.15_9\bin;D:\oracle\client64\19\client\bin;C:\Program Files\Python310\Scripts\;C:\Program Files\Python310\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Users\username\AppData\Local\Microsoft\WindowsApps;.
      java.runtime.name OpenJDK Runtime Environment
      java.runtime.version 11.0.15+9-LTS
      java.specification.name Java Platform API Specification
      java.specification.vendor Oracle Corporation
      java.specification.version 11
      java.vendor Amazon.com Inc.
      java.vendor.url https://aws.amazon.com/corretto/
      java.vendor.url.bug https://github.com/corretto/corretto-11/issues/
      java.vendor.version Corretto-11.0.15.9.1
      java.version 11.0.15
      java.version.date 2022-04-19
      java.vm.compressedOopsMode 32-bit
      java.vm.info mixed mode
      java.vm.name OpenJDK 64-Bit Server VM
      java.vm.specification.name Java Virtual Machine Specification
      java.vm.specification.vendor Oracle Corporation
      java.vm.specification.version 11
      java.vm.vendor Amazon.com Inc.
      java.vm.version 11.0.15+9-LTS
      jdk.debug release
      jna.loaded true
      jnidispatch.path C:\Users\username\AppData\Local\Temp\jna-813739984\jna10440559483708474225.dll
      line.separator  
      os.arch amd64
      os.name Windows Server 2019
      os.version 10.0
      path.separator ;
      sun.arch.data.model 64
      sun.boot.library.path C:\Program Files\Amazon Corretto\jdk11.0.15_9\bin
      sun.cpu.endian little
      sun.cpu.isalist amd64
      sun.desktop windows
      sun.io.unicode.encoding UnicodeLittle
      sun.java.command c:\Jenkinsnodeservice\slave.jar -jnlpUrl https://jenkins:8080/computer/PerformanceSERVER_NAME/jenkins-agent.jnlp -secret "secret" -workDir C:\jenkins
      sun.java.launcher SUN_STANDARD
      sun.jnu.encoding Cp1252
      sun.management.compiler HotSpot 64-Bit Tiered Compilers
      sun.os.patch.level  
      user.country US
      user.dir c:\Jenkinsnodeservice
      user.home C:\Users\username
      user.language en
      user.name username
      user.script  
      user.timezone Europe/Berlin
      user.variant

      Details regarding the Jenkins controller:

      Naam  â†“ Waarde
      awt.toolkit sun.awt.windows.WToolkit
      executable-war C:\Jenkins\Jenkins.war
      file.encoding Cp1252
      file.separator |
      hudson.lifecycle hudson.lifecycle.WindowsServiceLifecycle
      java.awt.graphicsenv sun.awt.Win32GraphicsEnvironment
      java.awt.headless true
      java.awt.printerjob sun.awt.windows.WPrinterJob
      java.class.path C:\Jenkins\jenkins.war
      java.class.version 55.0
      java.home C:\Program Files\Amazon Corretto\jdk11.0.15_9
      java.io.tmpdir C:\WINDOWS\TEMP|
      java.library.path C:\Program Files\Amazon Corretto\jdk11.0.15_9\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Amazon Corretto\jdk11.0.15_9\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\WINDOWS\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps;.
      java.runtime.name OpenJDK Runtime Environment
      java.runtime.version 11.0.15+9-LTS
      java.specification.name Java Platform API Specification
      java.specification.vendor Oracle Corporation
      java.specification.version 11
      java.vendor Amazon.com Inc.
      java.vendor.url https://aws.amazon.com/corretto/
      java.vendor.url.bug https://github.com/corretto/corretto-11/issues/
      java.vendor.version Corretto-11.0.15.9.1
      java.version 11.0.15
      java.version.date 2022-04-19
      java.vm.compressedOopsMode 32-bit
      java.vm.info mixed mode
      java.vm.name OpenJDK 64-Bit Server VM
      java.vm.specification.name Java Virtual Machine Specification
      java.vm.specification.vendor Oracle Corporation
      java.vm.specification.version 11
      java.vm.vendor Amazon.com Inc.
      java.vm.version 11.0.15+9-LTS
      jdk.debug release
      jetty.git.hash
      • (removed it don't know if it is confidential)
      jna.loaded true
      jnidispatch.path C:\WINDOWS\TEMP\jna-756962248\jna18350657977015144084.dll
      line.separator  
      mail.smtp.sendpartial true
      mail.smtps.sendpartial true
      octaneAllowedStorage C:\Jenkins\userContent|
      os.arch amd64
      os.name Windows Server 2019
      os.version 10.0
      path.separator ;
      sun.arch.data.model 64
      sun.awt.enableExtraMouseButtons true
      sun.boot.library.path C:\Program Files\Amazon Corretto\jdk11.0.15_9\bin
      sun.cpu.endian little
      sun.cpu.isalist amd64
      sun.desktop windows
      sun.io.unicode.encoding UnicodeLittle
      sun.java.command C:\Jenkins\jenkins.war {}httpPort=1 httpsPort=8443 httpsKeyStore=C:\Jenkins\secrets\server.keystore -httpsKeyStorePassword=changeit
      sun.java.launcher SUN_STANDARD
      sun.jnu.encoding Cp1252
      sun.management.compiler HotSpot 64-Bit Tiered Compilers
      sun.os.patch.level  
      user.country US
      user.dir C:\Jenkins
      user.home C:\WINDOWS\system32\config\systemprofile
      user.language en
      user.name SystemUser$
      user.script  
      user.timezone Europe/Berlin
      user.variant

      For a bit more context the output to console of the command we are executing:

      Started by user Hendricks, JGH (Joey)
      Running as SYSTEM
      Building remotely on PerformanceSERVER_NAME (Performance) in workspace C:\jenkins\workspace\generieke-acceptatie-performance-testen\1. latest-stable-build-direct
      [1. latest-stable-build-acc90-gpsv-direct] $ powershell.exe -NonInteractive -ExecutionPolicy Bypass -File C:\Users\username\AppData\Local\Temp\jenkins9449560261283910130.ps1
      OpenJDK 64-Bit Server VM warning: Option MaxRAMFraction was deprecated in version 10.0 and will likely be removed in a future release.
      WebServices.nlp file has been found in C:\Users\username\AppData\Local\Temp\NeoloadProjects\WebServices.
      Logging to C:\Users\username\AppData\Roaming\Neotys\NeoLoad\v7.10\logs, as user: username
      Found a valid NTS configuration : URL=[http://licenseserver:8080//] and USER=[service account]
      Leasing license ---...
      Leased license ---== with 150 web and 0 SAP VUs 120 minutes(s).
      Loading project: C:\Users\username\AppData\Local\Temp\NeoloadProjects\WebServices\WebServices.nlp
      Project GPS_WebServices loaded
      Found a valid NTS configuration : URL=[http://licenseserver:8080//] and USER=[service account]
      Launching scenario: WebServices_PiekLoad_60min (01h:02m:00s)
      Initializing...
      [LoadGenerator] OpenJDK 64-Bit Server VM warning: Option MaxRAMFraction was deprecated in version 10.0 and will likely be removed in a future release.
      
      Initializing Monitors...
      Pre-Monitoring...
      Running

      Thanks in advanced this issues has been a headache.

        1. controller_log.txt
          418 kB
        2. jenkins.err.log
          1.26 MB
        3. jenkins.err-1.log
          1.26 MB
        4. jenkins.err-2.log
          1.26 MB
        5. jenkins-master.log
          5 kB
        6. jenkins-slave.err.log
          436 kB
        7. jenkins-slave.wrapper.log
          382 kB
        8. remoting.log.0
          604 kB
        9. remoting.log-1.0
          604 kB
        10. remoting.log-2.0
          604 kB
        11. slave.log.1
          7 kB
        12. slave.log-1.1
          7 kB

          [JENKINS-69509] Unstable websocket connection (java.nio.channels.ClosedChannelException)

          Hierony Manurung added a comment - - edited

          This is the log that I can get from my master (I run the Jenkins master on docker container), I'm afraid it didn't show chain of events that leads up to the failure mode though

          jenkins-master.log

          Hierony Manurung added a comment - - edited This is the log that I can get from my master (I run the Jenkins master on docker container), I'm afraid it didn't show chain of events that leads up to the failure mode though jenkins-master.log

          Bryce Lovell added a comment -

          remoting.log.0jenkins.err.logslave.log.1

          basil hey, trying to give you the error logs you asked for, hope this helps.

          In jenkins.err.log, removed the timeout setting and restarted the service around line 13054.

          Kicked off a new build, you can see that the timeout error happened a few minutes thereafter at the bottom of the log. Happens randomly and in different spots when building.

          Not sure if related, but in the log, while Jenkins was idle, there are a ton of web socket exceptions being thrown with idle timeouts.

          Also included the remoting log on the agent and the slave log from the controller. Not sure if there is any useful information there. The slave log has a little more in the stack trace then what is show in the jenkins error log.

          Bryce Lovell added a comment - remoting.log.0 jenkins.err.log slave.log.1 basil hey, trying to give you the error logs you asked for, hope this helps. In jenkins.err.log, removed the timeout setting and restarted the service around line 13054. Kicked off a new build, you can see that the timeout error happened a few minutes thereafter at the bottom of the log. Happens randomly and in different spots when building. Not sure if related, but in the log, while Jenkins was idle, there are a ton of web socket exceptions being thrown with idle timeouts. Also included the remoting log on the agent and the slave log from the controller. Not sure if there is any useful information there. The slave log has a little more in the stack trace then what is show in the jenkins error log.

          Basil Crow added a comment - - edited

          Thanks brycelovell, based on those logs I can see that the ping thread isn't dying, so the fix in 2.373 at least does not seem to hurt, and is likely helping. The WebSocketTimeout exception you're seeing, in the absence of another exception from jenkins.websocket.Jetty10Provider#sendPing, seems like a legitimate failure to receive an acknowledgement from the ping request within the allotted timeout period. Could you try running with --httpKeepAliveTimeout=30000 (or --httpsKeepAliveTimeout=30000), which is the Jetty (but not Winstone) default? If things are more stable for you with that value, I might consider changing the Winstone default to match the upstream Jetty default.

          Basil Crow added a comment - - edited Thanks brycelovell , based on those logs I can see that the ping thread isn't dying, so the fix in 2.373 at least does not seem to hurt, and is likely helping. The WebSocketTimeout exception you're seeing, in the absence of another exception from jenkins.websocket.Jetty10Provider#sendPing , seems like a legitimate failure to receive an acknowledgement from the ping request within the allotted timeout period. Could you try running with --httpKeepAliveTimeout=30000 (or --httpsKeepAliveTimeout=30000 ), which is the Jetty (but not Winstone) default? If things are more stable for you with that value, I might consider changing the Winstone default to match the upstream Jetty default.

          Bryce Lovell added a comment -

          basil, updated the httpKeepAliveTimeout (http in my instance) to 30000, restarted, and have built 6 times consecutively. Haven't seen the error yet, so far so good.

          Bryce Lovell added a comment - basil , updated the httpKeepAliveTimeout (http in my instance) to 30000, restarted, and have built 6 times consecutively. Haven't seen the error yet, so far so good.

          Basil Crow added a comment -

          Jenkins 2.375 has been released with both jenkinsci/jenkins#7195 and jenkinsci/winstone#296 thereby fixing all known WebSocket regressions relating to the upgrade of Jetty from version 9 to version 10. I am closing this ticket; if you continue to encounter issues, please open a separate ticket with log files from both the controller (both general and agent-specific controller-side logs) and agent side (ideally a full support-core bundle) from several minutes before the disconnection event occurs all the way through to the disconnection event.

          Basil Crow added a comment - Jenkins 2.375 has been released with both jenkinsci/jenkins#7195 and jenkinsci/winstone#296 thereby fixing all known WebSocket regressions relating to the upgrade of Jetty from version 9 to version 10. I am closing this ticket; if you continue to encounter issues, please open a separate ticket with log files from both the controller (both general and agent-specific controller-side logs) and agent side (ideally a full support-core bundle) from several minutes before the disconnection event occurs all the way through to the disconnection event.

          George Yu added a comment -

          Tried 2.375 and got the problem in less than 2 hours, org.eclipse.jetty.websocket.core.exception.WebSocketTimeoutException: Connection Idle Timeout. We have jobs that run over 15 hours.  Reverted all the way back to Jenkins 2.346.3 LTS as a workaround

          George Yu added a comment - Tried 2.375 and got the problem in less than 2 hours, org.eclipse.jetty.websocket.core.exception.WebSocketTimeoutException: Connection Idle Timeout. We have jobs that run over 15 hours.  Reverted all the way back to Jenkins 2.346.3 LTS as a workaround

          Basil Crow added a comment -

          Basil Crow added a comment - gyu Please read JENKINS-69509 (comment) .

          George Yu added a comment -

          Created a separate ticket, JENKINS-69955

          George Yu added a comment - Created a separate ticket,  JENKINS-69955

          basil (I know this ticket is resolved I do however have one quick and on topic question.) Some colleagues of mine are also still reporting this error they want to increase the httpKeepAliveTimeout value. They cannot implement my workaround, so they are stuck. I am Googling to see how to do this, but I cannot find directly how to do this could elaborate this please or provide some reference material? As an update to all of you my initial fix to communicate a "." every second to the terminal still holds strong no more errors have been encounter since then if you have the ability to manipulate the logging in the terminal then this could serve as a temporary fix. Basil in advance thank you for your help, it has been priceless!

          joey hendricks added a comment - basil (I know this ticket is resolved I do however have one quick and on topic question.) Some colleagues of mine are also still reporting this error they want to increase the httpKeepAliveTimeout value. They cannot implement my workaround, so they are stuck. I am Googling to see how to do this, but I cannot find directly how to do this could elaborate this please or provide some reference material? As an update to all of you my initial fix to communicate a "." every second to the terminal still holds strong no more errors have been encounter since then if you have the ability to manipulate the logging in the terminal then this could serve as a temporary fix. Basil in advance thank you for your help, it has been priceless!

          Basil Crow added a comment -

          I will not answer any further questions in this ticket.

          Basil Crow added a comment - I will not answer any further questions in this ticket.

            basil Basil Crow
            ninjaman1159 joey hendricks
            Votes:
            6 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved: