Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48130

JNLP1-3 Channel Failure: RejectedExecutionException is not handled properly

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • remoting
    • None

      Background: I was analysing JIRA issues related to the NIOHub fatal channel termination causing massive disconnection of agents. It appears that the SingleLaneExecutor is not completely correctly used there...

      TL;DR: A single packet sent to the channel with pending shutdown may cause the termination of all remoting channels in JNLP1, JNLP2, CLI, and CLI2 protocols. JNLP4 does not seem to be affected.

      • When we receive the command in a particular NioTransport, message parts are are being submitted to its SingleLaneExecutor
      • If the executor service rejects the task (e.g. due to the pending shutdown), a runtime RejectedExecutionException is being thrown. E.g. here
      • This exception is not being caught on the NioTransport level and gets proparated to the top level of NioChannelHub#run()
      • NioChannelHub#run() catches the unhandled RuntimeException... and terminates the entire NioHub.

          [JENKINS-48130] JNLP1-3 Channel Failure: RejectedExecutionException is not handled properly

          Oleg Nenashev added a comment -

          I have found at least one support bundle where it happens, so I will treat it as a Major defect

          Oleg Nenashev added a comment - I have found at least one support bundle where it happens, so I will treat it as a Major defect

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/hudson/remoting/AtmostOneThreadExecutor.java
          src/main/java/hudson/remoting/JarCacheSupport.java
          src/main/java/hudson/remoting/SingleLaneExecutorService.java
          src/main/java/org/jenkinsci/remoting/nio/NioChannelHub.java
          src/main/java/org/jenkinsci/remoting/util/ExecutorServiceUtils.java
          src/test/java/hudson/remoting/PipeTest.java
          http://jenkins-ci.org/commit/remoting/eee8030c07f14b0228bc82392851385594efc2ad
          Log:
          JENKINS-48130 - Improve handling and diagnostics of RejectedExecutionException in the code (#156)

          • Introduce the reloable ExecutorServiceUtils#submitAsync() method
          • Rework existing callers of ExecutorService#submit() to ExecutorServiceUtils, process errors
          • Introduce the FatalRejectedExecutionException and proper handling in ExecutorServiceUtils
          • RejectedExecutionException in SingleLaneService should refer the base exec service
          • Polish the PipeTest implementation for a better diagnosability
          • Fix the double submission issue in the SingleLaneExecutor

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/remoting/AtmostOneThreadExecutor.java src/main/java/hudson/remoting/JarCacheSupport.java src/main/java/hudson/remoting/SingleLaneExecutorService.java src/main/java/org/jenkinsci/remoting/nio/NioChannelHub.java src/main/java/org/jenkinsci/remoting/util/ExecutorServiceUtils.java src/test/java/hudson/remoting/PipeTest.java http://jenkins-ci.org/commit/remoting/eee8030c07f14b0228bc82392851385594efc2ad Log: JENKINS-48130 - Improve handling and diagnostics of RejectedExecutionException in the code (#156) Introduce the reloable ExecutorServiceUtils#submitAsync() method Rework existing callers of ExecutorService#submit() to ExecutorServiceUtils, process errors Introduce the FatalRejectedExecutionException and proper handling in ExecutorServiceUtils RejectedExecutionException in SingleLaneService should refer the base exec service Polish the PipeTest implementation for a better diagnosability Fix the double submission issue in the SingleLaneExecutor

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          core/src/main/java/hudson/Launcher.java
          core/src/main/java/hudson/model/Computer.java
          core/src/main/java/hudson/slaves/ChannelPinger.java
          core/src/main/java/hudson/slaves/SlaveComputer.java
          core/src/main/java/jenkins/FilePathFilter.java
          core/src/main/java/jenkins/slaves/StandardOutputSwapper.java
          pom.xml
          test/src/test/java/hudson/bugs/JnlpAccessWithSecuredHudsonTest.java
          test/src/test/java/jenkins/security/Security218CliTest.java
          http://jenkins-ci.org/commit/jenkins/cb3990a4d6094260bea4571e7079fd0e3949047f
          Log:
          Update to Remoting 3.15 and Cleanup issues in Channel#current() usages (#3145)

          Pulls in fixes for: JENKINS-48133, JENKINS-48055, JENKINS-37566, JENKINS-48309, JENKINS-47965, JENKINS-48130, JENKINS-37670, JENKINS-37566, JENKINS-46724

          This change also adds some missing null/closing channel checks in the core.
          In some cases the change prevents spawning threads if the channel is in the invalid state.

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/hudson/Launcher.java core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/slaves/ChannelPinger.java core/src/main/java/hudson/slaves/SlaveComputer.java core/src/main/java/jenkins/FilePathFilter.java core/src/main/java/jenkins/slaves/StandardOutputSwapper.java pom.xml test/src/test/java/hudson/bugs/JnlpAccessWithSecuredHudsonTest.java test/src/test/java/jenkins/security/Security218CliTest.java http://jenkins-ci.org/commit/jenkins/cb3990a4d6094260bea4571e7079fd0e3949047f Log: Update to Remoting 3.15 and Cleanup issues in Channel#current() usages (#3145) Pulls in fixes for: JENKINS-48133 , JENKINS-48055 , JENKINS-37566 , JENKINS-48309 , JENKINS-47965 , JENKINS-48130 , JENKINS-37670 , JENKINS-37566 , JENKINS-46724 This change also adds some missing null/closing channel checks in the core. In some cases the change prevents spawning threads if the channel is in the invalid state.

          Oleg Nenashev added a comment -

          The change has been released in Remoting 3.15 and integrated towards Jenkins 2.98

          Oleg Nenashev added a comment - The change has been released in Remoting 3.15 and integrated towards Jenkins 2.98

            oleg_nenashev Oleg Nenashev
            oleg_nenashev Oleg Nenashev
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: