Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48130

JNLP1-3 Channel Failure: RejectedExecutionException is not handled properly

    XMLWordPrintable

Details

    • Bug
    • Status: Resolved (View Workflow)
    • Major
    • Resolution: Fixed
    • remoting
    • None

    Description

      Background: I was analysing JIRA issues related to the NIOHub fatal channel termination causing massive disconnection of agents. It appears that the SingleLaneExecutor is not completely correctly used there...

      TL;DR: A single packet sent to the channel with pending shutdown may cause the termination of all remoting channels in JNLP1, JNLP2, CLI, and CLI2 protocols. JNLP4 does not seem to be affected.

      • When we receive the command in a particular NioTransport, message parts are are being submitted to its SingleLaneExecutor
      • If the executor service rejects the task (e.g. due to the pending shutdown), a runtime RejectedExecutionException is being thrown. E.g. here
      • This exception is not being caught on the NioTransport level and gets proparated to the top level of NioChannelHub#run()
      • NioChannelHub#run() catches the unhandled RuntimeException... and terminates the entire NioHub.

      Attachments

        Issue Links

          is related to

          Task - A task that needs to be done. JENKINS-37566 FindBugs-clean remoting layer

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link CloudBees Internal OSS-2577

          Web Link https://github.com/jenkinsci/remoting/pull/156

          Activity

            Ascending order - Click to sort in descending order
            oleg_nenashev Oleg Nenashev added a comment -

            I have found at least one support bundle where it happens, so I will treat it as a Major defect

            oleg_nenashev Oleg Nenashev added a comment - I have found at least one support bundle where it happens, so I will treat it as a Major defect
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            src/main/java/hudson/remoting/AtmostOneThreadExecutor.java
            src/main/java/hudson/remoting/JarCacheSupport.java
            src/main/java/hudson/remoting/SingleLaneExecutorService.java
            src/main/java/org/jenkinsci/remoting/nio/NioChannelHub.java
            src/main/java/org/jenkinsci/remoting/util/ExecutorServiceUtils.java
            src/test/java/hudson/remoting/PipeTest.java
            http://jenkins-ci.org/commit/remoting/eee8030c07f14b0228bc82392851385594efc2ad
            Log:
            JENKINS-48130 - Improve handling and diagnostics of RejectedExecutionException in the code (#156)

            • Introduce the reloable ExecutorServiceUtils#submitAsync() method
            • Rework existing callers of ExecutorService#submit() to ExecutorServiceUtils, process errors
            • Introduce the FatalRejectedExecutionException and proper handling in ExecutorServiceUtils
            • RejectedExecutionException in SingleLaneService should refer the base exec service
            • Polish the PipeTest implementation for a better diagnosability
            • Fix the double submission issue in the SingleLaneExecutor
            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/remoting/AtmostOneThreadExecutor.java src/main/java/hudson/remoting/JarCacheSupport.java src/main/java/hudson/remoting/SingleLaneExecutorService.java src/main/java/org/jenkinsci/remoting/nio/NioChannelHub.java src/main/java/org/jenkinsci/remoting/util/ExecutorServiceUtils.java src/test/java/hudson/remoting/PipeTest.java http://jenkins-ci.org/commit/remoting/eee8030c07f14b0228bc82392851385594efc2ad Log: JENKINS-48130 - Improve handling and diagnostics of RejectedExecutionException in the code (#156) Introduce the reloable ExecutorServiceUtils#submitAsync() method Rework existing callers of ExecutorService#submit() to ExecutorServiceUtils, process errors Introduce the FatalRejectedExecutionException and proper handling in ExecutorServiceUtils RejectedExecutionException in SingleLaneService should refer the base exec service Polish the PipeTest implementation for a better diagnosability Fix the double submission issue in the SingleLaneExecutor
            scm_issue_link SCM/JIRA link daemon added a comment -

            Code changed in jenkins
            User: Oleg Nenashev
            Path:
            core/src/main/java/hudson/Launcher.java
            core/src/main/java/hudson/model/Computer.java
            core/src/main/java/hudson/slaves/ChannelPinger.java
            core/src/main/java/hudson/slaves/SlaveComputer.java
            core/src/main/java/jenkins/FilePathFilter.java
            core/src/main/java/jenkins/slaves/StandardOutputSwapper.java
            pom.xml
            test/src/test/java/hudson/bugs/JnlpAccessWithSecuredHudsonTest.java
            test/src/test/java/jenkins/security/Security218CliTest.java
            http://jenkins-ci.org/commit/jenkins/cb3990a4d6094260bea4571e7079fd0e3949047f
            Log:
            Update to Remoting 3.15 and Cleanup issues in Channel#current() usages (#3145)

            Pulls in fixes for: JENKINS-48133, JENKINS-48055, JENKINS-37566, JENKINS-48309, JENKINS-47965, JENKINS-48130, JENKINS-37670, JENKINS-37566, JENKINS-46724

            This change also adds some missing null/closing channel checks in the core.
            In some cases the change prevents spawning threads if the channel is in the invalid state.

            scm_issue_link SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: core/src/main/java/hudson/Launcher.java core/src/main/java/hudson/model/Computer.java core/src/main/java/hudson/slaves/ChannelPinger.java core/src/main/java/hudson/slaves/SlaveComputer.java core/src/main/java/jenkins/FilePathFilter.java core/src/main/java/jenkins/slaves/StandardOutputSwapper.java pom.xml test/src/test/java/hudson/bugs/JnlpAccessWithSecuredHudsonTest.java test/src/test/java/jenkins/security/Security218CliTest.java http://jenkins-ci.org/commit/jenkins/cb3990a4d6094260bea4571e7079fd0e3949047f Log: Update to Remoting 3.15 and Cleanup issues in Channel#current() usages (#3145) Pulls in fixes for: JENKINS-48133 , JENKINS-48055 , JENKINS-37566 , JENKINS-48309 , JENKINS-47965 , JENKINS-48130 , JENKINS-37670 , JENKINS-37566 , JENKINS-46724 This change also adds some missing null/closing channel checks in the core. In some cases the change prevents spawning threads if the channel is in the invalid state.
            oleg_nenashev Oleg Nenashev added a comment -

            The change has been released in Remoting 3.15 and integrated towards Jenkins 2.98

            oleg_nenashev Oleg Nenashev added a comment - The change has been released in Remoting 3.15 and integrated towards Jenkins 2.98

            People

              oleg_nenashev Oleg Nenashev
              oleg_nenashev Oleg Nenashev
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: