Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-48130

JNLP1-3 Channel Failure: RejectedExecutionException is not handled properly

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Major Major
    • remoting
    • None

      Background: I was analysing JIRA issues related to the NIOHub fatal channel termination causing massive disconnection of agents. It appears that the SingleLaneExecutor is not completely correctly used there...

      TL;DR: A single packet sent to the channel with pending shutdown may cause the termination of all remoting channels in JNLP1, JNLP2, CLI, and CLI2 protocols. JNLP4 does not seem to be affected.

      • When we receive the command in a particular NioTransport, message parts are are being submitted to its SingleLaneExecutor
      • If the executor service rejects the task (e.g. due to the pending shutdown), a runtime RejectedExecutionException is being thrown. E.g. here
      • This exception is not being caught on the NioTransport level and gets proparated to the top level of NioChannelHub#run()
      • NioChannelHub#run() catches the unhandled RuntimeException... and terminates the entire NioHub.

          [JENKINS-48130] JNLP1-3 Channel Failure: RejectedExecutionException is not handled properly

          Oleg Nenashev created issue -
          Oleg Nenashev made changes -
          Epic Link New: JENKINS-38833 [ 175240 ]

          Oleg Nenashev added a comment -

          I have found at least one support bundle where it happens, so I will treat it as a Major defect

          Oleg Nenashev added a comment - I have found at least one support bundle where it happens, so I will treat it as a Major defect
          Oleg Nenashev made changes -
          Remote Link New: This issue links to "https://github.com/jenkinsci/remoting/pull/156 (Web Link)" [ 18124 ]
          Oleg Nenashev made changes -
          Assignee New: Oleg Nenashev [ oleg_nenashev ]
          Oleg Nenashev made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Oleg Nenashev made changes -
          Status Original: In Progress [ 3 ] New: In Review [ 10005 ]
          Oleg Nenashev made changes -
          Link New: This issue is related to JENKINS-37566 [ JENKINS-37566 ]

          Code changed in jenkins
          User: Oleg Nenashev
          Path:
          src/main/java/hudson/remoting/AtmostOneThreadExecutor.java
          src/main/java/hudson/remoting/JarCacheSupport.java
          src/main/java/hudson/remoting/SingleLaneExecutorService.java
          src/main/java/org/jenkinsci/remoting/nio/NioChannelHub.java
          src/main/java/org/jenkinsci/remoting/util/ExecutorServiceUtils.java
          src/test/java/hudson/remoting/PipeTest.java
          http://jenkins-ci.org/commit/remoting/eee8030c07f14b0228bc82392851385594efc2ad
          Log:
          JENKINS-48130 - Improve handling and diagnostics of RejectedExecutionException in the code (#156)

          • Introduce the reloable ExecutorServiceUtils#submitAsync() method
          • Rework existing callers of ExecutorService#submit() to ExecutorServiceUtils, process errors
          • Introduce the FatalRejectedExecutionException and proper handling in ExecutorServiceUtils
          • RejectedExecutionException in SingleLaneService should refer the base exec service
          • Polish the PipeTest implementation for a better diagnosability
          • Fix the double submission issue in the SingleLaneExecutor

          SCM/JIRA link daemon added a comment - Code changed in jenkins User: Oleg Nenashev Path: src/main/java/hudson/remoting/AtmostOneThreadExecutor.java src/main/java/hudson/remoting/JarCacheSupport.java src/main/java/hudson/remoting/SingleLaneExecutorService.java src/main/java/org/jenkinsci/remoting/nio/NioChannelHub.java src/main/java/org/jenkinsci/remoting/util/ExecutorServiceUtils.java src/test/java/hudson/remoting/PipeTest.java http://jenkins-ci.org/commit/remoting/eee8030c07f14b0228bc82392851385594efc2ad Log: JENKINS-48130 - Improve handling and diagnostics of RejectedExecutionException in the code (#156) Introduce the reloable ExecutorServiceUtils#submitAsync() method Rework existing callers of ExecutorService#submit() to ExecutorServiceUtils, process errors Introduce the FatalRejectedExecutionException and proper handling in ExecutorServiceUtils RejectedExecutionException in SingleLaneService should refer the base exec service Polish the PipeTest implementation for a better diagnosability Fix the double submission issue in the SingleLaneExecutor
          James Dumay made changes -
          Remote Link New: This issue links to "CloudBees Internal OSS-2577 (Web Link)" [ 18246 ]

            oleg_nenashev Oleg Nenashev
            oleg_nenashev Oleg Nenashev
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: