• Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • core
    • Jenkins 2.238

      After launching an agent it keeps hanging with the following text :

       

      [05/06/20 20:34:24] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.3 This is a Unix agent

       

      This happens since a couple of weeks

       

      jstack reports a deadlock in attachment :

       

      Found one Java-level deadlock:
      =============================
      "pool-1-thread-4 for channel id=17236":
      waiting to lock monitor 0x00007fbd8c015bc8 (object 0x000000076e658428, a hudson.slaves.SlaveComputer$SlaveInitializer$1),
      which is held by "Channel reader thread: channel"
      "Channel reader thread: channel":
      waiting to lock monitor 0x00007fbd9401aed8 (object 0x000000076ded1cc8, a hudson.remoting.RemoteClassLoader),
      which is held by "pool-1-thread-4 for channel id=17236"

       

      On april 3 2020 in commit https://github.com/jenkinsci/jenkins/commit/3262d8b7cfbdd1a2e026ad34644e0f9607f5724d  

      a new synchronized  method was added that is partly the cause of this deadlock .See line 1026 in main/java/hudson/slaves/SlaveComputer.java 

       

       

       

          [JENKINS-62181] Remoting deadlock when launching remote agent

          Bart Heupers created issue -
          Bart Heupers made changes -
          Description Original: After launching an agent it keeps hangin with the follwing text :

           

          [05/06/20 20:34:24] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.3 This is a Unix agent

           

          jstack reports a deadlock in attachment :

           

          Found one Java-level deadlock:
          =============================
          "pool-1-thread-4 for channel id=17236":
           waiting to lock monitor 0x00007fbd8c015bc8 (object 0x000000076e658428, a hudson.slaves.SlaveComputer$SlaveInitializer$1),
           which is held by "Channel reader thread: channel"
          "Channel reader thread: channel":
           waiting to lock monitor 0x00007fbd9401aed8 (object 0x000000076ded1cc8, a hudson.remoting.RemoteClassLoader),
           which is held by "pool-1-thread-4 for channel id=17236"

           

          On april 3 2020 in commit [https://github.com/jenkinsci/jenkins/commit/3262d8b7cfbdd1a2e026ad34644e0f9607f5724d]  

          a new synchronized  method was added that is partly the cause of this deadlock .See line 1026 in main/java/hudson/slaves/SlaveComputer.java 

           

           

           
          New: After launching an agent it keeps hanging with the following text :

           

          [05/06/20 20:34:24] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.3 This is a Unix agent

           

          jstack reports a deadlock in attachment :

           

          Found one Java-level deadlock:
           =============================
           "pool-1-thread-4 for channel id=17236":
           waiting to lock monitor 0x00007fbd8c015bc8 (object 0x000000076e658428, a hudson.slaves.SlaveComputer$SlaveInitializer$1),
           which is held by "Channel reader thread: channel"
           "Channel reader thread: channel":
           waiting to lock monitor 0x00007fbd9401aed8 (object 0x000000076ded1cc8, a hudson.remoting.RemoteClassLoader),
           which is held by "pool-1-thread-4 for channel id=17236"

           

          On april 3 2020 in commit [https://github.com/jenkinsci/jenkins/commit/3262d8b7cfbdd1a2e026ad34644e0f9607f5724d]  

          a new synchronized  method was added that is partly the cause of this deadlock .See line 1026 in main/java/hudson/slaves/SlaveComputer.java 

           

           

           
          Bart Heupers made changes -
          Description Original: After launching an agent it keeps hanging with the following text :

           

          [05/06/20 20:34:24] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.3 This is a Unix agent

           

          jstack reports a deadlock in attachment :

           

          Found one Java-level deadlock:
           =============================
           "pool-1-thread-4 for channel id=17236":
           waiting to lock monitor 0x00007fbd8c015bc8 (object 0x000000076e658428, a hudson.slaves.SlaveComputer$SlaveInitializer$1),
           which is held by "Channel reader thread: channel"
           "Channel reader thread: channel":
           waiting to lock monitor 0x00007fbd9401aed8 (object 0x000000076ded1cc8, a hudson.remoting.RemoteClassLoader),
           which is held by "pool-1-thread-4 for channel id=17236"

           

          On april 3 2020 in commit [https://github.com/jenkinsci/jenkins/commit/3262d8b7cfbdd1a2e026ad34644e0f9607f5724d]  

          a new synchronized  method was added that is partly the cause of this deadlock .See line 1026 in main/java/hudson/slaves/SlaveComputer.java 

           

           

           
          New: After launching an agent it keeps hanging with the following text :

           

          [05/06/20 20:34:24] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.3 This is a Unix agent

           

          This happens since a couple of weeks

           

          jstack reports a deadlock in attachment :

           

          Found one Java-level deadlock:
           =============================
           "pool-1-thread-4 for channel id=17236":
           waiting to lock monitor 0x00007fbd8c015bc8 (object 0x000000076e658428, a hudson.slaves.SlaveComputer$SlaveInitializer$1),
           which is held by "Channel reader thread: channel"
           "Channel reader thread: channel":
           waiting to lock monitor 0x00007fbd9401aed8 (object 0x000000076ded1cc8, a hudson.remoting.RemoteClassLoader),
           which is held by "pool-1-thread-4 for channel id=17236"

           

          On april 3 2020 in commit [https://github.com/jenkinsci/jenkins/commit/3262d8b7cfbdd1a2e026ad34644e0f9607f5724d]  

          a new synchronized  method was added that is partly the cause of this deadlock .See line 1026 in main/java/hudson/slaves/SlaveComputer.java 

           

           

           

          Daniel Beck added a comment -

          FYI jglick since your change is mentioned.

          Daniel Beck added a comment - FYI jglick since your change is mentioned.

          Jesse Glick added a comment -

          The “new” method was just an override of an existing synchronized method, though the deadlock does smell like a regression in the new code before the super call.

          You have a custom logger on hudson.remoting.Channel (or hudson.remoting) at FINE or below I think? As a workaround, try removing that logger.

          Jesse Glick added a comment - The “new” method was just an override of an existing synchronized method, though the deadlock does smell like a regression in the new code before the super call. You have a custom logger on hudson.remoting.Channel (or hudson.remoting ) at FINE or below I think? As a workaround, try removing that logger.
          Jesse Glick made changes -
          Labels Original: deadlock New: deadlock regression
          Jesse Glick made changes -
          Component/s New: core [ 15593 ]
          Component/s Original: remoting [ 15489 ]
          Jesse Glick made changes -
          Assignee Original: Jeff Thompson [ jthompson ] New: Jesse Glick [ jglick ]
          Jesse Glick made changes -
          Status Original: Open [ 1 ] New: In Progress [ 3 ]
          Jesse Glick made changes -
          Labels Original: deadlock regression New: deadlock lts-candidate regression

            jglick Jesse Glick
            bheupers Bart Heupers
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: