Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-62181

Remoting deadlock when launching remote agent

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved (View Workflow)
    • Priority: Critical
    • Resolution: Fixed
    • Component/s: core
    • Environment:
    • Similar Issues:
    • Released As:
      Jenkins 2.238

      Description

      After launching an agent it keeps hanging with the following text :

       

      [05/06/20 20:34:24] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.3 This is a Unix agent

       

      This happens since a couple of weeks

       

      jstack reports a deadlock in attachment :

       

      Found one Java-level deadlock:
      =============================
      "pool-1-thread-4 for channel id=17236":
      waiting to lock monitor 0x00007fbd8c015bc8 (object 0x000000076e658428, a hudson.slaves.SlaveComputer$SlaveInitializer$1),
      which is held by "Channel reader thread: channel"
      "Channel reader thread: channel":
      waiting to lock monitor 0x00007fbd9401aed8 (object 0x000000076ded1cc8, a hudson.remoting.RemoteClassLoader),
      which is held by "pool-1-thread-4 for channel id=17236"

       

      On april 3 2020 in commit https://github.com/jenkinsci/jenkins/commit/3262d8b7cfbdd1a2e026ad34644e0f9607f5724d  

      a new synchronized  method was added that is partly the cause of this deadlock .See line 1026 in main/java/hudson/slaves/SlaveComputer.java 

       

       

       

        Attachments

          Issue Links

            Activity

            bheupers Bart Heupers created issue -
            bheupers Bart Heupers made changes -
            Field Original Value New Value
            Description After launching an agent it keeps hangin with the follwing text :

             

            [05/06/20 20:34:24] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.3 This is a Unix agent

             

            jstack reports a deadlock in attachment :

             

            Found one Java-level deadlock:
            =============================
            "pool-1-thread-4 for channel id=17236":
             waiting to lock monitor 0x00007fbd8c015bc8 (object 0x000000076e658428, a hudson.slaves.SlaveComputer$SlaveInitializer$1),
             which is held by "Channel reader thread: channel"
            "Channel reader thread: channel":
             waiting to lock monitor 0x00007fbd9401aed8 (object 0x000000076ded1cc8, a hudson.remoting.RemoteClassLoader),
             which is held by "pool-1-thread-4 for channel id=17236"

             

            On april 3 2020 in commit [https://github.com/jenkinsci/jenkins/commit/3262d8b7cfbdd1a2e026ad34644e0f9607f5724d]  

            a new synchronized  method was added that is partly the cause of this deadlock .See line 1026 in main/java/hudson/slaves/SlaveComputer.java 

             

             

             
            After launching an agent it keeps hanging with the following text :

             

            [05/06/20 20:34:24] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.3 This is a Unix agent

             

            jstack reports a deadlock in attachment :

             

            Found one Java-level deadlock:
             =============================
             "pool-1-thread-4 for channel id=17236":
             waiting to lock monitor 0x00007fbd8c015bc8 (object 0x000000076e658428, a hudson.slaves.SlaveComputer$SlaveInitializer$1),
             which is held by "Channel reader thread: channel"
             "Channel reader thread: channel":
             waiting to lock monitor 0x00007fbd9401aed8 (object 0x000000076ded1cc8, a hudson.remoting.RemoteClassLoader),
             which is held by "pool-1-thread-4 for channel id=17236"

             

            On april 3 2020 in commit [https://github.com/jenkinsci/jenkins/commit/3262d8b7cfbdd1a2e026ad34644e0f9607f5724d]  

            a new synchronized  method was added that is partly the cause of this deadlock .See line 1026 in main/java/hudson/slaves/SlaveComputer.java 

             

             

             
            bheupers Bart Heupers made changes -
            Description After launching an agent it keeps hanging with the following text :

             

            [05/06/20 20:34:24] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.3 This is a Unix agent

             

            jstack reports a deadlock in attachment :

             

            Found one Java-level deadlock:
             =============================
             "pool-1-thread-4 for channel id=17236":
             waiting to lock monitor 0x00007fbd8c015bc8 (object 0x000000076e658428, a hudson.slaves.SlaveComputer$SlaveInitializer$1),
             which is held by "Channel reader thread: channel"
             "Channel reader thread: channel":
             waiting to lock monitor 0x00007fbd9401aed8 (object 0x000000076ded1cc8, a hudson.remoting.RemoteClassLoader),
             which is held by "pool-1-thread-4 for channel id=17236"

             

            On april 3 2020 in commit [https://github.com/jenkinsci/jenkins/commit/3262d8b7cfbdd1a2e026ad34644e0f9607f5724d]  

            a new synchronized  method was added that is partly the cause of this deadlock .See line 1026 in main/java/hudson/slaves/SlaveComputer.java 

             

             

             
            After launching an agent it keeps hanging with the following text :

             

            [05/06/20 20:34:24] [SSH] Starting agent process: cd "/var/lib/jenkins" && java -jar remoting.jar -workDir /var/lib/jenkins May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir INFO: Using /var/lib/jenkins/remoting as a remoting work directory May 06, 2020 8:34:25 PM org.jenkinsci.remoting.engine.WorkDirManager setupLogging INFO: Both error and output logs will be printed to /var/lib/jenkins/remoting <===[JENKINS REMOTING CAPACITY]===>channel started Remoting version: 4.3 This is a Unix agent

             

            This happens since a couple of weeks

             

            jstack reports a deadlock in attachment :

             

            Found one Java-level deadlock:
             =============================
             "pool-1-thread-4 for channel id=17236":
             waiting to lock monitor 0x00007fbd8c015bc8 (object 0x000000076e658428, a hudson.slaves.SlaveComputer$SlaveInitializer$1),
             which is held by "Channel reader thread: channel"
             "Channel reader thread: channel":
             waiting to lock monitor 0x00007fbd9401aed8 (object 0x000000076ded1cc8, a hudson.remoting.RemoteClassLoader),
             which is held by "pool-1-thread-4 for channel id=17236"

             

            On april 3 2020 in commit [https://github.com/jenkinsci/jenkins/commit/3262d8b7cfbdd1a2e026ad34644e0f9607f5724d]  

            a new synchronized  method was added that is partly the cause of this deadlock .See line 1026 in main/java/hudson/slaves/SlaveComputer.java 

             

             

             
            jglick Jesse Glick made changes -
            Labels deadlock deadlock regression
            jglick Jesse Glick made changes -
            Component/s core [ 15593 ]
            Component/s remoting [ 15489 ]
            jglick Jesse Glick made changes -
            Assignee Jeff Thompson [ jthompson ] Jesse Glick [ jglick ]
            jglick Jesse Glick made changes -
            Status Open [ 1 ] In Progress [ 3 ]
            jglick Jesse Glick made changes -
            Labels deadlock regression deadlock lts-candidate regression
            jglick Jesse Glick made changes -
            Remote Link This issue links to "jenkins #4714 (Web Link)" [ 24902 ]
            jglick Jesse Glick made changes -
            Status In Progress [ 3 ] In Review [ 10005 ]
            markewaite Mark Waite made changes -
            Link This issue is duplicated by JENKINS-62244 [ JENKINS-62244 ]
            oleg_nenashev Oleg Nenashev made changes -
            Released As Jenkins 2.238
            Resolution Fixed [ 1 ]
            Status In Review [ 10005 ] Resolved [ 5 ]
            olivergondza Oliver Gondža made changes -
            Labels deadlock lts-candidate regression 2.235.1-fixed deadlock regression
            marcus_phi Marcus Philip made changes -
            Link This issue is related to JENKINS-63082 [ JENKINS-63082 ]
            jglick Jesse Glick made changes -
            Link This issue relates to JENKINS-63458 [ JENKINS-63458 ]

              People

              Assignee:
              jglick Jesse Glick
              Reporter:
              bheupers Bart Heupers
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: