Uploaded image for project: 'Jenkins'
  1. Jenkins
  2. JENKINS-61023

Jenkins agent threads deadlock due to remoting.jar

    XMLWordPrintable

    Details

    • Similar Issues:

      Description

      ref) https://wiki.jenkins.io/display/JENKINS/Remoting+issue

       

      description:

      we setup jenkins master and several agents. one of agents periodically deadlocked and cpu spiked. I found It because SynchronousCommandTransport$ReaderThread.run from remoting.jar. (ref: https://wiki.jenkins.io/display/JENKINS/Remoting+issue)

      I added some screenshots. I really appreciate if you take a look and help us to find root cause.

       

      agent server stats:

       

      cpu usage:

       

      tracking PID 15699: ( It was remoting.jar )

       

      agentthread dump:  

       

       

      Thanks!

        Attachments

        1. channel thread.png
          channel thread.png
          77 kB
        2. cpu usage of remoting_jar.png
          cpu usage of remoting_jar.png
          20 kB
        3. remoting_jar ps.png
          remoting_jar ps.png
          13 kB
        4. stats.png
          stats.png
          134 kB

          Activity

          doubleknd26 Kideok Kim created issue -
          doubleknd26 Kideok Kim made changes -
          Field Original Value New Value
          Attachment channel thread.png [ 50164 ]
          Attachment cpu usage of remoting_jar.png [ 50165 ]
          Attachment remoting_jar ps.png [ 50166 ]
          Attachment stats.png [ 50167 ]
          doubleknd26 Kideok Kim made changes -
          Description *descriptiion:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot.

           

          *slave server stats:*

          !image-2020-02-09-23-46-42-173.png|width=925,height=312!

           

          *cpu usage:*

          *!image-2020-02-09-23-48-03-663.png!*

           

          *tracking PID 15699: (* It was remoting.jar )

          *!image-2020-02-09-23-48-19-132.png!*

           

          *slave thread dump:*  !image-2020-02-09-23-47-26-493.png|width=849,height=291!

          I really appreciate if you take a look and help us to find root cause.

          Thanks!
          *descriptiion:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot.

           

          *slave server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *slave thread dump:*  

          !channel thread.png|thumbnail!

           

           

          I really appreciate if you take a look and help us to find root cause.

          Thanks!
          doubleknd26 Kideok Kim made changes -
          Description *descriptiion:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot.

           

          *slave server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *slave thread dump:*  

          !channel thread.png|thumbnail!

           

           

          I really appreciate if you take a look and help us to find root cause.

          Thanks!
          *description:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot.

           

          *slave server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *slave thread dump:*  

          !channel thread.png|thumbnail!

           

           

          I really appreciate if you take a look and help us to find root cause.

          Thanks!
          doubleknd26 Kideok Kim made changes -
          Description *description:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *remoting.jar*. Here is a screenshot.

           

          *slave server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *slave thread dump:*  

          !channel thread.png|thumbnail!

           

           

          I really appreciate if you take a look and help us to find root cause.

          Thanks!
          ref) [https://wiki.jenkins.io/display/JENKINS/Remoting+issue]

           

          *description:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *SynchronousCommandTransport$ReaderThread.run* from *remoting.jar*. (ref: [https://wiki.jenkins.io/display/JENKINS/Remoting+issue])

          I added some screenshots. I really appreciate if you take a look and help us to find root cause.

           

          *slave server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *slave thread dump:*  

          !channel thread.png|thumbnail!

           

           

          Thanks!
          slide_o_mix Alex Earl made changes -
          Summary Jenkins slave threads deadlock due to remoting.jar Jenkins agent threads deadlock due to remoting.jar
          slide_o_mix Alex Earl made changes -
          Description ref) [https://wiki.jenkins.io/display/JENKINS/Remoting+issue]

           

          *description:*

          we setup jenkins master and several slaves. one of slaves periodically deadlocked and cpu spiked. I found It because *SynchronousCommandTransport$ReaderThread.run* from *remoting.jar*. (ref: [https://wiki.jenkins.io/display/JENKINS/Remoting+issue])

          I added some screenshots. I really appreciate if you take a look and help us to find root cause.

           

          *slave server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *slave thread dump:*  

          !channel thread.png|thumbnail!

           

           

          Thanks!
          ref) [https://wiki.jenkins.io/display/JENKINS/Remoting+issue]

           

          *description:*

          we setup jenkins master and several agents. one of agents periodically deadlocked and cpu spiked. I found It because *SynchronousCommandTransport$ReaderThread.run* from *remoting.jar*. (ref: [https://wiki.jenkins.io/display/JENKINS/Remoting+issue])

          I added some screenshots. I really appreciate if you take a look and help us to find root cause.

           

          *agent server stats:*

          !stats.png|thumbnail!

           

          *cpu usage:*

          !cpu usage of remoting_jar.png|thumbnail!

           

          *tracking PID 15699: (* It was remoting.jar )

          !remoting_jar ps.png|thumbnail!

           

          *agentthread dump:*  

          !channel thread.png|thumbnail!

           

           

          Thanks!

            People

            Assignee:
            jthompson Jeff Thompson
            Reporter:
            doubleknd26 Kideok Kim
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated: